vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_1066.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

152 lines
26 KiB
HTML
Raw Blame History

<a name="mrs_01_1066"></a><a name="mrs_01_1066"></a>
<h1 class="topictitle1">Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS Through the Flume Client</h1>
<div id="body8662426"><div class="section" id="mrs_01_1066__en-us_topic_0000001173471374_sa09be57bc8424fa5868559f0b93a5e0f"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1066__en-us_topic_0000001173471374_a2fdbe65f2a6b44d29de98457ef8bf7a0">This section describes how to use Flume client to collect logs from the Topic list (test1) of Kafka client and save them to the <span class="filepath" id="mrs_01_1066__en-us_topic_0000001173471374_filepath6733502571000"><b>/flume/test</b></span> directory on HDFS.</p>
<div class="note" id="mrs_01_1066__en-us_topic_0000001173471374_n1c24e0864c8f46ec90574b6cdb537684"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p class="text" id="mrs_01_1066__en-us_topic_0000001173471374_a6eae353646a34effa2e1ace5538bddfd">By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see <a href="mrs_01_1069.html">Configuring the Encrypted Transmission</a>.</p>
</div></div>
</div>
<div class="section" id="mrs_01_1066__en-us_topic_0000001173471374_s2c591043ec4f4a73affa1eeec294c526"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1066__en-us_topic_0000001173471374_ua08e403ea2fa44f68dfa92b00eed1580"><li id="mrs_01_1066__en-us_topic_0000001173471374_l4ef5aa2f014648d8b55c2a4a5516615c">The cluster, HDFS, Kafka, and Flume service have been installed.</li><li id="mrs_01_1066__li1546131633911">The Flume client has been installed. For details about how to install the client, see <a href="mrs_01_1595.html">Installing the Flume Client on Clusters</a>.</li><li id="mrs_01_1066__en-us_topic_0000001173471374_l18bdf785d21642c0ac5be50903211f02">You have created user <strong id="mrs_01_1066__en-us_topic_0000001173471374_b1941017484395">flume_hdfs</strong> and authorized the HDFS directory and data to be operated during log verification. For details, see <a href="mrs_01_1856.html">Adding a Ranger Access Permission Policy for HDFS</a>.</li><li id="mrs_01_1066__en-us_topic_0000001173471374_l579cf22f24bf40f2b3bb933038a2ad0c">The network environment of the cluster is secure.</li></ul>
</div>
<div class="section" id="mrs_01_1066__en-us_topic_0000001173471374_s8a7bc8a8d6dc4259ad505e48de3fef69"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1066__en-us_topic_0000001173471374_ol82782043123318"><li id="mrs_01_1066__en-us_topic_0000001173471374_li2268134373311"><span>On FusionInsight Manager, choose <span class="menucascade" id="mrs_01_1066__en-us_topic_0000001173471374_menucascade136082026142517"><b><span class="uicontrol" id="mrs_01_1066__en-us_topic_0000001173471374_uicontrol6607102692517">System &gt; User</span></b></span> and choose <span class="menucascade" id="mrs_01_1066__en-us_topic_0000001173471374_menucascade1860832602514"><b><span class="uicontrol" id="mrs_01_1066__en-us_topic_0000001173471374_uicontrol26081026142514">More &gt; Download Authentication Credential</span></b></span> to download the Kerberos certificate file of user <strong id="mrs_01_1066__en-us_topic_0000001173471374_b1460942622515">flume_hdfs</strong> and save it to the local host.</span></li><li id="mrs_01_1066__en-us_topic_0000001173471374_li122788439332"><span>Configure the client parameters of the Flume role.</span><p><ol type="a" id="mrs_01_1066__en-us_topic_0000001173471374_ol32781439334"><li id="mrs_01_1066__en-us_topic_0000001173471374_li2277104353317">Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.<ol class="substepthirdol" id="mrs_01_1066__en-us_topic_0000001173471374_ol15277104353310"><li id="mrs_01_1066__en-us_topic_0000001173471374_li132682438330">Log in to FusionInsight Manager and choose <strong id="mrs_01_1066__en-us_topic_0000001173471374_b4876837921000">Cluster</strong> &gt; <em id="mrs_01_1066__en-us_topic_0000001173471374_i1956589771000">Name of the desired cluster</em> &gt; <strong id="mrs_01_1066__en-us_topic_0000001173471374_b6271113331000">Services</strong> &gt; <strong id="mrs_01_1066__en-us_topic_0000001173471374_b12830765441000">Flume</strong> &gt; <strong id="mrs_01_1066__en-us_topic_0000001173471374_b15232843291000">Configuration Tool</strong>.</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li122684438339">Set <strong id="mrs_01_1066__en-us_topic_0000001173471374_b817710562612">Agent Name</strong> to <strong id="mrs_01_1066__en-us_topic_0000001173471374_b191829582618">client</strong>. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.<p id="mrs_01_1066__en-us_topic_0000001173471374_p5268143183319">For example, use Kafka Source, File Channel, and HDFS Sink, as shown in <a href="#mrs_01_1066__en-us_topic_0000001173471374_fig1526804343314">Figure 1</a>.</p>
<div class="fignone" id="mrs_01_1066__en-us_topic_0000001173471374_fig1526804343314"><a name="mrs_01_1066__en-us_topic_0000001173471374_fig1526804343314"></a><a name="en-us_topic_0000001173471374_fig1526804343314"></a><span class="figcap"><b>Figure 1 </b>Example for the Flume configuration tool</span><br><span><img id="mrs_01_1066__en-us_topic_0000001173471374_image826854383317" src="en-us_image_0000001349059513.png"></span></div>
</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li8269164313334">Double-click the source, channel, and sink. Set corresponding configuration parameters by seeing <a href="#mrs_01_1066__en-us_topic_0000001173471374_table1127710438338">Table 1</a> based on the actual environment.<div class="note" id="mrs_01_1066__en-us_topic_0000001173471374_note2269164316333"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1066__en-us_topic_0000001173471374_ul2269843173311"><li id="mrs_01_1066__en-us_topic_0000001173471374_li5269124323312">If the client parameters of the Flume role have been configured, you can obtain the existing client parameter configuration file from <em id="mrs_01_1066__en-us_topic_0000001173471374_i12332220101000">client installation directory</em><strong id="mrs_01_1066__en-us_topic_0000001173471374_b7732194481000">/fusioninsight-flume-1.9.0/conf/properties.properties</strong> to ensure that the configuration is in concordance with the previous. Log in to FusionInsight Manager, choose <strong id="mrs_01_1066__en-us_topic_0000001173471374_b1861748051000">Cluster</strong> &gt; <em id="mrs_01_1066__en-us_topic_0000001173471374_i15215384441000">Name of the desired cluster</em> &gt; <strong id="mrs_01_1066__en-us_topic_0000001173471374_b16686914511000">Services</strong> &gt; <strong id="mrs_01_1066__en-us_topic_0000001173471374_b5925785101000">Flume</strong> &gt; <strong id="mrs_01_1066__en-us_topic_0000001173471374_b18610310601000">Configuration Tool</strong> &gt; <strong id="mrs_01_1066__en-us_topic_0000001173471374_b8418509921000">Import</strong>, import the file, and modify the configuration items related to non-encrypted transmission.</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li52697432336">It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.</li></ul>
</div></div>
</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li152771943153315">Click <strong id="mrs_01_1066__en-us_topic_0000001173471374_b17403381311000">Export</strong> to save the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b15190264141000">properties.properties</strong> configuration file to the local.
<div class="tablenoborder"><a name="mrs_01_1066__en-us_topic_0000001173471374_table1127710438338"></a><a name="en-us_topic_0000001173471374_table1127710438338"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1066__en-us_topic_0000001173471374_table1127710438338" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters to be modified of the Flume role client</caption><thead align="left"><tr id="mrs_01_1066__en-us_topic_0000001173471374_row2270164320335"><th align="left" class="cellrowborder" valign="top" width="33%" id="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1"><p id="mrs_01_1066__en-us_topic_0000001173471374_p1027004318337">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33%" id="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2"><p id="mrs_01_1066__en-us_topic_0000001173471374_p92701843153319">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="34%" id="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3"><p id="mrs_01_1066__en-us_topic_0000001173471374_p5270194312337">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1066__en-us_topic_0000001173471374_row8270184353310"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p427084318336">Name</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p52704439334">The value must be unique and cannot be left blank.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p15270164320335">test</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row527194311333"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p172702043173310">kafka.topics</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p15270154318335">Specifies the subscribed Kafka topic list, in which topics are separated by commas (,). This parameter cannot be left blank.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p152711643133310">test1</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row5271204353315"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p62716433333">kafka.consumer.group.id</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p7271543193320">Specifies the data group ID obtained from Kafka. This parameter cannot be left blank.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p182711043113320">flume</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row14272124313331"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p162711943193310">kafka.bootstrap.servers</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p62713436338">Specifies the bootstrap IP address and port list of Kafka. The default value is all Kafka lists in a Kafka cluster. If Kafka has been installed in the cluster and its configurations have been synchronized, this parameter can be left blank.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p727116434338">192.168.101.10:21007</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row15272114310339"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p8272174317331">batchSize</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p9272144393310">Specifies the number of events that Flume sends in a batch (number of data pieces).</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p1127214436333">61200</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row22721043123318"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p3272174317339">dataDirs</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p82726432338">Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b20104595681000">/srv/BigData/hadoop/dataX/flume/data</strong> directory can be used. <strong id="mrs_01_1066__en-us_topic_0000001173471374_b6661897821000">dataX</strong> ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p12272124319331">/srv/BigData/hadoop/data1/flume/data</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row22731143153317"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p17272104363311">checkpointDir</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p82739437336">Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b7306623451000">/srv/BigData/hadoop/dataX/flume/checkpoint</strong> directory can be used. <strong id="mrs_01_1066__en-us_topic_0000001173471374_b20490785581000">dataX</strong> ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p527344314330">/srv/BigData/hadoop/data1/flume/checkpoint</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row13273164312332"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p20273194316330">transactionCapacity</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p02731843123312">Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current Channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p1327315432338">61200</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row192741439338"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p1427314313333">hdfs.path</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p827334311334">Specifies the HDFS data write directory. This parameter cannot be left blank.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p1327494343317">hdfs://hacluster/flume/test</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row427444333312"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p6274194311330">hdfs.inUsePrefix</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p16274343193313">Specifies the prefix of the file that is being written to HDFS.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p12741543193317">TMP_</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row1927517437334"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p152741343143316">hdfs.batchSize</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p122740436337">Specifies the maximum number of events that can be written to HDFS once.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p102751643133318">61200</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row1727518436332"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p92757433335">hdfs.kerberosPrincipal</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p627514435335">Specifies the Kerberos authentication user, which is mandatory in security versions. This configuration is required only in security clusters.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p1427564311332">flume_hdfs</p>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row727724315337"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p152755433331">hdfs.kerberosKeytab</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p127574313316">Specifies the keytab file path for Kerberos authentication, which is mandatory in security versions. This configuration is required only in security clusters.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p0275104314335">/opt/test/conf/user.keytab</p>
<div class="note" id="mrs_01_1066__en-us_topic_0000001173471374_note427734323310"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1066__en-us_topic_0000001173471374_p16277144310337">Obtain the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b13167454951000">user.keytab</strong> file from the Kerberos certificate file of the user <strong id="mrs_01_1066__en-us_topic_0000001173471374_b16580011071000">flume_hdfs</strong>. In addition, ensure that the user who installs and runs the Flume client has the read and write permissions on the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b5966957081000">user.keytab</strong> file.</p>
</div></div>
</td>
</tr>
<tr id="mrs_01_1066__en-us_topic_0000001173471374_row3277164343314"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.1 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p2277204317338">hdfs.useLocalTimeStamp</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.2 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p827724318334">Specifies whether to use the local time. Possible values are <strong id="mrs_01_1066__en-us_topic_0000001173471374_b15227233971000">true</strong> and <strong id="mrs_01_1066__en-us_topic_0000001173471374_b12693067391000">false</strong>.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.4.3.2.4.1.3 "><p id="mrs_01_1066__en-us_topic_0000001173471374_p15277943133316">true</p>
</td>
</tr>
</tbody>
</table>
</div>
</li></ol>
</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li15278204311335">Upload the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b5721677701000">properties.properties</strong> file to <strong id="mrs_01_1066__en-us_topic_0000001173471374_b11730857911000">flume/conf/</strong> under the installation directory of the Flume client.</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li6278204310334">To connect the Flume client to the HDFS, you need to add the following configuration:<ol class="substepthirdol" id="mrs_01_1066__en-us_topic_0000001173471374_ol7278843173311"><li id="mrs_01_1066__en-us_topic_0000001173471374_li22781343113313">Download the Kerberos certificate of account <strong id="mrs_01_1066__en-us_topic_0000001173471374_b5692048041000">flume_hdfs</strong> and obtain the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b21250537971000">krb5.conf</strong> configuration file. Upload the configuration file to the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b12142365351000">fusioninsight-flume-1.9.0/conf/</strong> directory on the node where the client is installed.</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li1727864315338">In <strong id="mrs_01_1066__en-us_topic_0000001173471374_b11194893201000">fusioninsight-flume-1.9.0/conf/</strong>, create the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b20075104651000">jaas.conf</strong> configuration file.<p id="mrs_01_1066__en-us_topic_0000001173471374_p2278134311335"><strong id="mrs_01_1066__en-us_topic_0000001173471374_b327834310338">vi jaas.conf</strong></p>
<pre class="screen" id="mrs_01_1066__en-us_topic_0000001173471374_screen20278194317339">KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="<strong id="mrs_01_1066__en-us_topic_0000001173471374_b827864363310">/opt/test/conf/user.keytab</strong>"
principal="<strong id="mrs_01_1066__en-us_topic_0000001173471374_b1227894314334">flume_hdfs@</strong><em id="mrs_01_1066__en-us_topic_0000001173471374_i1235215297387">&lt;System domain name&gt;</em>"
useTicketCache=false
storeKey=true
debug=true;
};</pre>
<p id="mrs_01_1066__en-us_topic_0000001173471374_p162781435337">Values of <strong id="mrs_01_1066__en-us_topic_0000001173471374_b17392194781000">keyTab</strong> and <strong id="mrs_01_1066__en-us_topic_0000001173471374_b13958809631000">principal</strong> vary depending on the actual situation.</p>
</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li1627894317331">Obtain configuration files <strong id="mrs_01_1066__en-us_topic_0000001173471374_b4092624401000">core-site.xml</strong> and <strong id="mrs_01_1066__en-us_topic_0000001173471374_b10577338651000">hdfs-site.xml</strong> from <strong id="mrs_01_1066__en-us_topic_0000001173471374_b4598466481000">/opt/FusionInsight_Cluster_<em id="mrs_01_1066__en-us_topic_0000001173471374_i4182669051000">&lt;Cluster ID&gt;</em>_Flume_ClientConfig/Flume/config</strong> and upload them to <strong id="mrs_01_1066__en-us_topic_0000001173471374_b6542527891000">fusioninsight-flume-1.9.0/conf/</strong>.</li></ol>
</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li11278143193313">Restart the Flume service.<p id="mrs_01_1066__p1879517526112"><a name="mrs_01_1066__en-us_topic_0000001173471374_li11278143193313"></a><a name="en-us_topic_0000001173471374_li11278143193313"></a><strong id="mrs_01_1066__b138321612181211">flume-manager.sh restart</strong></p>
</li></ol>
</p></li></ol><ol start="3" id="mrs_01_1066__en-us_topic_0000001173471374_ol1526612430336"><li id="mrs_01_1066__en-us_topic_0000001173471374_li3266843173312"><span>Verify log transmission.</span><p><ol type="a" id="mrs_01_1066__en-us_topic_0000001173471374_ol926614313335"><li id="mrs_01_1066__en-us_topic_0000001173471374_li5266134323311">Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager</a>. Choose <strong id="mrs_01_1066__en-us_topic_0000001173471374_b1044967306">Cluster &gt; </strong><em id="mrs_01_1066__en-us_topic_0000001173471374_i174542713018">Name of the desired cluster</em> &gt; <strong id="mrs_01_1066__en-us_topic_0000001173471374_b1445516716012">Services</strong> &gt; <strong id="mrs_01_1066__en-us_topic_0000001173471374_b94551771904">HDFS</strong>, click the HDFS WebUI link of <strong id="mrs_01_1066__en-us_topic_0000001173471374_b1383991313366">NameNode</strong> (<em id="mrs_01_1066__en-us_topic_0000001173471374_i10476162215361">Node name</em>, <strong id="mrs_01_1066__en-us_topic_0000001173471374_b18473191710367">Active</strong>) to go to the HDFS WebUI, and choose <strong id="mrs_01_1066__en-us_topic_0000001173471374_b16456271107">Utilities &gt; Browse the file system</strong>.</li><li id="mrs_01_1066__en-us_topic_0000001173471374_li026614353315">Check whether the data is generated in the <strong id="mrs_01_1066__en-us_topic_0000001173471374_b7567374561000">/flume/test</strong> directory on the HDFS.<div class="fignone" id="mrs_01_1066__en-us_topic_0000001173471374_fig42663436332"><span class="figcap"><b>Figure 2 </b>Checking HDFS directories and files</span><br><span><img id="mrs_01_1066__en-us_topic_0000001173471374_image1416104620347" src="en-us_image_0000001296219300.png"></span></div>
</li></ol>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1059.html">Non-Encrypted Transmission</a></div>
</div>
</div>
View Git Blame Copy Permalink