vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide/mrs_01_1067.html
Yang, Tong 6182f91ba8 MRS component operation guide_normal 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-09 14:55:21 +00:00

226 lines
34 KiB
HTML
Raw Blame History

<a name="mrs_01_1067"></a><a name="mrs_01_1067"></a>
<h1 class="topictitle1">Typical Scenario: Collecting Local Static Logs and Uploading Them to HBase</h1>
<div id="body1590374515509"><div class="section" id="mrs_01_1067__s9f037f407033421282a22689e5a7739b"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1067__a74acb2762a234cf58ee775bf6fe30399">This section describes how to use the Flume client to collect static logs from a local host and save them to the <strong id="mrs_01_1067__b184371044105018">flume_test</strong> HBase table. In this scenario, multi-level agents are cascaded.</p>
<p id="mrs_01_1067__p1190153615238">This section applies to MRS 3.<em id="mrs_01_1067__i1277382012566">x</em> or later clusters.</p>
<div class="note" id="mrs_01_1067__n8b3fc02732014697a53ab02d88894c5e"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p class="text" id="mrs_01_1067__en-us_topic_0060185024_p742917144718">By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see <a href="mrs_01_1069.html">Configuring the Encrypted Transmission</a>. The configuration applies to scenarios where only the server is configured, for example, Spooldir Source+File Channel+HBase Sink.</p>
</div></div>
</div>
<div class="section" id="mrs_01_1067__sed3218aed88f44579304ef2130fb0cad"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1067__u3117eb6bf579491db0365da566af193d"><li id="mrs_01_1067__ld3f4165aab6444ae8d1725bfdd3b53fb">The cluster has been installed, including the HBase and Flume services.</li><li id="mrs_01_1067__li3520154014471">The Flume client has been installed. For details, see <span id="mrs_01_1067__ph18236135212506"><a href="https://docs.otc.t-systems.com/cmpntguide/mrs/mrs_01_0392.html" target="_blank" rel="noopener noreferrer">Installing the Flume Client</a></span>.</li><li id="mrs_01_1067__l3f6a41861ee945fab11f06e86762cefe">The network environment of the cluster is secure.</li><li id="mrs_01_1067__l3bc4d64fda194455b802b2debbba3959">An HBase table has been created by running the <strong id="mrs_01_1067__a1f9cddc78cfb4eb5a0e3c5a812c1368c">create 'flume_test', 'cf'</strong> command.</li><li id="mrs_01_1067__l8f629d1142e24b24a6b9193909f6c7cf">The system administrator has understood service requirements and prepared HBase administrator <strong id="mrs_01_1067__b2051516014410">flume_hbase</strong>.</li></ul>
</div>
<div class="section" id="mrs_01_1067__s2caaed08a75b42e58b548eeed73e1c12"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1067__o2622c2436dc4441fb30fee0354d305bf"><li id="mrs_01_1067__l0f2bf40dc3c94579b9b2aa589cdc7d02"><span>On FusionInsight Manager, choose <span class="menucascade" id="mrs_01_1067__menucascade451188927102913"><b><span class="uicontrol" id="mrs_01_1067__uicontrol1599116188102913">System &gt; User</span></b></span> and choose <span class="menucascade" id="mrs_01_1067__menucascade1999490490102913"><b><span class="uicontrol" id="mrs_01_1067__uicontrol2095060623102913">More &gt; Download Authentication Credential</span></b></span> to download the Kerberos certificate file of user <strong id="mrs_01_1067__b1115642060102913">flume_hbase</strong> and save it to the local host.</span></li><li id="mrs_01_1067__ldb57e1a8d5a54bc090662402b1892d1b"><span>Configure the client parameters of the Flume role.</span><p><ol type="a" id="mrs_01_1067__oa869c77c078e4fe9af19286e9aa070fd"><li id="mrs_01_1067__l449ab29c933c4e8b9db2019d87fd1fc2">Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.<ol class="substepthirdol" id="mrs_01_1067__o50b41ea0f2394471a63de09ad4709e7f"><li id="mrs_01_1067__l372c2447ab714d918c3f8f0e4dbcaa3d">Log in to FusionInsight Manager and choose<strong id="mrs_01_1067__b76611832151215"> Cluster</strong> &gt; <strong id="mrs_01_1067__b9670193241213">Services</strong>. On the page that is displayed, choose <strong id="mrs_01_1067__b26712321129">Flume</strong>. On the displayed page, click the <strong id="mrs_01_1067__b2067193271210">Configuration Tool</strong> tab.</li><li id="mrs_01_1067__l53b876ad1b8a44cd82e3bde2ef66927b">Set <strong id="mrs_01_1067__b2098483187102913">Agent Name</strong> to <strong id="mrs_01_1067__b1272222045102913">client</strong>. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.<p id="mrs_01_1067__aa5c01f4dc0614479985d4d8ba9c48e0b">Use SpoolDir Source, File Channel, and Avro Sink.</p>
</li><li id="mrs_01_1067__le23a2fd990d142f5a9d80e83fcafd38e">Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to <a href="#mrs_01_1067__table15735429105511">Table 1</a> based on the actual environment.<div class="note" id="mrs_01_1067__n53ba107ab1ae4062b840c510cfdc6439"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1067__uac41818be9df4afeb54382f6a0fc197e"><li id="mrs_01_1067__l96cb3083c17f4cf5933cd9808a03c77b">If you want to continue using the <strong id="mrs_01_1067__b7195185011217">properties.propretites</strong> file by modifying it, log in to FusionInsight Manager, choose <strong id="mrs_01_1067__b0197750151216">Cluster</strong> &gt; <em id="mrs_01_1067__i1119817503125">Name of the desired cluster</em> &gt; <strong id="mrs_01_1067__b4199105018122">Services</strong>. On the page that is displayed, choose <strong id="mrs_01_1067__b15199195012122">Flume</strong>. On the displayed page, click the <strong id="mrs_01_1067__b52004508128">Configuration Tool</strong> tab, click <strong id="mrs_01_1067__b132011250101220">Import</strong>, import the file, and modify the configuration items related to non-encrypted transmission.</li><li id="mrs_01_1067__lf91b31c36966434b9ed8d2e497c3924c">It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.</li></ul>
</div></div>
<div class="tablenoborder"><a name="mrs_01_1067__table15735429105511"></a><a name="table15735429105511"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1067__table15735429105511" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters to be modified for the Flume role client</caption><thead align="left"><tr id="mrs_01_1067__row1173682965518"><th align="left" class="cellrowborder" valign="top" width="33%" id="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1"><p id="mrs_01_1067__p18736152975516">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="33%" id="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2"><p id="mrs_01_1067__p1173612955516">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="34%" id="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3"><p id="mrs_01_1067__p3736429115512">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1067__row97361429145512"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p137362297558">Name</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p3736229145510">The value must be unique and cannot be left blank.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p673622913550">test</p>
</td>
</tr>
<tr id="mrs_01_1067__row57363295552"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p1273652914557">spoolDir</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p1736132915514">Specifies the directory where the file to be collected resides. This parameter cannot be left blank. The directory needs to exist and have the write, read, and execute permissions on the flume running user.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p11736329165514">/srv/BigData/hadoop/data1/zb</p>
</td>
</tr>
<tr id="mrs_01_1067__row14736132905511"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p20736129165519">trackerDir</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p1773612995512">Specifies the path for storing the metadata of files collected by Flume.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p17736162975518">/srv/BigData/hadoop/data1/tracker</p>
</td>
</tr>
<tr id="mrs_01_1067__row973616295551"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p18736929145513">batchSize</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p17361297557">Specifies the number of events that Flume sends in a batch (number of data pieces). A larger value indicates higher performance and lower timeliness.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p1874042945510">61200</p>
</td>
</tr>
<tr id="mrs_01_1067__row197403299557"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p18740132965513">dataDirs</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p1174013297557">Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the <strong id="mrs_01_1067__b1622786339102913">/srv/BigData/hadoop/dataX/flume/data</strong> directory can be used. <strong id="mrs_01_1067__b326350878102913">dataX</strong> ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p874017296552">/srv/BigData/hadoop/data1/flume/data</p>
</td>
</tr>
<tr id="mrs_01_1067__row1574012914558"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p874110293552">checkpointDir</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p117411929105513">Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the <strong id="mrs_01_1067__b1520486991102913">/srv/BigData/hadoop/dataX/flume/checkpoint</strong> directory can be used. <strong id="mrs_01_1067__b1975219541102913">dataX</strong> ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p15741162911556">/srv/BigData/hadoop/data1/flume/checkpoint</p>
</td>
</tr>
<tr id="mrs_01_1067__row1774192911556"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p1374117294558">transactionCapacity</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p67411329165511">Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current Channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p67411929105518">61200</p>
<p id="mrs_01_1067__p67412296550"></p>
</td>
</tr>
<tr id="mrs_01_1067__row57411429185511"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p147411129155511">hostname</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p3741229195510">Specifies the name or IP address of the host whose data is to be sent. This parameter cannot be left blank. Name or IP address must be configured to be the name or IP address that the Avro source associated with it.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p774113297557">192.168.108.11</p>
</td>
</tr>
<tr id="mrs_01_1067__row1174112291552"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p17741162913556">port</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p0741329125514">Specifies the port that sends the data. This parameter cannot be left blank. It must be consistent with the port that is monitored by the connected Avro Source.</p>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p174113299553">21154</p>
</td>
</tr>
<tr id="mrs_01_1067__row174112292557"><td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p4741162918556">ssl</p>
</td>
<td class="cellrowborder" valign="top" width="33%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p47411629195511">Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.)</p>
<p id="mrs_01_1067__p1174112912554">Only Sources of the Avro type have this configuration item.</p>
<ul id="mrs_01_1067__ul2074192917558"><li id="mrs_01_1067__li4741182915510"><strong id="mrs_01_1067__b1185864826102913">true</strong> indicates that the function is enabled.</li><li id="mrs_01_1067__li10741102985519"><strong id="mrs_01_1067__b1669392845102913">false</strong> indicates that the client authentication function is not enabled.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="34%" headers="mcps1.3.3.2.2.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p1774122925512">false</p>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="mrs_01_1067__lf7c652eb5b944e03b816672cf8b89701">Click <strong id="mrs_01_1067__b1428471369102913">Export</strong> to save the <strong id="mrs_01_1067__b1881806574102913">properties.properties</strong> configuration file to the local.</li></ol>
</li><li id="mrs_01_1067__lc74347aa2ef5429aa350fac790188b2f">Upload the <strong id="mrs_01_1067__b807838481102913">properties.properties</strong> file to <strong id="mrs_01_1067__b1842087684102913">flume/conf/</strong> under the installation directory of the Flume client.</li></ol>
</p></li><li id="mrs_01_1067__la69566c4f50d41e1a3f36cba8e6096a0"><span>Configure the server parameters of the Flume role and upload the configuration file to the cluster.</span><p><ol type="a" id="mrs_01_1067__o24086c7e5c4e4d7e9891b225bafdde0a"><li id="mrs_01_1067__l9aed8563ae4444caa8e3bfd71b125b6b">Use the Flume configuration tool on the FusionInsight Manager portal to configure the server parameters and generate the configuration file.<ol class="substepthirdol" id="mrs_01_1067__oe9f06f719fc2404f8d520cbae7a41118"><li id="mrs_01_1067__lf1d11f08d0b74d8db2b0f93659318847">Log in to FusionInsight Manager and choose<strong id="mrs_01_1067__b1074085312127"> Cluster</strong> &gt; <strong id="mrs_01_1067__b3740185315123">Services</strong>. On the page that is displayed, choose <strong id="mrs_01_1067__b16741135318124">Flume</strong>. On the displayed page, click the <strong id="mrs_01_1067__b117411253141210">Configuration Tool</strong> tab.</li><li id="mrs_01_1067__lb39a58ac3b124f9e9f5b96d56d190f20">Set <strong id="mrs_01_1067__b42233805102913">Agent Name</strong> to <strong id="mrs_01_1067__b230659354102913">server</strong>. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.<p id="mrs_01_1067__a3246b8c511fa463783f70f4b280b55ae">For example, use Avro Source, File Channel, and HBase Sink.</p>
</li><li id="mrs_01_1067__l53891171fa4b4a2497fc2591eed8a026">Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to <a href="#mrs_01_1067__table77819014563">Table 2</a> based on the actual environment.<div class="note" id="mrs_01_1067__n75e8a0c4b3ec49ebbcec98834620763f"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1067__u8d3b6bf42cb04ff19544315666d4c5e6"><li id="mrs_01_1067__l71473dc3a92b41f990b5520595540726">If the server parameters of the Flume role have been configured, you can choose <strong id="mrs_01_1067__b904306185102913">Cluster</strong> &gt; <em id="mrs_01_1067__i1445545874102913">Name of the desired cluster</em> &gt; <strong id="mrs_01_1067__b2026111845102913">Services</strong> &gt; <strong id="mrs_01_1067__b1586615516102913">Flume</strong> &gt; <strong id="mrs_01_1067__b1594232785102913">Instance</strong> on FusionInsight Manager. Then select the corresponding Flume role instance and click the <strong id="mrs_01_1067__b1133495924102913">Download</strong> button behind the <span class="parmname" id="mrs_01_1067__parmname453519947102913"><b>flume.config.file</b></span> parameter on the <strong id="mrs_01_1067__b1265580738102913">Instance Configurations</strong> page to obtain the existing server parameter configuration file. Choose <strong id="mrs_01_1067__b1418792374102913">Cluster</strong> &gt; <em id="mrs_01_1067__i471624899102913">Name of the desired cluster</em> &gt; <strong id="mrs_01_1067__b1688553143102913">Services</strong> &gt; <strong id="mrs_01_1067__b1160019553102913">Flume</strong> &gt; <strong id="mrs_01_1067__b419706830102913">Configuration Tool</strong> &gt; <strong id="mrs_01_1067__b1719741017102913">Import</strong>, import the file, and modify the configuration items related to non-encrypted transmission.</li><li id="mrs_01_1067__l68a8c84820134797a7887ec602c91701">It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.</li><li id="mrs_01_1067__l77f00c553a2447c491533c33c0685523">A unique checkpoint directory needs to be configured for each File Channel.</li></ul>
</div></div>
<div class="tablenoborder"><a name="mrs_01_1067__table77819014563"></a><a name="table77819014563"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1067__table77819014563" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameters to be modified for the Flume role server</caption><thead align="left"><tr id="mrs_01_1067__row1278190165618"><th align="left" class="cellrowborder" valign="top" width="23.232323232323232%" id="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1"><p id="mrs_01_1067__p15798015620">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="45.45454545454545%" id="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2"><p id="mrs_01_1067__p1179106562">Description</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="31.313131313131308%" id="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3"><p id="mrs_01_1067__p87920055620">Example Value</p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1067__row10791201566"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p07950155611">Name</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p167913014561">The value must be unique and cannot be left blank.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p1379130135612">test</p>
</td>
</tr>
<tr id="mrs_01_1067__row37919017566"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p879905561">bind</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p979704563">Specifies the IP address to which Avro Source is bound. This parameter cannot be left blank. It must be configured as the IP address that the server configuration file will upload.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p137910165616">192.168.108.11</p>
</td>
</tr>
<tr id="mrs_01_1067__row5791075616"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p779180185610">port</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p379307561">Specifies the ID of the port that the Avro Source monitors. This parameter cannot be left blank. It must be configured as an unused port.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p279100175616">21154</p>
</td>
</tr>
<tr id="mrs_01_1067__row2791404564"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p15796025615">ssl</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p179130135610">Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.)</p>
<p id="mrs_01_1067__p187917014564">Only Sources of the Avro type have this configuration item.</p>
<ul id="mrs_01_1067__ul57930115613"><li id="mrs_01_1067__li108050165614"><strong id="mrs_01_1067__b1653579006">true</strong> indicates that the function is enabled.</li><li id="mrs_01_1067__li2807095617"><strong id="mrs_01_1067__b2059452903">false</strong> indicates that the client authentication function is not enabled.</li></ul>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p7801709565">false</p>
</td>
</tr>
<tr id="mrs_01_1067__row48018045615"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p148018065613">dataDirs</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p18018085612">Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the <strong id="mrs_01_1067__b474617800">/srv/BigData/hadoop/dataX/flume/data</strong> directory can be used. <strong id="mrs_01_1067__b1390183208">dataX</strong> ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p108015085613">/srv/BigData/hadoop/data1/flumeserver/data</p>
</td>
</tr>
<tr id="mrs_01_1067__row68013045618"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p680140105611">checkpointDir</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p198080175613">Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the <strong id="mrs_01_1067__b1330782999102913">/srv/BigData/hadoop/dataX/flume/checkpoint</strong> directory can be used. <strong id="mrs_01_1067__b1746550738102913">dataX</strong> ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p128011014566">/srv/BigData/hadoop/data1/flumeserver/checkpoint</p>
</td>
</tr>
<tr id="mrs_01_1067__row148017025611"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p58010165613">transactionCapacity</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p1280401564">Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current Channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p980601569">61200</p>
</td>
</tr>
<tr id="mrs_01_1067__row138090155616"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p18070135614">table</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p188011095613">Specifies the HBase table name. This parameter cannot be left blank.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p15800015567">flume_test</p>
</td>
</tr>
<tr id="mrs_01_1067__row14809014566"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p78060165617">columnFamily</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p188010015568">Specifies the HBase column family name. This parameter cannot be left blank.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p1818016561">cf</p>
</td>
</tr>
<tr id="mrs_01_1067__row8811207568"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p208115065610">batchSize</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p128110014568">Specifies the maximum number of events written to HBase by Flume in a batch.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p128114014568">61200</p>
</td>
</tr>
<tr id="mrs_01_1067__row581902568"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p58130175620">kerberosPrincipal</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p38120055618">Specifies the Kerberos authentication user, which is mandatory in security versions. This configuration is required only in security clusters.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p8815020567">flume_hbase</p>
</td>
</tr>
<tr id="mrs_01_1067__row1781804563"><td class="cellrowborder" valign="top" width="23.232323232323232%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.1 "><p id="mrs_01_1067__p148112025615">kerberosKeytab</p>
</td>
<td class="cellrowborder" valign="top" width="45.45454545454545%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.2 "><p id="mrs_01_1067__p7818013561">Specifies the file path for Kerberos authentication, which is mandatory in security versions. This configuration is required only in security clusters.</p>
</td>
<td class="cellrowborder" valign="top" width="31.313131313131308%" headers="mcps1.3.3.2.3.2.1.1.1.3.3.2.4.1.3 "><p id="mrs_01_1067__p128130145613">/opt/test/conf/user.keytab</p>
<div class="note" id="mrs_01_1067__note138180185616"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1067__p17812017567">Obtain the <strong id="mrs_01_1067__b87432864412">user.keytab</strong> file from the Kerberos certificate file of the user <strong id="mrs_01_1067__b97448811443">flume_hbase</strong>. In addition, ensure that the user who installs and runs the Flume client has the read and write permissions on the <strong id="mrs_01_1067__b274417814442">user.keytab</strong> file.</p>
</div></div>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="mrs_01_1067__l9f8138372d084391b32e65b61d2e0e15">Click <strong id="mrs_01_1067__b1663063092102913">Export</strong> to save the <strong id="mrs_01_1067__b929200590102913">properties.properties</strong> configuration file to the local.</li></ol>
</li><li id="mrs_01_1067__en-us_topic_0060185024_li14078610019">Log in to FusionInsight Manager and choose <strong id="mrs_01_1067__b105102059164119">Cluster</strong> &gt; <em id="mrs_01_1067__i1851085912412">Name of the desired cluster</em> &gt; <strong id="mrs_01_1067__b451165917411">Services</strong> &gt; <strong id="mrs_01_1067__b1051195944111">Flume</strong>. On the displayed page, click the <strong id="mrs_01_1067__b1051275914115">Flume</strong> role on the <strong id="mrs_01_1067__b784875214410">Instance</strong> tab page.</li><li id="mrs_01_1067__l392a376a371b437790b4da4acf925702">Select the Flume role of the node where the configuration file is to be uploaded, choose <strong id="mrs_01_1067__b473311884102913">Instance Configurations</strong> &gt; <strong id="mrs_01_1067__b829473976102913">Import</strong> beside the <span class="parmname" id="mrs_01_1067__parmname121404499102913"><b>flume.config.file</b></span>, and select the <strong id="mrs_01_1067__b1315800417102913">properties.properties</strong> file.<div class="note" id="mrs_01_1067__n3e86a79ea98c4b8089687840b273b550"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1067__u5e3247ed1e78493fbf9ccdd8e0bf3d32"><li class="text" id="mrs_01_1067__lcabeb48a4ba44391b9d04b5aeb49dca4">An independent server configuration file can be uploaded to each Flume instance.</li><li class="text" id="mrs_01_1067__la7dfd46f8f4b4bf49eefb168bab2437a">This step is required for updating the configuration file. Modifying the configuration file on the background is an improper operation because the modification will be overwritten after configuration synchronization.</li></ul>
</div></div>
</li><li id="mrs_01_1067__l73a389a5b02d491c88fc93afd56e5097">Click <strong id="mrs_01_1067__b304494598102913">Save</strong>, and then click <strong id="mrs_01_1067__b560591023102913">OK</strong>.</li><li id="mrs_01_1067__l3988a98c72b544dd920b09e5fd34f0d3">Click <strong id="mrs_01_1067__b1061807966102913">Finish</strong>.</li></ol>
</p></li></ol><ol start="4" id="mrs_01_1067__odb5e2d791a0747efb1f3f44b2d08ed9a"><li id="mrs_01_1067__lf76fc1368b4140ada0bb17eee8bda411"><span>Verify log transmission.</span><p><ol type="a" id="mrs_01_1067__o8b3f15b88a5c4a6ebef95bb45e004119"><li id="mrs_01_1067__lb6d1ac0f7d5c482ca9954db87bad5a8d">Go to the directory where the HBase client is installed.<p id="mrs_01_1067__a35822f359f0c46e890dd774867744ee1"><a name="mrs_01_1067__lb6d1ac0f7d5c482ca9954db87bad5a8d"></a><a name="lb6d1ac0f7d5c482ca9954db87bad5a8d"></a><strong id="mrs_01_1067__b1795406682102913">cd /</strong><em id="mrs_01_1067__i231826345102913">Client installation directory</em><strong id="mrs_01_1067__b2102163647102913">/ HBase/hbase</strong></p>
<p id="mrs_01_1067__acd1212b6a1a2426b860249c5b9411c38"><strong id="mrs_01_1067__b50351756102913">kinit flume_hbase</strong> (Enter the password.)</p>
</li><li id="mrs_01_1067__l91832aa79f8b4f8fb939dfa7799d186c">Run the <strong id="mrs_01_1067__a384f7b42dcf44af8a5ddb61f5679c824">hbase shell</strong> command to access the HBase client.</li><li id="mrs_01_1067__lc2d1d1c01562406e953143cb65dd94f4">Run the <strong id="mrs_01_1067__b13177143415344">scan 'flume_test'</strong> statement. Logs are written in the HBase column family by line.<pre class="screen" id="mrs_01_1067__s07c629dad14b46ee9d4238d6fa4c23d6">hbase(main):001:0&gt; scan 'flume_test'
ROW COLUMN+CELL
2017-09-18 16:05:36,394 INFO [hconnection-0x415a3f6a-shared--pool2-t1] ipc.AbstractRpcClient: RPC Server Kerberos principal name for service=ClientService is hbase/hadoop.&lt;<em id="mrs_01_1067__i69518547347">system domain name</em>&gt;@&lt;<em id="mrs_01_1067__i1488315016354">system domain name</em>&gt;
default4021ff4a-9339-4151-a4d0-00f20807e76d column=cf:pCol, timestamp=1505721909388, value=Welcome to flume
incRow column=cf:iCol, timestamp=1505721909461, value=\x00\x00\x00\x00\x00\x00\x00\x01
2 row(s) in 0.3660 seconds</pre>
</li></ol>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1059.html">Non-Encrypted Transmission</a></div>
</div>
</div>
View Git Blame Copy Permalink