forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
112 lines
16 KiB
HTML
112 lines
16 KiB
HTML
<a name="mrs_01_1064"></a><a name="mrs_01_1064"></a>
|
|
|
|
<h1 class="topictitle1">Typical Scenario: Collecting Local Dynamic Logs and Uploading Them to HDFS</h1>
|
|
<div id="body1590374513444"><div class="section" id="mrs_01_1064__sdda1342b18904e9da5e591abab7a4475"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1064__a5d2a5f497df244a3921856479ced46d8">This section describes how to use the Flume client to collect dynamic logs from a local host and save them to the <span class="filepath" id="mrs_01_1064__filepath8916518466"><b>/flume/test</b></span> directory on HDFS.</p>
|
|
<p id="mrs_01_1064__p1190153615238">This section applies to MRS 3.<em id="mrs_01_1064__i14138155718553">x</em> or later clusters.</p>
|
|
<div class="note" id="mrs_01_1064__n148d01d4dd75467785f82c085a257092"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p class="text" id="mrs_01_1064__a8073caae9f9f4797b11c7337b6233a1f">By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see <a href="mrs_01_1069.html">Configuring the Encrypted Transmission</a>. The configuration applies to scenarios where only the Flume is configured, for example, Taildir Source+Memory Channel+HDFS Sink.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1064__s06c379243c8c47bab0eeab33003fa60b"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1064__ub171efdeb6c44516964e0654d30a1175"><li id="mrs_01_1064__l50a104139b3b4d0186cc2bceee3f48c5">The cluster has been installed, including the HDFS and Flume services.</li><li id="mrs_01_1064__li3520154014471">The Flume client has been installed. For details, see <span id="mrs_01_1064__ph19119161518475"><a href="https://docs.otc.t-systems.com/cmpntguide/mrs/mrs_01_0392.html" target="_blank" rel="noopener noreferrer">Installing the Flume Client</a></span>.</li><li id="mrs_01_1064__le2f1b6b3956b4407b1369ba29e202188">The network environment of the cluster is secure.</li><li id="mrs_01_1064__l15430c7cde61469c9dabcce2bf7a26c5">You have created user <strong id="mrs_01_1064__b10124164511714">flume_hdfs</strong> and authorized the HDFS directory and data to be operated during log verification.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1064__sab6c60ede75b4ed0ae6e0527b599d4ab"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1064__ofc08111386ec47d38a94ece151ea677f"><li id="mrs_01_1064__ld7ad919a407842eb8af3e715fde0dde6"><span>On FusionInsight Manager, choose <span class="menucascade" id="mrs_01_1064__menucascade129541557205219"><b><span class="uicontrol" id="mrs_01_1064__uicontrol1095435785218">System > User</span></b></span> and choose <span class="menucascade" id="mrs_01_1064__menucascade5955205755217"><b><span class="uicontrol" id="mrs_01_1064__uicontrol09542573525">More > Download Authentication Credential</span></b></span> to download the Kerberos certificate file of user <strong id="mrs_01_1064__b15955175719524">flume_hdfs</strong> and save it to the local host.</span></li><li id="mrs_01_1064__l448445cf61eb40d9ac54153394022349"><span>Set Flume parameters.</span><p><div class="p" id="mrs_01_1064__p432301919148">Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.<ol type="a" id="mrs_01_1064__o34f76daf400f47bc9f3add2f4c9a7b12"><li id="mrs_01_1064__l70d02c82c067447ca218d049d9c39c38">Log in to FusionInsight Manager and choose<strong id="mrs_01_1064__b168970384395"> Cluster</strong> > <strong id="mrs_01_1064__b189873843912">Services</strong>. On the page that is displayed, choose <strong id="mrs_01_1064__b19898143817397">Flume</strong>. On the displayed page, click the <strong id="mrs_01_1064__b1689953843913">Configuration Tool</strong> tab.</li><li id="mrs_01_1064__l209bc5e7a8ac464e9973353eff1aa633">Set <strong id="mrs_01_1064__b184525911354732">Agent Name</strong> to <strong id="mrs_01_1064__b158534629854732">client</strong>. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.<p id="mrs_01_1064__a1ddd4b509c834d00bedb7780be011035">Use Taildir Source, Memory Channel, and HDFS Sink.</p>
|
|
</li><li id="mrs_01_1064__l4efe027192324e5a9cac196947276131">Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to <a href="#mrs_01_1064__table275562484">Table 1</a> based on the actual environment.<div class="note" id="mrs_01_1064__n6e73ff484a674be3a1edcd17332110f2"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1064__u35f2274086a343d09cf29cb7d255c5dd"><li id="mrs_01_1064__l354eb65124f84e48a3816157e353b9c2">If you want to continue using the <strong id="mrs_01_1064__b13211191864819">properties.propretites</strong> file by modifying it, log in to FusionInsight Manager, choose <strong id="mrs_01_1064__b1421291812483">Cluster</strong> > <em id="mrs_01_1064__i19212141812483">Name of the desired cluster</em> > <strong id="mrs_01_1064__b4212111819481">Services</strong>. On the page that is displayed, choose <strong id="mrs_01_1064__b2021351814486">Flume</strong>. On the displayed page, click the <strong id="mrs_01_1064__b321331816482">Configuration Tool</strong> tab, click <strong id="mrs_01_1064__b521371844814">Import</strong>, import the file, and modify the configuration items related to non-encrypted transmission.</li><li id="mrs_01_1064__lb73d10dbd6aa43b098b5fe71b3b01518">It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.</li></ul>
|
|
</div></div>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1064__table275562484"></a><a name="table275562484"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1064__table275562484" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters to be modified for the Flume role client</caption><thead align="left"><tr id="mrs_01_1064__row177105611489"><th align="left" class="cellrowborder" valign="top" width="24.13%" id="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1"><p id="mrs_01_1064__p2718561486">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="38.81%" id="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2"><p id="mrs_01_1064__p6718564488">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="37.059999999999995%" id="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3"><p id="mrs_01_1064__p1371956144818">Example Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1064__row107165615485"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p1976569487">Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p0715560489">The value must be unique and cannot be left blank.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p47175634810">test</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1064__row37056104812"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p177165654817">filegroups</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p378568488">Specifies the file group list name. This parameter cannot be left blank. The value contains the following two parts:</p>
|
|
<ul id="mrs_01_1064__ul1771561487"><li id="mrs_01_1064__li157145611481"><strong id="mrs_01_1064__b193038341418">Name</strong>: name of the file group list.</li><li id="mrs_01_1064__li97856154816"><strong id="mrs_01_1064__b159784813115">filegroups</strong>: absolute path of dynamic log files.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p17845644811">-</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1064__row19812561484"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p981756184811">positionFile</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p148205684812">Specifies the location where the collected file information (file name and location from which the file collected) is saved. This parameter cannot be left blank. The file does not need to be created manually, but the Flume running user needs to have the write permission on its upper-level directory.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p2816564487">/home/omm/flume/positionfile</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1064__row138115618481"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p128125644815">batchSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p1687568482">Specifies the number of events that Flume sends in a batch.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p17885684818">61200</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1064__row3855604816"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p178956164819">hdfs.path</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p1883566483">Specifies the HDFS data write directory. This parameter cannot be left blank.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p188105674811">hdfs://hacluster/flume/test</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1064__row78115619483"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p128125617487">hdfs.filePrefix</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p5835694815">Specifies the file name prefix after data is written to HDFS.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p188145618482">TMP_</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1064__row10817567481"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p19817562483">hdfs.batchSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p1488560489">Specifies the maximum number of events that can be written to HDFS once.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p15895654816">61200</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1064__row28155684810"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p16811560482">hdfs.kerberosPrincipal</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p108125654811">Specifies the Kerberos authentication user, which is mandatory in security versions. This configuration is required only in security clusters.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p58115615487">flume_hdfs</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1064__row1581256194813"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p1085568481">hdfs.kerberosKeytab</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p1384566481">Specifies the keytab file path for Kerberos authentication, which is mandatory in security versions. This configuration is required only in security clusters.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p6815614481">/opt/test/conf/user.keytab</p>
|
|
<div class="note" id="mrs_01_1064__note2935654814"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1064__p29456154814">Obtain the <strong id="mrs_01_1064__b109342336354732">user.keytab</strong> file from the Kerberos certificate file of the user <strong id="mrs_01_1064__b43325796654732">flume_hdfs</strong>. In addition, ensure that the user who installs and runs the Flume client has the read and write permissions on the <strong id="mrs_01_1064__b58107808154732">user.keytab</strong> file.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1064__row4905619489"><td class="cellrowborder" valign="top" width="24.13%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.1 "><p id="mrs_01_1064__p119125684819">hdfs.useLocalTimeStamp</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.81%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.2 "><p id="mrs_01_1064__p9985613482">Specifies whether to use the local time. Possible values are <strong id="mrs_01_1064__b80873488154732">true</strong> and <strong id="mrs_01_1064__b118495203354732">false</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.059999999999995%" headers="mcps1.3.3.2.2.2.1.1.3.3.2.4.1.3 "><p id="mrs_01_1064__p6985674813">true</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_1064__l78938a30f82d4a5283b7c4aaa1bb79b1"><a name="mrs_01_1064__l78938a30f82d4a5283b7c4aaa1bb79b1"></a><a name="l78938a30f82d4a5283b7c4aaa1bb79b1"></a>Click <strong id="mrs_01_1064__b141082691454732">Export</strong> to save the <strong id="mrs_01_1064__b15681578154732">properties.properties</strong> configuration file to the local.</li></ol>
|
|
</div>
|
|
</p></li><li id="mrs_01_1064__li641934318148"><span>Upload the configuration file.</span><p><p id="mrs_01_1064__p299714411415">Upload the file exported in <a href="#mrs_01_1064__l78938a30f82d4a5283b7c4aaa1bb79b1">2.d</a> to the <em id="mrs_01_1064__i1474731720472">Flume client installation directory</em><strong id="mrs_01_1064__b12748121714472">/fusioninsight-flume-</strong><span id="mrs_01_1064__text574921784710"><em id="mrs_01_1064__i19749617154719">Flume component version number</em></span><strong id="mrs_01_1064__b474911717479">/conf</strong> directory of the cluster.</p>
|
|
</p></li></ol><ol start="4" id="mrs_01_1064__o459d32d154134b2283f1310d032ba9e0"><li id="mrs_01_1064__l606003fd5fee4e08a4d702c49e1ada56"><span>Verify log transmission.</span><p><ol type="a" id="mrs_01_1064__o4210a083bc3d4707aa6f740325e6e2d8"><li id="mrs_01_1064__l12f48276ae8d42f491602b3838d87713">Log in to FusionInsight Manager as a user who has the management permission on HDFS. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager (MRS 3.x or Later)</a>. Choose <strong id="mrs_01_1064__b36412264482">Cluster</strong> > <strong id="mrs_01_1064__b26572613483">Services</strong> > <strong id="mrs_01_1064__b10651526104816">HDFS</strong>. On the page that is displayed, click the <strong id="mrs_01_1064__b87701819457">NameNode(</strong><em id="mrs_01_1064__i0306727558">Node name</em><strong id="mrs_01_1064__b57405234511">,Active)</strong> link next to <strong id="mrs_01_1064__b66512674815">NameNode WebUI</strong> to go to the HDFS web UI. On the displayed page, choose <strong id="mrs_01_1064__b166614267480">Utilities</strong> > <strong id="mrs_01_1064__b666182614486">Browse the file system</strong>.</li><li id="mrs_01_1064__l8f465fb880d04703ab7b31d5349e6528">Check whether the data is generated in the <strong id="mrs_01_1064__b173985711654732">/flume/test</strong> directory on the HDFS.</li></ol>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1059.html">Non-Encrypted Transmission</a></div>
|
|
</div>
|
|
</div>
|
|
|