forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
166 lines
38 KiB
HTML
166 lines
38 KiB
HTML
<a name="mrs_01_0397"></a><a name="mrs_01_0397"></a>
|
|
|
|
<h1 class="topictitle1">Using Flume from Scratch</h1>
|
|
<div id="body1589421630526"><div class="section" id="mrs_01_0397__s6020f8e1de5644d2becca6a1c9dd7b98"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0397__a78492c95b3d84580991937af4802369a">You can use Flume to import collected log information to Kafka.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0397__sdd14a34b7dc44c2ab9cabb19599a033a"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_0397__u0c52e5bb70f64537997c99cf4de32541"><li id="mrs_01_0397__l11118dd36a974dfc8762539ad916ddf6">A streaming cluster that contains components such as Flume and Kafka and has Kerberos authentication enabled has been created. </li><li id="mrs_01_0397__l9d613f4f7c21477aa8e688973b1ad35c">The streaming cluster can properly communicate with the node where logs are generated.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_0397__s1f8b9b2a99104941860eb2956cc0057c"><h4 class="sectiontitle">Using the Flume Client (Versions Earlier Than MRS 3.x)</h4><div class="note" id="mrs_01_0397__n51bce0a8f0e64a5a8a5ffbf77630e44f"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0397__a42e0004dfc9441148c19537b580a2de8">You do not need to perform <a href="#mrs_01_0397__l78730912572649fd8edfda3920dc20cf">2</a> to <a href="#mrs_01_0397__lfde322e0f3de4ccb88b4e195e65f9993">6</a> for a normal cluster.</p>
|
|
</div></div>
|
|
<ol id="mrs_01_0397__o403b9d3871a94c49802642eea27749af"><li id="mrs_01_0397__li26835010457"><span>Install the Flume client.</span><p><p id="mrs_01_0397__p158895102454">Install the Flume client in a directory, for example, <strong id="mrs_01_0397__b49001343596">/opt/Flumeclient</strong>, on the node where logs are generated by referring to <a href="mrs_01_1594.html">Installing the Flume Client on Clusters of Versions Earlier Than MRS 3.x</a>. The Flume client installation directories in the following steps are only examples. Change them to the actual installation directories.</p>
|
|
</p></li><li id="mrs_01_0397__l78730912572649fd8edfda3920dc20cf"><a name="mrs_01_0397__l78730912572649fd8edfda3920dc20cf"></a><a name="l78730912572649fd8edfda3920dc20cf"></a><span>Copy the configuration file of the authentication server from the Master1 node to the <em id="mrs_01_0397__i97131237193316">Flume client installation directory</em><strong id="mrs_01_0397__b175084422334">/fusioninsight-flume-</strong><em id="mrs_01_0397__i176931013163418">Flume component version number</em><strong id="mrs_01_0397__b2541256103314">/conf</strong> directory on the node where the Flume client is installed.</span><p><p id="mrs_01_0397__p798051210138">For versions earlier than MRS 1.9.2, <strong id="mrs_01_0397__b588414271117"><span id="mrs_01_0397__text545174019335">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b7884152121113">/FusionInsight/etc/1_</strong><em id="mrs_01_0397__i13151154893315">X</em><strong id="mrs_01_0397__b1584016781114">_KerberosClient/kdc.conf</strong> is used as the full file path.</p>
|
|
<p id="mrs_01_0397__p13504151551317">For versions earlier than MRS 3.<em id="mrs_01_0397__i153056356139">x</em>, <strong id="mrs_01_0397__b219491819130"><span id="mrs_01_0397__text1422165393315">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b11941618171310">/MRS_Current/1_</strong><em id="mrs_01_0397__i1125517567332">X</em><strong id="mrs_01_0397__b1989519211130">_KerberosClient/etc/kdc.conf</strong> is used as the full file path.</p>
|
|
<p id="mrs_01_0397__p1812741181214">In the preceding paths, <span class="parmname" id="mrs_01_0397__parmname4993101192116"><b>X</b></span> indicates a random number. Change it based on the site requirements. The file must be saved by the user who installs the Flume client, for example, user <strong id="mrs_01_0397__b107925408536">root</strong>.</p>
|
|
</p></li><li id="mrs_01_0397__li13806122116487"><span>Check the service IP address of any node where the Flume role is deployed.</span><p><ul id="mrs_01_0397__ul1692910323486"><li id="mrs_01_0397__li792916322486">For versions earlier than MRS 1.9.2, log in to MRS Manager. Choose <strong id="mrs_01_0397__b1462164445414">Cluster</strong> > <strong id="mrs_01_0397__b234136163719">Services</strong> > <strong id="mrs_01_0397__b175516385377">Flume</strong> > <strong id="mrs_01_0397__b688214015372">Instance</strong>. Query <span class="parmname" id="mrs_01_0397__parmname172398430334"><b>Service IP Address</b></span> of any node on which the Flume role is deployed.</li><li id="mrs_01_0397__li979413816488">For MRS 1.9.2 to versions earlier than 3.x, click the cluster name on the MRS console and choose <em id="mrs_01_0397__i1311325145810">Name of the desired cluster</em> > <strong id="mrs_01_0397__b1481185933910">Components</strong> > <strong id="mrs_01_0397__b101901577404">Flume</strong> > <strong id="mrs_01_0397__b122475984012">Instances</strong> to view <span class="parmname" id="mrs_01_0397__parmname86251247123318"><b>Business IP Address</b></span> of any node where the Flume role is deployed.</li></ul>
|
|
</p></li><li id="mrs_01_0397__l762ab29694a642ac8ae1a0609cb97c9b"><a name="mrs_01_0397__l762ab29694a642ac8ae1a0609cb97c9b"></a><a name="l762ab29694a642ac8ae1a0609cb97c9b"></a><span>Copy the user authentication file from this node to the <em id="mrs_01_0397__i893801515338">Flume client installation directory</em><span class="filepath" id="mrs_01_0397__fd57a94849eb24f3790e5f19a860fd212"><b>/fusioninsight-flume-Flume component version number/conf</b></span> directory on the Flume client node.</span><p><p id="mrs_01_0397__p18616514193">For versions earlier than MRS 1.9.2, <strong id="mrs_01_0397__b7986557112411"><span id="mrs_01_0397__text836216620348">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b6986175712413">/FusionInsight/FusionInsight-Flume-</strong><span id="mrs_01_0397__text93623614346"><em id="mrs_01_0397__i17981114203410">Flume component version number</em></span><strong id="mrs_01_0397__b66738210254">/flume/conf/flume.keytab</strong> is used as the full file path.</p>
|
|
<p id="mrs_01_0397__p843725211914">For versions earlier than 3.<em id="mrs_01_0397__i153481102329">x</em>, <strong id="mrs_01_0397__b1051091853213"><span id="mrs_01_0397__text1960713188348">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b651010185322">/MRS_</strong><em id="mrs_01_0397__i2078312693411">XXX</em><strong id="mrs_01_0397__b1247202611324">/install/FusionInsight-Flume-</strong><span id="mrs_01_0397__text76073181349"><em id="mrs_01_0397__i727802313345">Flume component version number</em></span><strong id="mrs_01_0397__b1860144773215">/flume/conf/flume.keytab</strong> is used as the full file path.</p>
|
|
<p id="mrs_01_0397__p77268218240">In the preceding paths, <span class="parmname" id="mrs_01_0397__parmname141931219135214"><b>XXX</b></span> indicates the product version number. Change it based on the site requirements. The file must be saved by the user who installs the Flume client, for example, user <strong id="mrs_01_0397__b207261321122411">root</strong>.</p>
|
|
</p></li><li id="mrs_01_0397__l904b0591afae407d8ead20c06ca6bda6"><span>Copy the <span class="filepath" id="mrs_01_0397__f1372ed26dd3e444e93bf81f269454e4e"><b>jaas.conf</b></span> file from this node to the <span class="filepath" id="mrs_01_0397__f990b8fe4e45d42a8bdeaab468ad1fd3f"><b>conf</b></span> directory on the Flume client node.</span><p><p id="mrs_01_0397__p444152518249">For versions earlier than MRS 1.9.2, <strong id="mrs_01_0397__b734314150389"><span id="mrs_01_0397__text2058885819348">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b3343115103811">/FusionInsight/etc/1_</strong><em id="mrs_01_0397__i2012516583517">X</em><strong id="mrs_01_0397__b996162119385">_Flume/jaas.conf</strong> is used as the full file path.</p>
|
|
<p id="mrs_01_0397__p1925513263245">For versions earlier than MRS 3.<em id="mrs_01_0397__i12691353391">x</em>, <strong id="mrs_01_0397__b1070245153910"><span id="mrs_01_0397__text19700175133913">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b470511511390">/MRS_Current/1_</strong><em id="mrs_01_0397__i1970835133914">X</em><strong id="mrs_01_0397__b157109513910">_Flume/etc/jaas.conf</strong> is used as the full file path.</p>
|
|
<p id="mrs_01_0397__p1096823443317">In the preceding path, <span class="parmname" id="mrs_01_0397__parmname164049192911"><b>X</b></span> indicates a random number. Change it based on the site requirements. The file must be saved by the user who installs the Flume client, for example, user <strong id="mrs_01_0397__b296819349335">root</strong>.</p>
|
|
</p></li><li id="mrs_01_0397__lfde322e0f3de4ccb88b4e195e65f9993"><a name="mrs_01_0397__lfde322e0f3de4ccb88b4e195e65f9993"></a><a name="lfde322e0f3de4ccb88b4e195e65f9993"></a><span>Log in to the Flume client node and go to the client installation directory. Run the following command to modify the file:</span><p><p id="mrs_01_0397__a08785a99d6a642b0969f624dcdfdac7b"><strong id="mrs_01_0397__en-us_topic_0068225666_b93221911938">vi conf/jaas.conf</strong></p>
|
|
<p id="mrs_01_0397__en-us_topic_0068225666_p740524175416">Change the full path of the user authentication file defined by <span class="parmname" id="mrs_01_0397__parmname5801144914535"><b>keyTab</b></span> to the <span class="filepath" id="mrs_01_0397__f05179cbd1d92436caebf1dd7cbbbb880"><b>Flume client installation directory/fusioninsight-flume-<span id="mrs_01_0397__text225163401610"><em id="mrs_01_0397__i5192162033611">Flume component version number</em></span>/conf</b></span> saved in <a href="#mrs_01_0397__l762ab29694a642ac8ae1a0609cb97c9b">4</a>, and save the modification and exit.</p>
|
|
</p></li><li id="mrs_01_0397__l0ef61845752d459d8e290359685d2f46"><span>Run the following command to modify the <span class="filepath" id="mrs_01_0397__fa9d02736a2e44d1394f691094b812c9a"><b>flume-env.sh</b></span> configuration file of the Flume client:</span><p><p id="mrs_01_0397__a8266d015f05049919264b780fd27fde8"><strong id="mrs_01_0397__b10958923637">vi </strong><em id="mrs_01_0397__a37dfff98f13b4d979e493a2af00db384">Flume client installation directory</em><strong id="mrs_01_0397__b122612372314">/fusioninsight-flume-</strong><span id="mrs_01_0397__text17823182845418"><em id="mrs_01_0397__i1882382855411">Flume component version number</em></span><strong id="mrs_01_0397__b22621375318">/conf/flume-env.sh</strong></p>
|
|
<p id="mrs_01_0397__ae09af85b2073488da26ab3ecffeffa48">Add the following information after <span class="parmvalue" id="mrs_01_0397__pc6f27cf3d9e34ebd951a43cda607a8c3"><b>-XX:+UseCMSCompactAtFullCollection</b></span>:</p>
|
|
<pre class="screen" id="mrs_01_0397__s587f8c2bcaff474998bbcfe709e906ff">-Djava.security.krb5.conf=<em id="mrs_01_0397__aafbc984f516e4664a9e15d31af5f8ff3">Flume client installation directory</em>/fusioninsight-flume-1.9.0/conf/kdc.conf -Djava.security.auth.login.config=<em id="mrs_01_0397__a3c39b6e7e6a14210bbff946d68d2202a">Flume client installation directory</em>/fusioninsight-flume-1.9.0/conf/jaas.conf -Dzookeeper.request.timeout=120000</pre>
|
|
<p id="mrs_01_0397__en-us_topic_0068225666_p14675223454">Example: <strong id="mrs_01_0397__b6326917612">"-XX:+UseCMSCompactAtFullCollection -Djava.security.krb5.conf=</strong><strong id="mrs_01_0397__b3565464297">/opt/FlumeClient</strong>/<strong id="mrs_01_0397__b18439814578">fusioninsight-flume</strong><strong id="mrs_01_0397__b181441710716">-</strong><span id="mrs_01_0397__text1235696618"><em id="mrs_01_0397__i435119669">Flume component version number</em></span><strong id="mrs_01_0397__b694212198619">/conf/kdc.conf -Djava.security.auth.login.config=</strong><strong id="mrs_01_0397__b11777162342911">/opt/FlumeClient</strong>/<strong id="mrs_01_0397__b1822017281177">fusioninsight-flume-</strong><span id="mrs_01_0397__text994511197614"><em id="mrs_01_0397__i894519192610">Flume component version number</em></span><strong id="mrs_01_0397__b109439198616">/conf/jaas.conf -Dzookeeper.request.timeout=120000"</strong></p>
|
|
<p id="mrs_01_0397__a57a21d9b8dce48ea893ac82ff4fa88b7">Change <em id="mrs_01_0397__i292774315221">Flume client installation directory</em> to the actual installation directory. Then save and exit.</p>
|
|
</p></li><li id="mrs_01_0397__l13c4cf9e0db44bbb9c55a8ba627f5ed9"><span>Run the following command to restart the Flume client:</span><p><p id="mrs_01_0397__p1836219562278"><strong id="mrs_01_0397__b2657112412812">cd </strong><em id="mrs_01_0397__i126601224182820">Flume client installation directory</em><strong id="mrs_01_0397__b165842416282">/fusioninsight-flume-</strong><span id="mrs_01_0397__text1936318563277"><em id="mrs_01_0397__i15363135662720">Flume component version number</em></span><strong id="mrs_01_0397__b536315618279">/bin</strong></p>
|
|
<p id="mrs_01_0397__p143635560272"><strong id="mrs_01_0397__b8363656162711">./flume-manage.sh restart</strong></p>
|
|
<p id="mrs_01_0397__p18263122932816">Example:</p>
|
|
<p id="mrs_01_0397__a53e631b959614158b597a7afb6753690"><strong id="mrs_01_0397__b150968419">cd /opt/FlumeClient/fusioninsight-flume-</strong><span id="mrs_01_0397__text11432135563"><em id="mrs_01_0397__i0143113205619">Flume component version number</em></span><strong id="mrs_01_0397__b1150069411">/bin</strong></p>
|
|
<p id="mrs_01_0397__a81d84cf9bb9848179d102ccd4e7529c1"><strong id="mrs_01_0397__en-us_topic_0068225666_b239063391760">./flume-manage.sh restart</strong></p>
|
|
</p></li><li id="mrs_01_0397__lc7a2186ec18f4972831e99c62a482053"><span>Run the following command to configure and save jobs in the Flume client configuration file <strong id="mrs_01_0397__b16282491187">properties.properties</strong> based on service requirements.</span><p><p id="mrs_01_0397__a157d7a9642074d83ba69453d1cb26be5"><strong id="mrs_01_0397__b189281212541">vi </strong><em id="mrs_01_0397__i7438151216419">Flume client installation directory</em><strong id="mrs_01_0397__b593816164410">/fusioninsight-flume-</strong><span id="mrs_01_0397__text517113302560"><em id="mrs_01_0397__i1171173045610">Flume component version number</em></span><strong id="mrs_01_0397__b17938181610416">/conf/properties.properties</strong></p>
|
|
<p id="mrs_01_0397__en-us_topic_0068225666_p515527111833">The following uses SpoolDir Source+File Channel+Kafka Sink as an example:</p>
|
|
<pre class="screen" id="mrs_01_0397__sb4388416d3684139aceb897919c52981">#########################################################################################
|
|
client.sources = static_log_source
|
|
client.channels = static_log_channel
|
|
client.sinks = kafka_sink
|
|
#########################################################################################
|
|
#LOG_TO_HDFS_ONLINE_1
|
|
|
|
client.sources.static_log_source.type = spooldir
|
|
client.sources.static_log_source.spoolDir = <em id="mrs_01_0397__i95132322546">Monitoring directory</em>
|
|
client.sources.static_log_source.fileSuffix = .COMPLETED
|
|
client.sources.static_log_source.ignorePattern = ^$
|
|
client.sources.static_log_source.trackerDir = <em id="mrs_01_0397__i4205253185418">Metadata storage path during transmission</em>
|
|
client.sources.static_log_source.maxBlobLength = 16384
|
|
client.sources.static_log_source.batchSize = 51200
|
|
client.sources.static_log_source.inputCharset = UTF-8
|
|
client.sources.static_log_source.deserializer = LINE
|
|
client.sources.static_log_source.selector.type = replicating
|
|
client.sources.static_log_source.fileHeaderKey = file
|
|
client.sources.static_log_source.fileHeader = false
|
|
client.sources.static_log_source.basenameHeader = true
|
|
client.sources.static_log_source.basenameHeaderKey = basename
|
|
client.sources.static_log_source.deletePolicy = never
|
|
|
|
client.channels.static_log_channel.type = file
|
|
client.channels.static_log_channel.dataDirs = <em id="mrs_01_0397__i0229156115712">Data cache path. Multiple paths, separated by commas (,), can be configured to improve performance.</em>
|
|
client.channels.static_log_channel.checkpointDir = <em id="mrs_01_0397__i5122193611574">Checkpoint storage path</em>
|
|
client.channels.static_log_channel.maxFileSize = 2146435071
|
|
client.channels.static_log_channel.capacity = 1000000
|
|
client.channels.static_log_channel.transactionCapacity = 612000
|
|
client.channels.static_log_channel.minimumRequiredSpace = 524288000
|
|
|
|
client.sinks.kafka_sink.type = org.apache.flume.sink.kafka.KafkaSink
|
|
client.sinks.kafka_sink.kafka.topic = <em id="mrs_01_0397__i19320715905">Topic to which data is written, for example, <strong id="mrs_01_0397__b45110318547">flume_test</strong></em>
|
|
client.sinks.kafka_sink.kafka.bootstrap.servers = <em id="mrs_01_0397__aa6421766a92846b0910f8d4f7d558b61">XXX</em>.<em id="mrs_01_0397__a89f61f3d2b9a4da9b90d9e1db09f256f">XXX</em>.<em id="mrs_01_0397__a03a82f24d8924770a579bbfdb5577bf5">XXX</em>.<em id="mrs_01_0397__en-us_topic_0068225666_i450889851851">XXX</em>:<em id="mrs_01_0397__i173112511215">Kafka port number</em>,<em id="mrs_01_0397__en-us_topic_0068225666_i73457951853">XXX</em>.<em id="mrs_01_0397__en-us_topic_0068225666_i58491551854">XXX</em>.<em id="mrs_01_0397__en-us_topic_0068225666_i404351741856">XXX</em>.<em id="mrs_01_0397__en-us_topic_0068225666_i365603841857">XXX</em>:<em id="mrs_01_0397__i5338103912310">Kafka port number</em>,<em id="mrs_01_0397__en-us_topic_0068225666_i285304001859">XXX</em>.<em id="mrs_01_0397__a4566267c4494465c8bb36319c67a35cd">XXX</em>.<em id="mrs_01_0397__a73227fb5a90a4657a3dfb64f127ab301">XXX</em>.<em id="mrs_01_0397__a58fbb1bf1b5f43cf8a80978edcc33a98">XXX</em>:<em id="mrs_01_0397__i121898421839">Kafka port number</em>
|
|
client.sinks.kafka_sink.flumeBatchSize = 1000
|
|
client.sinks.kafka_sink.kafka.producer.type = sync
|
|
client.sinks.kafka_sink.kafka.security.protocol = SASL_PLAINTEXT
|
|
client.sinks.kafka_sink.kafka.kerberos.domain.name = <em id="mrs_01_0397__i11557195517119">Kafka domain name. This parameter is mandatory for a security cluster, for example, <strong id="mrs_01_0397__b048853675419">hadoop.xxx.com</strong>.</em>
|
|
client.sinks.kafka_sink.requiredAcks = 0
|
|
|
|
client.sources.static_log_source.channels = static_log_channel
|
|
client.sinks.kafka_sink.channel = static_log_channel</pre>
|
|
<div class="note" id="mrs_01_0397__note1677181812420"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0397__ul1636841220520"><li id="mrs_01_0397__li1422216201058"><strong id="mrs_01_0397__b204031210553">client.sinks.kafka_sink.kafka.topic</strong>: Topic to which data is written. If the topic does not exist in Kafka, it is automatically created by default.</li><li id="mrs_01_0397__li1622252012515"><strong id="mrs_01_0397__b0453714135618">client.sinks.kafka_sink.kafka.bootstrap.servers</strong>: List of Kafka Brokers, which are separated by commas (,). By default, the port is <strong id="mrs_01_0397__b25362277573">21007</strong> for a security cluster and <strong id="mrs_01_0397__b682615221207">9092</strong> for a normal cluster.</li><li id="mrs_01_0397__li11351941470"><strong id="mrs_01_0397__b105081204114">client.sinks.kafka_sink.kafka.security.protocol</strong>: The value is <strong id="mrs_01_0397__b1786715618119">SASL_PLAINTEXT</strong> for a security cluster and <strong id="mrs_01_0397__b10247638328">PLAINTEXT</strong> for a normal cluster.</li><li id="mrs_01_0397__li76061315459"><strong id="mrs_01_0397__b191071347230">client.sinks.kafka_sink.kafka.kerberos.domain.name</strong>:<p id="mrs_01_0397__p18606181511517">You do not need to set this parameter for a normal cluster. For a security cluster, the value of this parameter is the value of <strong id="mrs_01_0397__b17733903215100">kerberos.domain.name</strong> in the Kafka cluster.</p>
|
|
<p id="mrs_01_0397__p17607181513519">For versions earlier than MRS 1.9.2, obtain the value by checking <strong id="mrs_01_0397__b164681535101010"><span id="mrs_01_0397__text171381316762">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b13468193515101">/FusionInsight/etc/1_</strong><em id="mrs_01_0397__i45218191761">X</em><strong id="mrs_01_0397__b155161441181017">_Broker/server.properties</strong> on the node where the broker instance resides.</p>
|
|
<p id="mrs_01_0397__p106071815155">Obtain the value for versions earlier than MRS 3.<em id="mrs_01_0397__i15797558139">x</em> by checking <strong id="mrs_01_0397__b4602550161215"><span id="mrs_01_0397__text10571451969">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b17602050191211">/MRS_Current/1_</strong><em id="mrs_01_0397__i141400121964">X</em><strong id="mrs_01_0397__b131964558129">_Broker/etc/server.properties</strong> on the node where the broker instance resides.</p>
|
|
<p id="mrs_01_0397__p5607815252">In the preceding paths, <span class="parmname" id="mrs_01_0397__parmname9533492955100"><b>X</b></span> indicates a random number. Change it based on site requirements. The file must be saved by the user who installs the Flume client, for example, user <strong id="mrs_01_0397__b1639875518">root</strong>.</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_0397__l218366db914b4699896a7e73c483d1f1"><span>After the parameters are set and saved, the Flume client automatically loads the content configured in <strong id="mrs_01_0397__b18263314271">properties.properties</strong>. When new log files are generated by spoolDir, the files are sent to Kafka producers and can be consumed by Kafka consumers. </span></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0397__section197021472420"><h4 class="sectiontitle">Using the Flume Client (MRS 3.x or Later)</h4><div class="note" id="mrs_01_0397__note6127174918413"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0397__p1012744917411">You do not need to perform <a href="#mrs_01_0397__li81278495417">2</a> to <a href="#mrs_01_0397__li31329494415">6</a> for a normal cluster.</p>
|
|
</div></div>
|
|
<ol id="mrs_01_0397__ol1712704919420"><li id="mrs_01_0397__li297651112467"><span>Install the Flume client.</span><p><p id="mrs_01_0397__p1897616111466">Install the Flume client in a directory, for example, <strong id="mrs_01_0397__b1056192312261">/opt/Flumeclient</strong>, on the node where logs are generated by referring to <a href="mrs_01_1595.html">Installing the Flume Client on MRS 3.x or Later Clusters</a>. The Flume client installation directories in the following steps are only examples. Change them to the actual installation directories.</p>
|
|
</p></li><li id="mrs_01_0397__li81278495417"><a name="mrs_01_0397__li81278495417"></a><a name="li81278495417"></a><span>Copy the configuration file of the authentication server from the Master1 node to the <em id="mrs_01_0397__i467011272592">Flume client installation directory</em><strong id="mrs_01_0397__b1455144116599">/fusioninsight-flume-</strong><span id="mrs_01_0397__text47312414715"><em id="mrs_01_0397__i7238195525911">Flume component version number</em></span><strong id="mrs_01_0397__b146484519020">/conf</strong> directory on the node where the Flume client is installed.</span><p><p id="mrs_01_0397__p1912818491043">The full file path is <strong id="mrs_01_0397__b31803913312">${BIGDATA_HOME}/FusionInsight_</strong><span id="mrs_01_0397__text361035191118"><strong id="mrs_01_0397__b718112915333">BASE_</strong><em id="mrs_01_0397__i163311813193115">XXX</em></span><strong id="mrs_01_0397__b5187419193315">/1_</strong><span id="mrs_01_0397__text1036612458314"><em id="mrs_01_0397__i13462124914318">X</em></span><strong id="mrs_01_0397__b04131232333">_KerberosClient/etc/kdc.conf</strong>. In the preceding path, <span class="parmname" id="mrs_01_0397__parmname1575274511338"><b>XXX</b></span> indicates the product version number. <span class="parmname" id="mrs_01_0397__parmname7755164583313"><b>X</b></span> indicates a random number. Replace them based on site requirements. The file must be saved by the user who installs the Flume client, for example, user <strong id="mrs_01_0397__b333209018">root</strong>.</p>
|
|
</p></li><li id="mrs_01_0397__li6128114911418"><span>Check the service IP address of any node where the Flume role is deployed.</span><p><p id="mrs_01_0397__p109301819113114">Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager (MRS 3.x or Later)</a>. Choose <strong id="mrs_01_0397__b2032411289519">Cluster > Services > Flume > Instance</strong>. Check the service IP address of any node where the Flume role is deployed.</p>
|
|
</p></li><li id="mrs_01_0397__li4130849748"><a name="mrs_01_0397__li4130849748"></a><a name="li4130849748"></a><span>Copy the user authentication file from this node to the <em id="mrs_01_0397__i1630391498">Flume client installation directory</em><span class="filepath" id="mrs_01_0397__filepath1473138144"><b>/fusioninsight-flume-Flume component version number/conf</b></span> directory on the Flume client node.</span><p><p id="mrs_01_0397__p15131104920419">The full file path is <strong id="mrs_01_0397__b1261520361350">${BIGDATA_HOME}/FusionInsight_Porter_</strong><em id="mrs_01_0397__i1435635293516">XXX</em><strong id="mrs_01_0397__b15446104713515">/install/FusionInsight-Flume-</strong><em id="mrs_01_0397__i943315023618">Flume component version number</em><strong id="mrs_01_0397__b122147103611">/flume/conf/flume.keytab</strong>.</p>
|
|
<p id="mrs_01_0397__p101311749743">In the preceding paths, <span class="parmname" id="mrs_01_0397__parmname472768707"><b>XXX</b></span> indicates the product version number. Change it based on the site requirements. The file must be saved by the user who installs the Flume client, for example, user <strong id="mrs_01_0397__b1227937073">root</strong>.</p>
|
|
</p></li><li id="mrs_01_0397__li313120496411"><span>Copy the <span class="filepath" id="mrs_01_0397__filepath1259082109"><b>jaas.conf</b></span> file from this node to the <span class="filepath" id="mrs_01_0397__filepath570760916"><b>conf</b></span> directory on the Flume client node.</span><p><p id="mrs_01_0397__p108276123317">The full file path is <strong id="mrs_01_0397__b587410810371">${BIGDATA_HOME}/FusionInsight_Current/1_</strong><em id="mrs_01_0397__i148271319183720">X</em><strong id="mrs_01_0397__b107401213173716">_Flume/etc/jaas.conf</strong>.</p>
|
|
<p id="mrs_01_0397__p121320491049">In the preceding path, <span class="parmname" id="mrs_01_0397__parmname5517144455217"><b>X</b></span> indicates a random number. Change it based on the site requirements. The file must be saved by the user who installs the Flume client, for example, user <strong id="mrs_01_0397__b1066213164">root</strong>.</p>
|
|
</p></li><li id="mrs_01_0397__li31329494415"><a name="mrs_01_0397__li31329494415"></a><a name="li31329494415"></a><span>Log in to the Flume client node and go to the client installation directory. Run the following command to modify the file:</span><p><p id="mrs_01_0397__p1513234914415"><strong id="mrs_01_0397__b11132114911416">vi conf/jaas.conf</strong></p>
|
|
<p id="mrs_01_0397__p8132134917416">Change the full path of the user authentication file defined by <span class="parmname" id="mrs_01_0397__parmname304068234"><b>keyTab</b></span> to the <span class="filepath" id="mrs_01_0397__filepath2003084318"><b>Flume client installation directory/fusioninsight-flume-<span id="mrs_01_0397__text1930181299"><em id="mrs_01_0397__i487892622">Flume component version number</em></span>/conf</b></span> saved in <a href="#mrs_01_0397__li4130849748">4</a>, and save the modification and exit.</p>
|
|
</p></li><li id="mrs_01_0397__li01322491414"><span>Run the following command to modify the <span class="filepath" id="mrs_01_0397__filepath38954248"><b>flume-env.sh</b></span> configuration file of the Flume client:</span><p><p id="mrs_01_0397__p51325495417"><strong id="mrs_01_0397__b1509514088">vi </strong><em id="mrs_01_0397__i1812114680">Flume client installation directory</em><strong id="mrs_01_0397__b655574013">/fusioninsight-flume-</strong><span id="mrs_01_0397__text217710191"><em id="mrs_01_0397__i731681110">Flume component version number</em></span><strong id="mrs_01_0397__b1289277344">/conf/flume-env.sh</strong></p>
|
|
<p id="mrs_01_0397__p10133164913412">Add the following information after <span class="parmvalue" id="mrs_01_0397__parmvalue900600841"><b>-XX:+UseCMSCompactAtFullCollection</b></span>:</p>
|
|
<pre class="screen" id="mrs_01_0397__screen9133249544">-Djava.security.krb5.conf=<em id="mrs_01_0397__i372408698">Flume client installation directory</em>/fusioninsight-flume-1.9.0/conf/kdc.conf -Djava.security.auth.login.config=<em id="mrs_01_0397__i362010831">Flume client installation directory</em>/fusioninsight-flume-1.9.0/conf/jaas.conf -Dzookeeper.request.timeout=120000</pre>
|
|
<p id="mrs_01_0397__p181335491640">Example: <strong id="mrs_01_0397__b184321941133719">"-XX:+UseCMSCompactAtFullCollection -Djava.security.krb5.conf=</strong><strong id="mrs_01_0397__b3441124117373">/opt/FlumeClient</strong>/<strong id="mrs_01_0397__b1644118412377">fusioninsight-flume</strong><strong id="mrs_01_0397__b0441184120374">-</strong><span id="mrs_01_0397__text5442144119371"><em id="mrs_01_0397__i7442041123719">Flume component version number</em></span><strong id="mrs_01_0397__b1444216417373">/conf/kdc.conf -Djava.security.auth.login.config=</strong><strong id="mrs_01_0397__b1744204113719">/opt/FlumeClient</strong>/<strong id="mrs_01_0397__b4443124183719">fusioninsight-flume-</strong><span id="mrs_01_0397__text3443541153713"><em id="mrs_01_0397__i104431441113716">Flume component version number</em></span><strong id="mrs_01_0397__b11443184113719">/conf/jaas.conf -Dzookeeper.request.timeout=120000"</strong></p>
|
|
<p id="mrs_01_0397__p7133549541">Change <em id="mrs_01_0397__i1747806650">Flume client installation directory</em> to the actual installation directory. Then save and exit.</p>
|
|
</p></li><li id="mrs_01_0397__li15133949946"><span>Run the following command to restart the Flume client:</span><p><p id="mrs_01_0397__p9863101120393"><strong id="mrs_01_0397__b229517591370">cd </strong><em id="mrs_01_0397__i629605915378">Flume client installation directory</em><strong id="mrs_01_0397__b19296959153719">/fusioninsight-flume-</strong><span id="mrs_01_0397__text6297185983711"><em id="mrs_01_0397__i18296195923712">Flume component version number</em></span><strong id="mrs_01_0397__b11297135913711">/bin</strong></p>
|
|
<p id="mrs_01_0397__p15863111123914"><strong id="mrs_01_0397__b9863911143918">./flume-manage.sh restart</strong></p>
|
|
<p id="mrs_01_0397__p2063491363912">Example:</p>
|
|
<p id="mrs_01_0397__p513313491046"><strong id="mrs_01_0397__b1462484460">cd /opt/FlumeClient/fusioninsight-flume-</strong><span id="mrs_01_0397__text2098288146"><em id="mrs_01_0397__i487111452">Flume component version number</em></span><strong id="mrs_01_0397__b793671374">/bin</strong></p>
|
|
<p id="mrs_01_0397__p21334491644"><strong id="mrs_01_0397__b013313491647">./flume-manage.sh restart</strong></p>
|
|
</p></li><li id="mrs_01_0397__li1391318335141"><span>Configure jobs based on actual service scenarios.</span><p><ul id="mrs_01_0397__ul143551339101419"><li id="mrs_01_0397__li1355339151418">Some parameters, for MRS 3.<em id="mrs_01_0397__i1262063915212">x</em> or later, can be configured on Manager.</li><li id="mrs_01_0397__li18902144615149">Set the parameters in the <strong id="mrs_01_0397__b1683775314417">properties.properties</strong> file. The following uses SpoolDir Source+File Channel+Kafka Sink as an example.<p id="mrs_01_0397__p7419182722014">Run the following command on the node where the Flume client is installed. Configure and save jobs in the Flume client configuration file <strong id="mrs_01_0397__b12015481643">properties.properties</strong> based on actual service requirements.</p>
|
|
<p id="mrs_01_0397__p1126102042013"><strong id="mrs_01_0397__b1840436028">vi </strong><em id="mrs_01_0397__i329927176">Flume client installation directory</em><strong id="mrs_01_0397__b758370028">/fusioninsight-flume-</strong><span id="mrs_01_0397__text1887635697"><em id="mrs_01_0397__i944320618">Flume component version number</em></span><strong id="mrs_01_0397__b2097568844">/conf/properties.properties</strong></p>
|
|
<pre class="screen" id="mrs_01_0397__screen152611320142014">#########################################################################################
|
|
client.sources = static_log_source
|
|
client.channels = static_log_channel
|
|
client.sinks = kafka_sink
|
|
#########################################################################################
|
|
#LOG_TO_HDFS_ONLINE_1
|
|
|
|
client.sources.static_log_source.type = spooldir
|
|
client.sources.static_log_source.spoolDir = <em id="mrs_01_0397__i184511510103910">Monitoring directory</em>
|
|
client.sources.static_log_source.fileSuffix = .COMPLETED
|
|
client.sources.static_log_source.ignorePattern = ^$
|
|
client.sources.static_log_source.trackerDir = <em id="mrs_01_0397__i8881213153915">Metadata storage path during transmission</em>
|
|
client.sources.static_log_source.maxBlobLength = 16384
|
|
client.sources.static_log_source.batchSize = 51200
|
|
client.sources.static_log_source.inputCharset = UTF-8
|
|
client.sources.static_log_source.deserializer = LINE
|
|
client.sources.static_log_source.selector.type = replicating
|
|
client.sources.static_log_source.fileHeaderKey = file
|
|
client.sources.static_log_source.fileHeader = false
|
|
client.sources.static_log_source.basenameHeader = true
|
|
client.sources.static_log_source.basenameHeaderKey = basename
|
|
client.sources.static_log_source.deletePolicy = never
|
|
|
|
client.channels.static_log_channel.type = file
|
|
client.channels.static_log_channel.dataDirs = <em id="mrs_01_0397__i1884141673911">Data cache path. Multiple paths, separated by commas (,), can be configured to improve performance.</em>
|
|
client.channels.static_log_channel.checkpointDir = <em id="mrs_01_0397__i3486121918396">Checkpoint storage path</em>
|
|
client.channels.static_log_channel.maxFileSize = 2146435071
|
|
client.channels.static_log_channel.capacity = 1000000
|
|
client.channels.static_log_channel.transactionCapacity = 612000
|
|
client.channels.static_log_channel.minimumRequiredSpace = 524288000
|
|
|
|
client.sinks.kafka_sink.type = org.apache.flume.sink.kafka.KafkaSink
|
|
client.sinks.kafka_sink.kafka.topic = <em id="mrs_01_0397__i965311225399">Topic to which data is written, for example, <strong id="mrs_01_0397__b365332215391">flume_test</strong></em>
|
|
client.sinks.kafka_sink.kafka.bootstrap.servers = <em id="mrs_01_0397__i1117469208">XXX</em>.<em id="mrs_01_0397__i1281317150">XXX</em>.<em id="mrs_01_0397__i212227060">XXX</em>.<em id="mrs_01_0397__i2093152102">XXX</em>:<em id="mrs_01_0397__i208992129">Kafka port number</em>,<em id="mrs_01_0397__i1676159719">XXX</em>.<em id="mrs_01_0397__i1848854889">XXX</em>.<em id="mrs_01_0397__i690594805">XXX</em>.<em id="mrs_01_0397__i1751552559">XXX</em>:<em id="mrs_01_0397__i1302113153">Kafka port number</em>,<em id="mrs_01_0397__i1015049785">XXX</em>.<em id="mrs_01_0397__i1648374724">XXX</em>.<em id="mrs_01_0397__i1468749894">XXX</em>.<em id="mrs_01_0397__i593685745">XXX</em>:<em id="mrs_01_0397__i19561183">Kafka port number</em>
|
|
client.sinks.kafka_sink.flumeBatchSize = 1000
|
|
client.sinks.kafka_sink.kafka.producer.type = sync
|
|
client.sinks.kafka_sink.kafka.security.protocol = SASL_PLAINTEXT
|
|
client.sinks.kafka_sink.kafka.kerberos.domain.name = <em id="mrs_01_0397__i16432172518395">Kafka domain name. This parameter is mandatory for a security cluster, for example, <strong id="mrs_01_0397__b943211252390">hadoop.xxx.com</strong>.</em>
|
|
client.sinks.kafka_sink.requiredAcks = 0
|
|
|
|
client.sources.static_log_source.channels = static_log_channel
|
|
client.sinks.kafka_sink.channel = static_log_channel</pre>
|
|
<div class="note" id="mrs_01_0397__note1927092016207"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0397__ul1927062072014"><li id="mrs_01_0397__li122711120182014"><strong id="mrs_01_0397__b1439928183913">client.sinks.kafka_sink.kafka.topic</strong>: Topic to which data is written. If the topic does not exist in Kafka, it is automatically created by default.</li><li id="mrs_01_0397__li12718205208"><strong id="mrs_01_0397__b27451633123910">client.sinks.kafka_sink.kafka.bootstrap.servers</strong>: List of Kafka Brokers, which are separated by commas (,). By default, the port is <strong id="mrs_01_0397__b690314364391">21007</strong> for a security cluster and <strong id="mrs_01_0397__b13903173623917">9092</strong> for a normal cluster.</li><li id="mrs_01_0397__li6271192012204"><strong id="mrs_01_0397__b85364018393">client.sinks.kafka_sink.kafka.security.protocol</strong>: The value is <strong id="mrs_01_0397__b12538406393">SASL_PLAINTEXT</strong> for a security cluster and <strong id="mrs_01_0397__b35394018398">PLAINTEXT</strong> for a normal cluster.</li><li id="mrs_01_0397__li17271132010203"><strong id="mrs_01_0397__b2331194323914">client.sinks.kafka_sink.kafka.kerberos.domain.name</strong>:<p id="mrs_01_0397__p5271720102016">You do not need to set this parameter for a normal cluster. For a security cluster, the value of this parameter is the value of <strong id="mrs_01_0397__b8881165017396">kerberos.domain.name</strong> in the Kafka cluster.</p>
|
|
<p id="mrs_01_0397__p182711620162015">For versions earlier than MRS 1.9.2, obtain the value by checking <strong id="mrs_01_0397__b10658653143915"><span id="mrs_01_0397__text1365712532391">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b1765845310392">/FusionInsight/etc/1_</strong><em id="mrs_01_0397__i0658165311396">X</em><strong id="mrs_01_0397__b56583535398">_Broker/server.properties</strong> on the node where the broker instance resides.</p>
|
|
<p id="mrs_01_0397__p1327242018201">Obtain the value for versions earlier than MRS 3.<em id="mrs_01_0397__i166070567398">x</em> by checking <strong id="mrs_01_0397__b2060818567395"><span id="mrs_01_0397__text56082566392">${BIGDATA_HOME}</span></strong><strong id="mrs_01_0397__b146081256183914">/MRS_Current/1_</strong><em id="mrs_01_0397__i176091256143911">X</em><strong id="mrs_01_0397__b1609156193919">_Broker/etc/server.properties</strong> on the node where the broker instance resides.</p>
|
|
<p id="mrs_01_0397__p1427202012203">In the preceding paths, <span class="parmname" id="mrs_01_0397__parmname43368094017"><b>X</b></span> indicates a random number. Change it based on site requirements. The file must be saved by the user who installs the Flume client, for example, user <strong id="mrs_01_0397__b650686645">root</strong>.</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</li></ul>
|
|
</p></li><li id="mrs_01_0397__li1313611498420"><span>After the parameters are set and saved, the Flume client automatically loads the content configured in <strong id="mrs_01_0397__b177471138174013">properties.properties</strong>. When new log files are generated by spoolDir, the files are sent to Kafka producers and can be consumed by Kafka consumers. </span></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0390.html">Using Flume</a></div>
|
|
</div>
|
|
</div>
|
|
|