forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
44 lines
13 KiB
HTML
44 lines
13 KiB
HTML
<a name="mrs_01_1031"></a><a name="mrs_01_1031"></a>
|
|
|
|
<h1 class="topictitle1">Using Kafka from Scratch</h1>
|
|
<div id="body1590118474320"><div class="section" id="mrs_01_1031__section43461647164218"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1031__p8060118">You can create, query, and delete topics on a cluster client. </p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1031__section1022821213273"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_1031__p614503812278">The client has been installed. For example, the client is installed in the <strong id="mrs_01_1031__b53165215138">/opt/hadoopclient</strong> directory. The client directory in the following operations is only an example. Change it to the actual installation directory.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1031__section179001923123114"><h4 class="sectiontitle">Using the Kafka Client (Versions Earlier Than MRS 3.x)</h4><ol id="mrs_01_1031__ol62572327321"><li id="mrs_01_1031__li13806122116487"><span>Access the ZooKeeper instance page.</span><p><ul id="mrs_01_1031__ul1692910323486"><li id="mrs_01_1031__li792916322486">For versions earlier than MRS 1.9.2, log in to MRS Manager and choose <strong id="mrs_01_1031__b1584916171418">Services</strong> > <strong id="mrs_01_1031__b2090181614141">ZooKeeper</strong> > <strong id="mrs_01_1031__b179071612143">Instance</strong>.</li><li id="mrs_01_1031__li979413816488">For MRS 1.9.2 or later to versions earlier than 3.x, click the cluster name on the MRS console and choose <strong id="mrs_01_1031__b56141727181418">Components</strong> > <strong id="mrs_01_1031__b562032781420">ZooKeeper</strong> > <strong id="mrs_01_1031__b2620122751416">Instances</strong>.<div class="note" id="mrs_01_1031__note1664505711110"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1031__p158007468574">If the <strong id="mrs_01_1031__b33924111304">Components</strong> tab is unavailable, complete IAM user synchronization first. (On the <strong id="mrs_01_1031__b1239291112016">Dashboard</strong> page, click <strong id="mrs_01_1031__b939315111101">Synchronize</strong> on the right side of <span class="parmname" id="mrs_01_1031__parmname73938113016"><b>IAM User Sync</b></span> to synchronize IAM users.)</p>
|
|
</div></div>
|
|
</li></ul>
|
|
</p></li><li id="mrs_01_1031__l2f4a4c4b2bbb427a890864b6ae357ea8"><span>View the IP addresses of the ZooKeeper role instance.</span><p><p id="mrs_01_1031__a2c7ffa9191bf467cb17b66c27ef78e5a">Record any IP address of the ZooKeeper instance.</p>
|
|
</p></li><li id="mrs_01_1031__l9b2a798722044c01975eaaee4bebd7dd"><span>Log in to the node where the client is installed.</span></li><li id="mrs_01_1031__l14680ebec7694c71a1e3056f9ae0f72b"><span>Run the following command to switch to the client directory, for example, <strong id="mrs_01_1031__b94710385151">/opt/hadoopclient/Kafka/kafka/bin</strong>.</span><p><p id="mrs_01_1031__af3976b170fc4424ba173ec0c73381a13"><strong id="mrs_01_1031__b295019434">cd /opt/hadoopclient/Kafka/kafka/bin</strong></p>
|
|
</p></li><li id="mrs_01_1031__l9974bee6f19240e692c4157eeab1a308"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_1031__abf436e581670449ca4950d9225d58143"><strong id="mrs_01_1031__b571916241312">source /opt/hadoopclient/bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1031__li1215513270361"><span>If Kerberos authentication is enabled for the current cluster, run the following command to authenticate the current user. If Kerberos authentication is disabled for the current cluster, skip this step.</span><p><p id="mrs_01_1031__p1678314421362"><strong id="mrs_01_1031__b1282141319180">kinit</strong> <em id="mrs_01_1031__i9985122951911">Kafka user</em></p>
|
|
</p></li><li id="mrs_01_1031__li1147589014556"><a name="mrs_01_1031__li1147589014556"></a><a name="li1147589014556"></a><span>Create a topic.</span><p><p id="mrs_01_1031__p3617414814556"><strong id="mrs_01_1031__b17303132094019">sh kafka-topics.sh --create --topic </strong><em id="mrs_01_1031__i193111620194014">Topic name</em><strong id="mrs_01_1031__b15303192054017"> --partitions</strong> <em id="mrs_01_1031__aa940a246ae2e42c7bbff2dca5878a13f">Number of partitions occupied by the topic</em> <strong id="mrs_01_1031__b1122717119010">--replication-factor</strong> <em id="mrs_01_1031__a31cb2d75a02c4813a2de4393fd9a12ea">Number of replicas of the topic</em> <strong id="mrs_01_1031__b1686617281503">--zookeeper</strong> <em id="mrs_01_1031__adcf559b8c1194c6881c1c7c7b715c203">IP address of the node where the ZooKeeper instance resides<strong id="mrs_01_1031__b15929114214020">:</strong>clientPort</em><strong id="mrs_01_1031__a82a1c91d46d2459d82cf7205264a9db6">/kafka</strong></p>
|
|
<p id="mrs_01_1031__p96347471817">Example: <strong id="mrs_01_1031__b13655235916">sh kafka-topics.sh --create --topic </strong><strong id="mrs_01_1031__b1474852918246">TopicTest</strong><strong id="mrs_01_1031__b1965516316915"> --partitions</strong> <strong id="mrs_01_1031__b582936122412">3</strong> <strong id="mrs_01_1031__b1165517314915">--replication-factor</strong> <strong id="mrs_01_1031__b2788245202420">3</strong> <strong id="mrs_01_1031__b196551934919">--zookeeper</strong> <strong id="mrs_01_1031__b1430819532248">10.10.10.100:2181</strong><strong id="mrs_01_1031__b1865653895">/kafka</strong></p>
|
|
</p></li><li id="mrs_01_1031__li4442485714556"><span>Run the following command to view the topic information in the cluster:</span><p><p id="mrs_01_1031__p15751135191215"><strong id="mrs_01_1031__b10171455171212">sh kafka-topics.sh --list --zookeeper </strong><em id="mrs_01_1031__i9173135511220">IP address of the node where the ZooKeeper instance resides<strong id="mrs_01_1031__b9111031204818">:</strong>clientPort</em><strong id="mrs_01_1031__b14172955171212">/kafka</strong></p>
|
|
<p id="mrs_01_1031__p954781332312">Example: <strong id="mrs_01_1031__b18959135182315">sh kafka-topics.sh --list --zookeeper </strong><strong id="mrs_01_1031__b176397510243">10.10.10.100:2181</strong><strong id="mrs_01_1031__b1295915350232">/kafka</strong></p>
|
|
</p></li><li id="mrs_01_1031__li1769484514556"><span>Delete the topic created in <a href="#mrs_01_1031__li1147589014556">7</a>.</span><p><p id="mrs_01_1031__p2503588514556"><strong id="mrs_01_1031__b184631439804">sh kafka-topics.sh --delete --topic</strong> <em id="mrs_01_1031__a3bb92ab95f644f1bae029b234b16f195">Topic name</em> <strong id="mrs_01_1031__b1665818581108">--zookeeper</strong> <em id="mrs_01_1031__a5e852321e9b846e2bb3e069d5afa7c86">IP address of the node where the ZooKeeper instance resides</em>:<em id="mrs_01_1031__a39646f6f17374119b87533c396460803">clientPort</em><strong id="mrs_01_1031__ade536fe9693d41518b0dcd8943091491">/kafka</strong></p>
|
|
<p id="mrs_01_1031__p0169143162511">Example: <strong id="mrs_01_1031__b8432162416256">sh kafka-topics.sh --delete --topic</strong> <strong id="mrs_01_1031__b19620740142519">TopicTest</strong> <strong id="mrs_01_1031__b114331924192518">--zookeeper</strong> <strong id="mrs_01_1031__b14679175682511">10.10.10.100:2181</strong><strong id="mrs_01_1031__b6433102410259">/kafka</strong></p>
|
|
<p id="mrs_01_1031__p71201059288">Type <strong id="mrs_01_1031__b12144530121920">y</strong> and press <strong id="mrs_01_1031__b1144530111913">Enter</strong>.</p>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_1031__section2695144004617"><h4 class="sectiontitle">Using the Kafka Client (MRS 3.x or Later)</h4><ol id="mrs_01_1031__ol1980725416469"><li id="mrs_01_1031__li1280725416467"><span>Access the ZooKeeper instance page.</span><p><p id="mrs_01_1031__p041836175710">Log in to FusionInsight Manager. For details, see <a href="mrs_01_2124.html">Accessing FusionInsight Manager (MRS 3.x or Later)</a>. Choose <strong id="mrs_01_1031__b1212106633111149">Cluster</strong> > <em id="mrs_01_1031__i846383776111149">Name of the desired cluster</em> > <strong id="mrs_01_1031__b959103026111149">Services</strong> > <strong id="mrs_01_1031__b781653376111149">ZooKeeper</strong> > <strong id="mrs_01_1031__b472259796111149">Instance</strong>.</p>
|
|
</p></li><li id="mrs_01_1031__li118082541465"><span>View the IP addresses of the ZooKeeper role instance.</span><p><p id="mrs_01_1031__p98081854134614">Record any IP address of the ZooKeeper instance.</p>
|
|
</p></li><li id="mrs_01_1031__li14808105414460"><span>Log in to the node where the client is installed.</span></li><li id="mrs_01_1031__li16808115412465"><span>Run the following command to switch to the client directory, for example, <strong id="mrs_01_1031__b189513124168">/opt/hadoopclient/Kafka/kafka/bin</strong>.</span><p><p id="mrs_01_1031__p7808165412465"><strong id="mrs_01_1031__b8808165415468">cd /opt/hadoopclient/Kafka/kafka/bin</strong></p>
|
|
</p></li><li id="mrs_01_1031__li17808195411460"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_1031__p188084545468"><strong id="mrs_01_1031__b380818549466">source /opt/hadoopclient/bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1031__li174606506355"><span>If Kerberos authentication is enabled for the current cluster, run the following command to authenticate the current user. If Kerberos authentication is disabled for the current cluster, skip this step.</span><p><p id="mrs_01_1031__p227711443610"><strong id="mrs_01_1031__b427203417">kinit</strong> <em id="mrs_01_1031__i983292940">Kafka user</em></p>
|
|
</p></li><li id="mrs_01_1031__li156781252154418"><span>Log in to FusionInsight Manager, choose <strong id="mrs_01_1031__b1219194213292">Cluster</strong> > <strong id="mrs_01_1031__b5287947162912">Name of the desired cluster</strong> > <strong id="mrs_01_1031__b1544115518291">Services</strong> > <strong id="mrs_01_1031__b71708109303">ZooKeeper</strong>, and click the <strong id="mrs_01_1031__b19138935143014">Configurations</strong> tab and then <strong id="mrs_01_1031__b02301346183020">All Configurations</strong>. On the displayed page, search for the <strong id="mrs_01_1031__b1639633619320">clientPort</strong> parameter and record its value.</span></li><li id="mrs_01_1031__li4808125415465"><a name="mrs_01_1031__li4808125415465"></a><a name="li4808125415465"></a><span>Create a topic.</span><p><p id="mrs_01_1031__p280865464613"><strong id="mrs_01_1031__b1859216522">sh kafka-topics.sh --create --topic </strong><em id="mrs_01_1031__i2045577658">Topic name</em><strong id="mrs_01_1031__b552546681"> --partitions</strong> <em id="mrs_01_1031__i313359955">Number of partitions occupied by the topic</em> <strong id="mrs_01_1031__b1372144023">--replication-factor</strong> <em id="mrs_01_1031__i150099588">Number of replicas of the topic</em> <strong id="mrs_01_1031__b1509194707">--zookeeper</strong> <em id="mrs_01_1031__i1378560812">IP address of the node where the ZooKeeper instance resides<strong id="mrs_01_1031__b701656545">:</strong>clientPort</em><strong id="mrs_01_1031__b958412839">/kafka</strong></p>
|
|
<p id="mrs_01_1031__p1430419652918">Example: <strong id="mrs_01_1031__b10187154352612">sh kafka-topics.sh --create --topic </strong><strong id="mrs_01_1031__b61882435260">TopicTest</strong><strong id="mrs_01_1031__b201911143142613"> --partitions</strong> <strong id="mrs_01_1031__b1319112436267">3</strong> <strong id="mrs_01_1031__b19192143182615">--replication-factor</strong> <strong id="mrs_01_1031__b13192174322615">3</strong> <strong id="mrs_01_1031__b141922043202613">--zookeeper</strong> <strong id="mrs_01_1031__b121941743172610">10.10.10.100:2181</strong><strong id="mrs_01_1031__b2194134332614">/kafka</strong></p>
|
|
</p></li><li id="mrs_01_1031__li08091154164619"><span>Run the following command to view the topic information in the cluster:</span><p><p id="mrs_01_1031__p5809154164618"><strong id="mrs_01_1031__b1626690109">sh kafka-topics.sh --list --zookeeper </strong><em id="mrs_01_1031__i782242570">IP address of the node where the ZooKeeper instance resides<strong id="mrs_01_1031__b1034254278">:</strong>clientPort</em><strong id="mrs_01_1031__b1751409297">/kafka</strong></p>
|
|
<p id="mrs_01_1031__p4889155022918">Example: <strong id="mrs_01_1031__b2225114713266">sh kafka-topics.sh --list --zookeeper </strong><strong id="mrs_01_1031__b7226547162619">10.10.10.100:2181</strong><strong id="mrs_01_1031__b422654715262">/kafka</strong></p>
|
|
</p></li><li id="mrs_01_1031__li19809195411460"><span>Delete the topic created in <a href="#mrs_01_1031__li4808125415465">8</a>.</span><p><p id="mrs_01_1031__p4809135464615"><strong id="mrs_01_1031__b1696493876">sh kafka-topics.sh --delete --topic</strong> <em id="mrs_01_1031__i237043267">Topic name</em> <strong id="mrs_01_1031__b1218425419">--zookeeper</strong> <em id="mrs_01_1031__i881437892">IP address of the node where the ZooKeeper instance resides</em>:<em id="mrs_01_1031__i1853764335">clientPort</em><strong id="mrs_01_1031__b482209164">/kafka</strong></p>
|
|
<p id="mrs_01_1031__p12952121313301">Example: <strong id="mrs_01_1031__b1596974852616">sh kafka-topics.sh --delete --topic</strong> <strong id="mrs_01_1031__b7970748162612">TopicTest</strong> <strong id="mrs_01_1031__b13970154812269">--zookeeper</strong> <strong id="mrs_01_1031__b2097194842610">10.10.10.100:2181</strong><strong id="mrs_01_1031__b497110484266">/kafka</strong></p>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0375.html">Using Kafka</a></div>
|
|
</div>
|
|
</div>
|
|
|