Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

32 lines
9.2 KiB
HTML

<a name="mrs_01_0379"></a><a name="mrs_01_0379"></a>
<h1 class="topictitle1">Managing Messages in Kafka Topics</h1>
<div id="body8662426"><div class="section" id="mrs_01_0379__en-us_topic_0000001219351149_s97aa33d045f046bb9c1d9907172b3db2"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0379__en-us_topic_0000001219351149_a5d6cb2522caf46b6ae5e00750c6f3d83">You can produce or consume messages in Kafka topics using the MRS cluster client. For clusters with Kerberos authentication enabled, you must have the permission to perform these operations.</p>
</div>
<div class="section" id="mrs_01_0379__en-us_topic_0000001219351149_sa5c652338d284a47ad73fc0e957155c5"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_0379__en-us_topic_0000001219351149_a1acbd534cd91469080e7578d5bafdc50">You have installed the Kafka client.</p>
</div>
<div class="section" id="mrs_01_0379__en-us_topic_0000001219351149_s87eef4ea94af42b7944cf554191823a4"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_0379__en-us_topic_0000001219351149_o734dac617c7c4e8dadaf85650ff773b7"><li id="mrs_01_0379__en-us_topic_0000001219351149_li777313202051"><span>Go to the Kafka service page.</span><p><p id="mrs_01_0379__p63998293316">Log in to FusionInsight Manager and choose <strong id="mrs_01_0379__en-us_topic_0000001219351149_b919323182615">Cluster</strong> &gt; <em id="mrs_01_0379__en-us_topic_0000001219351149_i519583132613">Name of the desired cluster</em> &gt; <strong id="mrs_01_0379__en-us_topic_0000001219351149_b191971531192615">Services</strong> &gt; <strong id="mrs_01_0379__en-us_topic_0000001219351149_b181981331172614">Kafka</strong>.</p>
</p></li><li id="mrs_01_0379__en-us_topic_0000001219351149_l17ba03ec896f47d8a5c53b1e8f726441"><span>Click <span class="uicontrol" id="mrs_01_0379__en-us_topic_0000001219351149_uicontrol1334161117819"><b>instance</b></span>. Query the IP addresses of the Kafka instances.</span><p><p id="mrs_01_0379__en-us_topic_0000001219351149_ac99617dfc17b4cd89397cfb63d2de888">Record the IP address of any Kafka instance.</p>
</p></li><li id="mrs_01_0379__en-us_topic_0000001219351149_lc99c3810794b4305a8805127c0448528"><span>Prepare the client based on service requirements. Log in to the node where the client is installed.</span></li><li id="mrs_01_0379__en-us_topic_0000001219351149_l43ebcdf0488f43aa8a6dc5925fa6d624"><span>Run the following command to switch to the client directory, for example, /<strong id="mrs_01_0379__en-us_topic_0000001219351149_b177601388257">opt</strong><strong id="mrs_01_0379__en-us_topic_0000001219351149_b9760193814255"></strong><strong id="mrs_01_0379__en-us_topic_0000001219351149_b7760113862512">/client/Kafka/kafka/bin</strong>.</span><p><p id="mrs_01_0379__en-us_topic_0000001219351149_abf921a6902ab4d60bb2418a62dd04f71"><strong id="mrs_01_0379__en-us_topic_0000001219351149_aa2ffb20c51ca4d0bbf4035c8445d8c74">cd /opt/client/Kafka/kafka/bin</strong></p>
</p></li><li id="mrs_01_0379__en-us_topic_0000001219351149_l6c733670bf264983a44af845bf95d835"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_0379__en-us_topic_0000001219351149_afb1d4384127a43e1b162f119632db3bd"><strong id="mrs_01_0379__en-us_topic_0000001219351149_ab5d39d6fe1ff4198a45fa9cb49fd05c2">source /opt/client/bigdata_env</strong></p>
</p></li><li id="mrs_01_0379__en-us_topic_0000001219351149_l62c83e0a6fe7451db024ee5be8542c12"><span>For clusters with Kerberos authentication enabled, run the following command to authenticate the user. For clusters with Kerberos authentication disabled, skip this step.</span><p><p id="mrs_01_0379__en-us_topic_0000001219351149_a12b63597f2374ecfac281f9803f541e2"><strong id="mrs_01_0379__en-us_topic_0000001219351149_b12323201355115">kinit</strong> <em id="mrs_01_0379__en-us_topic_0000001219351149_i15328613205116">Kafka user</em></p>
<p id="mrs_01_0379__en-us_topic_0000001219351149_p1517912384815">Example:</p>
<p id="mrs_01_0379__en-us_topic_0000001219351149_ab973d2e6dc584ce996bd87ba46250f10"><strong id="mrs_01_0379__en-us_topic_0000001219351149_a310ad170e91146c897a1ba5f4e634d43">kinit admin</strong></p>
</p></li><li id="mrs_01_0379__en-us_topic_0000001219351149_l4998af2788284d209339aff58f21ad42"><span>Manage messages in Kafka topics using the following commands:</span><p><ul id="mrs_01_0379__en-us_topic_0000001219351149_u6117f8baf6e94bf4bab52f9a13b8867a"><li id="mrs_01_0379__en-us_topic_0000001219351149_lb4468efc44c4410db1fdd9bed4ed5742">Producing messages<p id="mrs_01_0379__en-us_topic_0000001219351149_a7b38507bd22d444abfaf96672ddff8b1"><a name="mrs_01_0379__en-us_topic_0000001219351149_lb4468efc44c4410db1fdd9bed4ed5742"></a><a name="en-us_topic_0000001219351149_lb4468efc44c4410db1fdd9bed4ed5742"></a><strong id="mrs_01_0379__en-us_topic_0000001219351149_b136457202402">sh kafka-console-producer.sh --broker-list</strong> <em id="mrs_01_0379__en-us_topic_0000001219351149_ad4cb407ba2c54021864adb859619ccd3">IP address of the node where the Kafka instance resides</em><strong id="mrs_01_0379__en-us_topic_0000001219351149_b9444153024010">:9092 --topic</strong> <em id="mrs_01_0379__en-us_topic_0000001219351149_a41e4d250d37c4f70becacf0e227b9c5b">Topic name</em> <strong id="mrs_01_0379__en-us_topic_0000001219351149_b18571133914015">--producer.config /opt/client/Kafka/kafka/config/producer.properties</strong></p>
<p id="mrs_01_0379__en-us_topic_0000001219351149_adbfec31492e84dec85245866152e21be">You can input specified information as the messages produced by the producer and then press <strong id="mrs_01_0379__en-us_topic_0000001219351149_en-us_topic_0054328610_b842352706145827">Enter</strong> to send the messages. To end message producing, press <span class="parmname" id="mrs_01_0379__en-us_topic_0000001219351149_en-us_topic_0054328610_parmname53594428115310"><b>Ctrl + C</b></span> to exit.</p>
</li><li id="mrs_01_0379__en-us_topic_0000001219351149_lf644862db08e49a3a0220751b224e3f5">Consuming messages<p id="mrs_01_0379__en-us_topic_0000001219351149_a70e2de5e4e8f4fb1ab3d7729f99ae5b2"><a name="mrs_01_0379__en-us_topic_0000001219351149_lf644862db08e49a3a0220751b224e3f5"></a><a name="en-us_topic_0000001219351149_lf644862db08e49a3a0220751b224e3f5"></a><strong id="mrs_01_0379__en-us_topic_0000001219351149_b762345014018">sh kafka-console-consumer.sh --topic</strong> <em id="mrs_01_0379__en-us_topic_0000001219351149_a35f63752e3e94aa7abc4c0d5b680607b">Topic name</em> <strong id="mrs_01_0379__en-us_topic_0000001219351149_b44661734419">--bootstrap-server</strong> <em id="mrs_01_0379__en-us_topic_0000001219351149_a91c908b9ff4e4c9487134f1292b65532">IP address of the node where the Kafka instance resides</em><strong id="mrs_01_0379__en-us_topic_0000001219351149_add1bb9f278eb479288c746d3fd6ac33f">:9092 --consumer.config /opt/client/Kafka/kafka/config/consumer.properties</strong></p>
<p id="mrs_01_0379__en-us_topic_0000001219351149_a1ed7692cb70141c3be5396648a495f01">In the configuration file, <span class="parmname" id="mrs_01_0379__en-us_topic_0000001219351149_en-us_topic_0054328610_parmname131291742291820"><b>group.id</b></span> (indicating the consumer group) is set to <span class="parmvalue" id="mrs_01_0379__en-us_topic_0000001219351149_en-us_topic_0054328610_parmvalue3068481691826"><b>example-group1</b></span> by default. Users can change the value as required. The value takes effect each time consumption occurs.</p>
<p id="mrs_01_0379__en-us_topic_0000001219351149_aa70ea7c8ccec40f7862182db36e3d103">By default, the system reads unprocessed messages in the current consumer group when the command is executed. If a new consumer group is specified in the configuration file and the <span class="parmname" id="mrs_01_0379__en-us_topic_0000001219351149_en-us_topic_0054328610_parmname4937532169517"><b>--from-beginning</b></span> parameter is added to the command, the system reads all messages that have not been automatically deleted in Kafka.</p>
</li></ul>
<div class="note" id="mrs_01_0379__en-us_topic_0000001219351149_n423e2322dc654e578f61b5c2eb04c0ff"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0379__en-us_topic_0000001219351149_u504276addcca4640900fe37d9f5a21ce"><li id="mrs_01_0379__en-us_topic_0000001219351149_laae5cccb4fce48399c17d97f9c67c6ac">For the IP address of the node where the Kafka instance locates, use the IP address of any broker instance.</li><li id="mrs_01_0379__en-us_topic_0000001219351149_l0dbcf512675a44c8abfce63d72b0c036">If Kerberos authentication is enabled, change the port to <span class="parmvalue" id="mrs_01_0379__en-us_topic_0000001219351149_en-us_topic_0054328610_parmvalue37116123175139"><b>21007</b></span>.</li><li id="mrs_01_0379__en-us_topic_0000001219351149_li3741142312283">By default, the ZooKeeper's <span class="parmname" id="mrs_01_0379__en-us_topic_0000001219351149_en-us_topic_0244909890_pd3e452bfa0d24adda216f431d1bcebc0"><b>clientPort</b></span> value is <span class="parmvalue" id="mrs_01_0379__en-us_topic_0000001219351149_en-us_topic_0244909890_p67aea0e3f02b49829ae8bf8dea1fc2d7"><b>2<span id="mrs_01_0379__en-us_topic_0000001219351149_en-us_topic_0244909890_text59469239252">181</span></b></span>.</li></ul>
</div></div>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0375.html">Using Kafka</a></div>
</div>
</div>