vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dms/umn/kafka-ug-0014.html
Chen, Junjie 1588318a68 DMS UMN 20240723 version 
Reviewed-by: Antonova, Ekaterina <ekantono@noreply.gitea.eco.tsi-dev.otc-service.com>
Co-authored-by: Chen, Junjie <chenjunjie@huawei.com>
Co-committed-by: Chen, Junjie <chenjunjie@huawei.com>
2024-08-14 12:31:58 +00:00

129 lines
19 KiB
HTML
Raw Permalink Blame History

<a name="kafka-ug-0014"></a><a name="kafka-ug-0014"></a>
<h1 class="topictitle1">Viewing and Resetting Kafka Consumption Offsets</h1>
<div id="body0000001241621406"><p id="kafka-ug-0014__p11638355589">A consumption offset indicates the consumption progress of a consumer. This section describes how to view and reset consumption offsets.</p>
<div class="notice" id="kafka-ug-0014__note228194010371"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="kafka-ug-0014__p1528154073712">Messages may be retrieved more than once after the offset is reset. Exercise caution when performing this operation.</p>
</div></div>
<div class="section" id="kafka-ug-0014__section8241820195010"><h4 class="sectiontitle">Prerequisites</h4><p id="kafka-ug-0014__p6328625145014">The consumer offset cannot be reset on the fly. You must first stop retrieval of the desired consumer group.</p>
<div class="notice" id="kafka-ug-0014__note3382518125816"><span class="noticetitle"><img src="public_sys-resources/notice_3.0-en-us.png"> </span><div class="noticebody"><p id="kafka-ug-0014__p53821618115813">After a client is stopped, the server considers the client offline only after the time period specified in <strong id="kafka-ug-0014__b333103411017">ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG</strong> (1000 ms by default).</p>
</div></div>
</div>
<div class="section" id="kafka-ug-0014__section10785080316"><h4 class="sectiontitle">Viewing Consumer Offsets (Console)</h4><ol id="kafka-ug-0014__ol168540273812"><li id="kafka-ug-0014__li285417271983"><span>Log in to the console.</span></li><li id="kafka-ug-0014__li4854162712814"><span>Click <span><img id="kafka-ug-0014__image1363965111107" src="en-us_image_0143929918.png"></span> in the upper left corner to select a region.</span><p><div class="note" id="kafka-ug-0014__note128546279817"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="kafka-ug-0014__p1685422711816">Select the region where your Kafka instance is located.</p>
</div></div>
</p></li><li id="kafka-ug-0014__li62547586143"><span>Click <strong id="kafka-ug-0014__b187677460595717">Service List</strong> and choose <strong id="kafka-ug-0014__b146841523695717">Application</strong> &gt; <strong id="kafka-ug-0014__b160124893495717">Distributed Message Service</strong>. The Kafka instance list is displayed.</span></li><li id="kafka-ug-0014__li98548271981"><span>Click the desired Kafka instance to view the instance details.</span></li><li id="kafka-ug-0014__li1285417275812"><span>In the navigation pane, choose <strong id="kafka-ug-0014__b16867605412">Consumer Groups</strong>.</span></li><li id="kafka-ug-0014__li188541727584"><span>Click the name of the desired consumer group.</span></li><li id="kafka-ug-0014__li186129567324"><span>On the <strong id="kafka-ug-0014__b23461221114119">Consumer Offset</strong> tab page, view the list of topics that the consumer group has subscribed to, total number of messages accumulated in the topic, message consumption progress in each partition of the topic (accumulated messages, offset, latest offset, consumer ID, consumer address, and client ID).</span></li><li id="kafka-ug-0014__li15501253173615"><span>(Optional) To query the consumer offsets of a specific topic, enter the topic name in the search box and press <strong id="kafka-ug-0014__b189013004614">Enter</strong>.</span></li></ol>
</div>
<div class="section" id="kafka-ug-0014__section105713498316"><h4 class="sectiontitle">Viewing Consumer Offsets (Kafka CLI)</h4><ul id="kafka-ug-0014__ul551916226323"><li id="kafka-ug-0014__li11519112253212">For a Kafka instance with ciphertext access disabled, run the following command in the <strong id="kafka-ug-0014__b135492056291">/bin</strong> directory of the Kafka client:<pre class="screen" id="kafka-ug-0014__screen1651912210321">./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --offsets --describe --all-groups</pre>
<p id="kafka-ug-0014__p251111134315">Parameter description: <strong id="kafka-ug-0014__b134406338451251">connection-address</strong> indicates the Kafka instance address, which can be obtained in the <strong id="kafka-ug-0014__b150607531851251">Connection</strong> area on the <strong id="kafka-ug-0014__b200934741051251">Basic Information</strong> page on the Kafka console.</p>
<p id="kafka-ug-0014__p1968375217431">Example:</p>
<pre class="screen" id="kafka-ug-0014__screen1773018065718">[root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9092,192.168.xx.xx:9092,192.168.xx.xx:9092 --offsets --describe --all-groups
Consumer group '__consumer-group-dial-test' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
__consumer-group-dial-test __dms_dial_test 0 350 350 0 - - -
__consumer-group-dial-test __dms_dial_test 1 350 350 0 - - -
__consumer-group-dial-test __dms_dial_test 2 350 350 0 - - -
Consumer group 'test' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test topic-01 0 5 5 0 - - -
test topic-01 1 3 3 0 - - -
test topic-01 2 10 10 0 - - -
[root@ecs-kafka bin]#</pre>
</li><li id="kafka-ug-0014__li16519132216327">For a Kafka instance with ciphertext access enabled, do as follows:<ol id="kafka-ug-0014__ol13519182213216"><li id="kafka-ug-0014__li17434105674910">(Optional) Modify the client configuration file.<div class="p" id="kafka-ug-0014__p12707257114914"><a name="kafka-ug-0014__li17434105674910"></a><a name="li17434105674910"></a>View <strong id="kafka-ug-0014__b184286886251320">Security Protocol</strong> in the <strong id="kafka-ug-0014__b18163868051320">Connection</strong> area on the <strong id="kafka-ug-0014__b8655494751320">Basic Information</strong> page on the Kafka console. The configuration settings vary depending on the protocol.<ul id="kafka-ug-0014__ul2643717214"><li id="kafka-ug-0014__li4646711216">SASL_PLAINTEXT: Skip this step if the username and password are already set. Otherwise, create the <strong id="kafka-ug-0014__b146568151330">ssl-user-config.properties</strong> file in the <strong id="kafka-ug-0014__b33993736351330">/config</strong> directory on the Kafka client and add the following content to the file:<pre class="screen" id="kafka-ug-0014__screen1431203505712">security.protocol=SASL_PLAINTEXT
# If the SASL mechanism is SCRAM-SHA-512, configure as follows:
sasl.jaas.config=org.apache.kafka.common.security.<strong id="kafka-ug-0014__kafka-ug-180604018_b883312781111">scram.</strong><strong id="kafka-ug-0014__kafka-ug-180604018_b1783311741112">ScramLoginModule</strong> required \
username="**********" \
password="**********";
sasl.mechanism=<strong id="kafka-ug-0014__kafka-ug-180604018_b128331377110">SCRAM-SHA-512</strong>
# If the SASL mechanism is PLAIN, configure as follows:
sasl.jaas.config=org.apache.kafka.common.security.<strong id="kafka-ug-0014__kafka-ug-180604018_b166481219196">plain.PlainLoginModule</strong> required \
username="**********" \
password="**********";
sasl.mechanism=<strong id="kafka-ug-0014__kafka-ug-180604018_b964813191391">PLAIN</strong></pre>
<p id="kafka-ug-0014__p2629183165718">Parameter description: <strong id="kafka-ug-0014__b1183042314103927">username</strong> and <strong id="kafka-ug-0014__b530357212103927">password</strong> are the ones you set when enabling ciphertext access for the first time or when creating a user.</p>
</li><li id="kafka-ug-0014__li493632862118">SASL_SSL: Skip this step if the username, password, and SSL certificate are already set. Otherwise, create the <strong id="kafka-ug-0014__b28057774151330">ssl-user-config.properties</strong> file in the <strong id="kafka-ug-0014__b94293072351330">/config</strong> directory on the Kafka client and add the following content to the file:<pre class="screen" id="kafka-ug-0014__screen1586842195817">security.protocol=SASL_SSL
ssl.truststore.location={ssl_truststore_path}
ssl.truststore.password=dms@kafka
ssl.endpoint.identification.algorithm=
# If the SASL mechanism is SCRAM-SHA-512, configure as follows:
sasl.jaas.config=org.apache.kafka.common.security.<strong id="kafka-ug-0014__kafka-ug-180604018_b4657131681320">scram.</strong><strong id="kafka-ug-0014__kafka-ug-180604018_b20657171612138">ScramLoginModule</strong> required \
username="**********" \
password="**********";
sasl.mechanism=<strong id="kafka-ug-0014__kafka-ug-180604018_b15657121620131">SCRAM-SHA-512</strong>
# If the SASL mechanism is PLAIN, configure as follows:
sasl.jaas.config=org.apache.kafka.common.security.<strong id="kafka-ug-0014__kafka-ug-180604018_b0657101619133">plain.PlainLoginModule</strong> required \
username="**********" \
password="**********";
sasl.mechanism=<strong id="kafka-ug-0014__kafka-ug-180604018_b1665720164136">PLAIN</strong></pre>
<p id="kafka-ug-0014__p574013714580">Parameter description:</p>
<ul id="kafka-ug-0014__ul179961085914"><li id="kafka-ug-0014__kafka-ug-180604018_li9578173610155"><strong id="kafka-ug-0014__kafka-ug-180604018_b15940166202617">ssl.truststore.location</strong>: path for storing the <strong id="kafka-ug-0014__kafka-ug-180604018_b1994056112615">client.jks</strong> certificate. Even in Windows, you need to use slashes (/) for the certificate path. Do not use backslashes (\), which are used by default for paths in Windows. Otherwise, the client will fail to obtain the certificate.</li><li id="kafka-ug-0014__kafka-ug-180604018_li4578153611159"><strong id="kafka-ug-0014__kafka-ug-180604018_b02112214397">ssl.truststore.password</strong>: server certificate password, which must be set to <strong id="kafka-ug-0014__kafka-ug-180604018_b3211521103916">dms@kafka</strong> and cannot be changed.</li><li id="kafka-ug-0014__kafka-ug-180604018_li2057818362154"><strong id="kafka-ug-0014__kafka-ug-180604018_b163656567392">ssl.endpoint.identification.algorithm</strong>: whether to verify the certificate domain name. <strong id="kafka-ug-0014__kafka-ug-180604018_b4285516104010">This parameter must be left blank, which indicates disabling domain name verification</strong>.</li><li id="kafka-ug-0014__kafka-ug-180604018_li194571669169"><strong id="kafka-ug-0014__kafka-ug-180604018_b16925191015415">username</strong> and <strong id="kafka-ug-0014__kafka-ug-180604018_b17925141034110">password</strong>: username and password you set when enabling ciphertext access for the first time or when creating a user.</li></ul>
</li></ul>
</div>
</li><li id="kafka-ug-0014__li155191222323">Run the following command in the <strong id="kafka-ug-0014__b1134982215299">/bin</strong> directory of the Kafka client:<pre class="screen" id="kafka-ug-0014__screen17519102263211">./kafka-consumer-groups.sh --bootstrap-server ${connection-address} --offsets --describe --all-groups --command-config ../config/ssl-user-config.properties </pre>
<p id="kafka-ug-0014__p126011419592">Parameter description: <strong id="kafka-ug-0014__b11963848351251">connection-address</strong> indicates the Kafka instance address, which can be obtained in the <strong id="kafka-ug-0014__b52932843851251">Connection</strong> area on the <strong id="kafka-ug-0014__b49028672951251">Basic Information</strong> page on the Kafka console.</p>
<p id="kafka-ug-0014__p1260194165918">Example:</p>
<pre class="screen" id="kafka-ug-0014__screen12471174925911">[root@ecs-kafka bin]# ./kafka-consumer-groups.sh --bootstrap-server 192.168.xx.xx:9093,192.168.xx.xx:9093,192.168.xx.xx:9093 --offsets --describe --all-groups --command-config ../config/ssl-user-config.properties
Consumer group '__consumer-group-dial-test' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
__consumer-group-dial-test __dms_dial_test 0 347 347 0 - - -
__consumer-group-dial-test __dms_dial_test 1 347 347 0 - - -
__consumer-group-dial-test __dms_dial_test 2 347 347 0 - - -
Consumer group 'test' has no active members.
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test topic-01 0 5 5 0 - - -
test topic-01 1 3 3 0 - - -
test topic-01 2 10 10 0 - - -
[root@ecs-kafka bin]#</pre>
</li></ol>
</li></ul>
</div>
<div class="section" id="kafka-ug-0014__section149419481115"><h4 class="sectiontitle">Resetting Consumer Offsets</h4><ol id="kafka-ug-0014__ol1694563316919"><li id="kafka-ug-0014__li9945143318917"><span>Log in to the console.</span></li><li id="kafka-ug-0014__li994563315915"><span>Click <span><img id="kafka-ug-0014__image906217797" src="en-us_image_0143929918.png"></span> in the upper left corner to select a region.</span><p><div class="note" id="kafka-ug-0014__note394516330912"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="kafka-ug-0014__p1694512333917">Select the region where your Kafka instance is located.</p>
</div></div>
</p></li><li id="kafka-ug-0014__li13196447165814"><span>Click <strong id="kafka-ug-0014__b979836041">Service List</strong> and choose <strong id="kafka-ug-0014__b473603243">Application</strong> &gt; <strong id="kafka-ug-0014__b834426554">Distributed Message Service</strong>. The Kafka instance list is displayed.</span></li><li id="kafka-ug-0014__li494514336914"><span>Click the desired Kafka instance to view the instance details.</span></li><li id="kafka-ug-0014__li1694510331918"><span>In the navigation pane, choose the <strong id="kafka-ug-0014__b1276914312567">Consumer Groups</strong> tab.</span></li><li id="kafka-ug-0014__li19451233495"><span>Click the name of the desired consumer group.</span></li><li id="kafka-ug-0014__li1149819404"><span>On the <strong id="kafka-ug-0014__b10592242182315">Consumer Offset</strong> tab page, you can perform the following operations:</span><p><ul id="kafka-ug-0014__ul2584134041616"><li id="kafka-ug-0014__li11584124011165">To reset the consumer offset of all partitions of a single topic, click <strong id="kafka-ug-0014__b11297185202512">Reset Consumer Offset</strong> in the row containing the desired topic.</li><li id="kafka-ug-0014__li7736134315194">To reset the consumer offset of a single partition of a single topic, click <strong id="kafka-ug-0014__b1522941814259">Reset Consumer Offset</strong> in the row containing the desired partition.</li><li id="kafka-ug-0014__li1472164312121">To reset the consumer offset of all partitions in all topics, click <strong id="kafka-ug-0014__b14893115120258">One-touch Reset Consumer Offset</strong> above the list.</li></ul>
</p></li><li id="kafka-ug-0014__li1950985117152"><span>In the displayed <strong id="kafka-ug-0014__b661694720253">Reset Consumer Offset</strong> dialog box, set the parameters by referring to <a href="#kafka-ug-0014__table13921162119239">Table 1</a>.</span><p>
<div class="tablenoborder"><a name="kafka-ug-0014__table13921162119239"></a><a name="table13921162119239"></a><table cellpadding="4" cellspacing="0" summary="" id="kafka-ug-0014__table13921162119239" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters for resetting the consumer offset</caption><thead align="left"><tr id="kafka-ug-0014__row892110213237"><th align="left" class="cellrowborder" valign="top" width="23.21%" id="mcps1.3.6.2.8.2.1.2.3.1.1"><p id="kafka-ug-0014__p1092212214238">Parameter</p>
</th>
<th align="left" class="cellrowborder" valign="top" width="76.79%" id="mcps1.3.6.2.8.2.1.2.3.1.2"><p id="kafka-ug-0014__p1922021192315">Description</p>
</th>
</tr>
</thead>
<tbody><tr id="kafka-ug-0014__row2092252117235"><td class="cellrowborder" valign="top" width="23.21%" headers="mcps1.3.6.2.8.2.1.2.3.1.1 "><p id="kafka-ug-0014__p49221721172316">Reset By</p>
</td>
<td class="cellrowborder" valign="top" width="76.79%" headers="mcps1.3.6.2.8.2.1.2.3.1.2 "><p id="kafka-ug-0014__p1692242118231">You can reset an offset by:</p>
<ul id="kafka-ug-0014__ul84171314202711"><li id="kafka-ug-0014__li18417161432717">Time: Reset the offset to the specified time.</li><li id="kafka-ug-0014__li114421818162718">Offset: Reset the offset to the specified position.</li></ul>
<p id="kafka-ug-0014__p16547940152117">If you reset offsets in batches, they can only be reset to the specified time.</p>
</td>
</tr>
<tr id="kafka-ug-0014__row109221921102310"><td class="cellrowborder" valign="top" width="23.21%" headers="mcps1.3.6.2.8.2.1.2.3.1.1 "><p id="kafka-ug-0014__p129222021132318">Time</p>
</td>
<td class="cellrowborder" valign="top" width="76.79%" headers="mcps1.3.6.2.8.2.1.2.3.1.2 "><p id="kafka-ug-0014__p492212215230">Set this parameter if <strong id="kafka-ug-0014__b761915227272">Reset By</strong> is set to <strong id="kafka-ug-0014__b20743262275">Time</strong>.</p>
<p id="kafka-ug-0014__p1686610483293">Select a time point. After the reset is complete, retrieval starts from this time point.</p>
<ul id="kafka-ug-0014__ul1434561314174"><li id="kafka-ug-0014__li1634591371716"><strong id="kafka-ug-0014__b4521852142819">Earliest</strong>: earliest offset</li><li id="kafka-ug-0014__li1168912199177"><strong id="kafka-ug-0014__b186131877375">Custom</strong>: a custom time point</li><li id="kafka-ug-0014__li292318222176"><strong id="kafka-ug-0014__b115902295295">Latest</strong>: latest offset</li></ul>
</td>
</tr>
<tr id="kafka-ug-0014__row17922132119237"><td class="cellrowborder" valign="top" width="23.21%" headers="mcps1.3.6.2.8.2.1.2.3.1.1 "><p id="kafka-ug-0014__p092292119237">Offset</p>
</td>
<td class="cellrowborder" valign="top" width="76.79%" headers="mcps1.3.6.2.8.2.1.2.3.1.2 "><p id="kafka-ug-0014__p4922122113232">Set this parameter if <strong id="kafka-ug-0014__b15453943172915">Reset By</strong> is set to <strong id="kafka-ug-0014__b64539435294">Offset</strong>.</p>
<p id="kafka-ug-0014__p7109457133219">Enter an offset, which is greater than or equal to 0. After the reset is complete, retrieval starts from this offset.</p>
</td>
</tr>
</tbody>
</table>
</div>
</p></li><li id="kafka-ug-0014__li3479141412313"><span>Click <strong id="kafka-ug-0014__b927216403017">OK</strong>.</span></li><li id="kafka-ug-0014__li1987375812215"><span>Click <strong id="kafka-ug-0014__b378132616309">Yes</strong> in the confirmation dialog box. The consumer offset is reset.</span></li></ol>
</div>
<p id="kafka-ug-0014__p8060118"></p>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="kafka-ug-0011.html">Managing Consumer Groups</a></div>
</div>
</div>
View Git Blame Copy Permalink