forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
46 lines
11 KiB
HTML
46 lines
11 KiB
HTML
<a name="mrs_01_24534"></a><a name="mrs_01_24534"></a>
|
|
|
|
<h1 class="topictitle1">Migrating Data Between Kafka Nodes</h1>
|
|
<div id="body0000001215605640"><div class="section" id="mrs_01_24534__section3106125805312"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24534__p456912112294">This section describes how to use Kafka client commands to migrate partition data between disks on a node without stopping the Kafka service.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24534__section1526183519545"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24534__ul556961116291"><li id="mrs_01_24534__li105691711182915">The system administrator has understood service requirements and prepared a Kafka user (belonging to the <strong id="mrs_01_24534__b213212352022235">kafkaadmin</strong> group and not required for the normal mode).</li><li id="mrs_01_24534__li5569121111297">The Kafka client has been installed.</li><li id="mrs_01_24534__li11694523193118">The Kafka instance status and disk status are normal.</li><li id="mrs_01_24534__li654214467344">Based on the current disk space usage of the partition to be migrated, ensure that the disk space will be sufficient after the migration.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_24534__section42321249555"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24534__ol82491327205517"><li id="mrs_01_24534__li1924932710556"><span>Log in as a client installation user to the node on which the Kafka client is installed.</span></li><li id="mrs_01_24534__li52492273553"><span>Run the following command to switch to the Kafka client installation directory, for example, <strong id="mrs_01_24534__b10376195317337">/opt/kafkaclient</strong>:</span><p><p id="mrs_01_24534__p053212717416"><strong id="mrs_01_24534__b1674721043">cd /opt/kafkaclient</strong></p>
|
|
</p></li><li id="mrs_01_24534__li1724972775516"><span>Run the following command to set environment variables:</span><p><p id="mrs_01_24534__p45691311152914"><strong id="mrs_01_24534__b7569311132919">source bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_24534__li62499277557"><span>Run the following command to authenticate the user (skip this step in normal mode):</span><p><p id="mrs_01_24534__p25691711122914"><strong id="mrs_01_24534__b256941118299">kinit</strong> <em id="mrs_01_24534__i2569181162917">Component service user</em></p>
|
|
</p></li><li id="mrs_01_24534__li2024918275556"><span>Run the following command to switch to the Kafka client directory:</span><p><p id="mrs_01_24534__p1561195265311"><strong id="mrs_01_24534__b861145212538">cd Kafka/kafka/bin</strong></p>
|
|
</p></li><li id="mrs_01_24534__li420725319552"><a name="mrs_01_24534__li420725319552"></a><a name="li420725319552"></a><span>Run the following command to view the topic details of the partition to be migrated:</span><p><p id="mrs_01_24534__p1689515228115"><strong id="mrs_01_24534__b17807106438">Security mode:</strong></p>
|
|
<p id="mrs_01_24534__p1515585185418"><strong id="mrs_01_24534__b100510922622235">./kafka-topics.sh --describe --bootstrap-server</strong> <em id="mrs_01_24534__i97417223822235">IP address of the Kafka</em><em id="mrs_01_24534__i163191358722235">cluster:21007</em> <strong id="mrs_01_24534__b114947744622235">--command-config ../config/client.properties</strong> <strong id="mrs_01_24534__b98843515522235">--topic</strong> <em id="mrs_01_24534__i8854092822235">topic name</em></p>
|
|
<p id="mrs_01_24534__p187625281919"><strong id="mrs_01_24534__b589419316434">Normal mode:</strong></p>
|
|
<p id="mrs_01_24534__p14291425722"><strong id="mrs_01_24534__b9827625026">./kafka-topics.sh --describe --bootstrap-server</strong> <em id="mrs_01_24534__i1982742520219">IP address of the Kafka cluster</em><em id="mrs_01_24534__i1682716252211">:21005</em> <strong id="mrs_01_24534__b6827132516217">--command-config ../config/client.properties</strong> <strong id="mrs_01_24534__b78271625721">--topic</strong> <em id="mrs_01_24534__i882718251720">Topic name</em></p>
|
|
<p id="mrs_01_24534__p1644215419016"><span><img id="mrs_01_24534__image270174219011" src="en-us_image_0000001349289873.png"></span></p>
|
|
</p></li><li id="mrs_01_24534__li1824951465613"><a name="mrs_01_24534__li1824951465613"></a><a name="li1824951465613"></a><span>Run the following command to query the mapping between <strong id="mrs_01_24534__b3298154613352">Broker_ID</strong> and the IP address:</span><p><p id="mrs_01_24534__p159254173215"><strong id="mrs_01_24534__b169259110322">./kafka-broker-info.sh --zookeeper </strong><em id="mrs_01_24534__i120223393622235">IP address of the ZooKeeper quorumpeer instance</em>:<em id="mrs_01_24534__i3275125422235">ZooKeeper port number</em><strong id="mrs_01_24534__b15174907022235">/kafka</strong></p>
|
|
<pre class="screen" id="mrs_01_24534__screen16625165832712">Broker_ID IP_Address
|
|
--------------------------
|
|
4 192.168.0.100
|
|
5 192.168.0.101
|
|
6 192.168.0.102</pre>
|
|
<div class="p" id="mrs_01_24534__p1720264311514"><div class="note" id="mrs_01_24534__note183961743134015"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24534__ul4278192283814"><li id="mrs_01_24534__li024511411817">IP address of the ZooKeeper quorumpeer instance<p id="mrs_01_24534__p92456140814"><a name="mrs_01_24534__li024511411817"></a><a name="li024511411817"></a>To obtain IP addresses of all ZooKeeper quorumpeer instances, log in to FusionInsight Manager and choose <strong id="mrs_01_24534__b173347918022235">Cluster</strong> > <strong id="mrs_01_24534__b147420489322235">Services</strong> > <strong id="mrs_01_24534__b107933900022235">ZooKeeper</strong>. On the displayed page, click <strong id="mrs_01_24534__b62221762422235">Instance</strong> and view the IP addresses of all the hosts where the quorumpeer instances locate.</p>
|
|
</li><li id="mrs_01_24534__li624515142816">Port number of the ZooKeeper client<p id="mrs_01_24534__p1724512142082"><a name="mrs_01_24534__li624515142816"></a><a name="li624515142816"></a>Log in to FusionInsight Manager and choose <strong id="mrs_01_24534__b72253318822235">Cluster</strong> > <strong id="mrs_01_24534__b9594769622235">Service</strong> > <strong id="mrs_01_24534__b85687752422235">ZooKeeper</strong>. On the displayed page, click <strong id="mrs_01_24534__b17324332022235">Configurations</strong> and check the value of <strong id="mrs_01_24534__b64103511722235">clientPort</strong>. The default value is <strong id="mrs_01_24534__b146692992922235">24002</strong>.</p>
|
|
</li></ul>
|
|
</div></div>
|
|
</div>
|
|
</p></li><li id="mrs_01_24534__li1230081019282"><a name="mrs_01_24534__li1230081019282"></a><a name="li1230081019282"></a><span>Obtain the partition distribution and node information from the command output in <a href="#mrs_01_24534__li420725319552">6</a> and <a href="#mrs_01_24534__li1824951465613">7</a>, and create the JSON file for reallocation in the current directory.</span><p><div class="p" id="mrs_01_24534__p38811911182918">To migrate data in the partition whose <strong id="mrs_01_24534__b1638520173916">Broker_ID</strong> is <strong id="mrs_01_24534__b57992037143917">6</strong> to the <strong id="mrs_01_24534__b553719485381">/srv/BigData/hadoop/data1/kafka-logs</strong> directory, the required JSON configuration file is as follows:<pre class="screen" id="mrs_01_24534__screen1054859101815">{"partitions":[{"topic": "testws","partition": 2,"replicas": [6,5],"log_dirs": ["/srv/BigData/hadoop/data1/kafka-logs","any"]}],"version":1}</pre>
|
|
<div class="note" id="mrs_01_24534__note1823692273715"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24534__ul159881625104812"><li id="mrs_01_24534__li598918257488"><strong id="mrs_01_24534__b126551518408">topic</strong> indicates the topic name, for example, <strong id="mrs_01_24534__b65991158154011">testws</strong>.</li><li id="mrs_01_24534__li09892252483"><strong id="mrs_01_24534__b1953815454116">partition</strong> indicates the topic partition.</li><li id="mrs_01_24534__li10989162518481">The number in <strong id="mrs_01_24534__b122215185414">replicas</strong> corresponds to <strong id="mrs_01_24534__b17274142417413">Broker_ID</strong>.</li><li id="mrs_01_24534__li16989102519480"><strong id="mrs_01_24534__b1934643212415">log_dirs</strong> indicates the path of the disk to be migrated. In this example, <strong id="mrs_01_24534__b1376862274213">log_dirs</strong> of the node whose <strong id="mrs_01_24534__b13180111014434">Broker_ID</strong> is <strong id="mrs_01_24534__b16677322204311">5</strong> is set to <strong id="mrs_01_24534__b13261219184413">any</strong>, and that of the node whose <strong id="mrs_01_24534__b54457533430">Broker_ID</strong> is <strong id="mrs_01_24534__b16319574437">6</strong> is set to <strong id="mrs_01_24534__b07288274412">/srv/BigData/hadoop/data1/kafka-logs.</strong> Note that the path must correspond to the node.</li></ul>
|
|
</div></div>
|
|
</div>
|
|
</p></li><li id="mrs_01_24534__li826965415581"><span>Run the following command to perform reallocation:</span><p><p id="mrs_01_24534__p1487510510588"><strong id="mrs_01_24534__b43653144416">Security mode:</strong></p>
|
|
<p id="mrs_01_24534__p6424103235810"><strong id="mrs_01_24534__b17622105104413">./kafka-reassign-partitions.sh</strong><strong id="mrs_01_24534__b206231259445"> --bootstrap-server</strong><em id="mrs_01_24534__i362411524418"> Service IP address of Broker</em><strong id="mrs_01_24534__b176241257446">:21007 </strong><strong id="mrs_01_24534__b46250519441">--command-config ../config/client.properties</strong><strong id="mrs_01_24534__b76250518449"> --zookeeper </strong><em id="mrs_01_24534__i1626956440">{zk_host}:{port}</em><strong id="mrs_01_24534__b362815574412">/kafka</strong><strong id="mrs_01_24534__b20629125114418"> --reassignment-json-file</strong><em id="mrs_01_24534__i126311152441"><em id="mrs_01_24534__i1863014504414"> Path of the JSON file compiled in <a href="#mrs_01_24534__li1230081019282">8</a></em><strong id="mrs_01_24534__b163175194416"> </strong></em><strong id="mrs_01_24534__b063215544416"> --</strong><strong id="mrs_01_24534__b963315114419">execute</strong></p>
|
|
<p id="mrs_01_24534__p1977117335597"><strong id="mrs_01_24534__b9454168164415">Normal mode:</strong></p>
|
|
<p id="mrs_01_24534__p10579531705"><strong id="mrs_01_24534__b1482441518449">./kafka-reassign-partitions.sh</strong><strong id="mrs_01_24534__b1383381517440"> --bootstrap-server</strong><em id="mrs_01_24534__i17834715114410"> Service IP address of Broker</em><strong id="mrs_01_24534__b2834141504418">:21005 </strong><strong id="mrs_01_24534__b1835121517448">--command-config ../config/client.properties</strong><strong id="mrs_01_24534__b128357153446"> --zookeeper </strong><em id="mrs_01_24534__i783514158445">{zk_host}:{port}</em><strong id="mrs_01_24534__b178357151448">/kafka</strong><strong id="mrs_01_24534__b9836111574420"> --reassignment-json-file</strong><em id="mrs_01_24534__i19837111514416"><em id="mrs_01_24534__i17836171544412"> Path of the JSON file compiled in <a href="#mrs_01_24534__li1230081019282">8</a></em><strong id="mrs_01_24534__b583731514446"> </strong></em><strong id="mrs_01_24534__b78371915114414"> --</strong><strong id="mrs_01_24534__b148381615154416">execute</strong></p>
|
|
<p id="mrs_01_24534__p1142517329583">If message "Successfully started reassignment of partitions" is displayed, the execution is successful.</p>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0375.html">Using Kafka</a></div>
|
|
</div>
|
|
</div>
|
|
|