doc-exports/docs/mrs/component-operation-guide/mrs_01_24299.html

<a name="mrs_01_24299"></a><a name="mrs_01_24299"></a>

<h1 class="topictitle1">Balancing Data After Kafka Node Scale-Out</h1>
<div id="body0000001219572083"><div class="section" id="mrs_01_24299__section4827236132212"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24299__p40367418151531">This section describes how to use the Kafka balancing tool on the client to balance the load of the Kafka cluster after Kafka nodes are scaled out.</p>
<p id="mrs_01_24299__p1944684185813">This section applies to versions earlier than MRS 3.<em id="mrs_01_24299__i8694031073">x</em>. For MRS 3.<em id="mrs_01_24299__i2365004112">x</em> or later, see <a href="mrs_01_1040.html">Kafka Balancing Tool Instructions</a>.</p>
</div>
<div class="section" id="mrs_01_24299__section931651784616"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24299__ul19139195714561"><li id="mrs_01_24299__li26261426155215">The system administrator has understood service requirements and prepared a Kafka administrator (belonging to the <strong id="mrs_01_24299__b187977461522724">kafkaadmin</strong> group and not required for the normal mode).</li><li id="mrs_01_24299__li71391357135619">The Kafka client has been installed, for example, in the <strong id="mrs_01_24299__b6206125217816">/opt/kafkaclient</strong> directory.</li><li id="mrs_01_24299__li1913955785616">Two topics named <strong id="mrs_01_24299__b114918203919">test_2</strong> and <strong id="mrs_01_24299__b839792315911">test_3</strong> has been created by referring to <a href="mrs_01_0376.html#mrs_01_0376__lef5a65dfacd94aca8c9991c442b4a360">7</a>. The <strong id="mrs_01_24299__b20445343495">move-kafka-topic.json</strong> file has been created in the <strong id="mrs_01_24299__b22191458191014">/opt/kafkaclient/Kafka/kafka</strong> directory. The topic format is as follows:<pre class="screen" id="mrs_01_24299__screen0155174711503">{
"topics":
[{"topic":"test_2"},{"topic":"test_3"}],
"version":1
}</pre>
</li></ul>
</div>
<div class="section" id="mrs_01_24299__section1232025214223"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24299__ol124875612551"><li id="mrs_01_24299__li122480564554"><span>Log in to the node where the Kafka client is installed as the client installation user.</span></li><li id="mrs_01_24299__li16918114569"><span>Run the following command to switch to the client installation directory:</span><p><p id="mrs_01_24299__p1735010414542"><strong id="mrs_01_24299__b435018418540">cd /opt/kafkaclient</strong></p>
</p></li><li id="mrs_01_24299__li196159493194"><span>Run the following command to set environment variables:</span><p><p id="mrs_01_24299__p1359972013202"><strong id="mrs_01_24299__b1920152216">source bigdata_env</strong></p>
</p></li><li id="mrs_01_24299__li15295192685612"><span>Run the following command to perform user authentication (skip this step if the cluster is in normal mode):</span><p><p id="mrs_01_24299__p735004119545"><strong id="mrs_01_24299__b34792343522724">kinit</strong> <em id="mrs_01_24299__i83966117922724">Component service user</em></p>
</p></li><li id="mrs_01_24299__li161674215560"><span>Run the following command to go to the <strong id="mrs_01_24299__b515519287138">bin</strong> directory of the Kafka client:</span><p><p id="mrs_01_24299__p4202205010398"><strong id="mrs_01_24299__b18463951104813">cd Kafka/kafka/bin</strong></p>
</p></li><li id="mrs_01_24299__li1423719131942"><a name="mrs_01_24299__li1423719131942"></a><a name="li1423719131942"></a><span>Run the following command to generate an execution plan:</span><p><p id="mrs_01_24299__p11259201313410"><strong id="mrs_01_24299__b1025018131545">./kafka-reassign-partitions.sh --zookeeper  </strong><em id="mrs_01_24299__i183361512161120">172.16.0.119:</em><strong id="mrs_01_24299__b5864201271120">2181</strong><strong id="mrs_01_24299__b152502131747">/kafka --topics-to-move-json-file ../move-kafka-topic.json --broker-list "</strong><em id="mrs_01_24299__i32503131948">1,2,3</em><strong id="mrs_01_24299__b122591513944">" --generate</strong></p>
<div class="note" id="mrs_01_24299__note162599139411"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24299__ul1226387187"><li id="mrs_01_24299__li1822616818188"><strong id="mrs_01_24299__b11691164820137">172.16.0.119</strong>: service IP address of the ZooKeeper instance</li><li id="mrs_01_24299__li112263819189"><strong id="mrs_01_24299__b998811461415">--broker-list "1,2,3"</strong>: list of broker instances. <strong id="mrs_01_24299__b103551943121419">1,2,3</strong> indicates all broker IDs after a scale-out.</li></ul>
</div></div>
<p id="mrs_01_24299__p852673011412"><span><img id="mrs_01_24299__image10526203041414" src="en-us_image_0000001349289889.png"></span></p>
</p></li><li id="mrs_01_24299__li324111163179"><span>Run the <strong id="mrs_01_24299__b105723918235">vim ../reassignment.json</strong> command to create the <strong id="mrs_01_24299__b189561512192316">reassignment.json</strong> file and save it to the <strong id="mrs_01_24299__b196059202234">/opt/kafkaclient/Kafka/kafka</strong> directory.</span><p><div class="p" id="mrs_01_24299__p8481191841719">Copy the content under <strong id="mrs_01_24299__b146701345241">Proposed partition reassignment configuration</strong> generated in <a href="#mrs_01_24299__li1423719131942">6</a> to the <strong id="mrs_01_24299__b1483710584242">reassignment.json</strong> file, as shown in the follows:<pre class="screen" id="mrs_01_24299__screen6999622181616">{"version":1,"partitions":[{"topic":"test","partition":4,"replicas":[1,2],"log_dirs":["any","any"]},{"topic":"test","partition":1,"replicas":[1,3],"log_dirs":["any","any"]},{"topic":"test","partition":3,"replicas":[3,1],"log_dirs":["any","any"]},{"topic":"test","partition":0,"replicas":[3,2],"log_dirs":["any","any"]},{"topic":"test","partition":2,"replicas":[2,1],"log_dirs":["any","any"]}]}</pre>
</div>
</p></li><li id="mrs_01_24299__li540619226338"><span>Run the following command to redistribute partitions:</span><p><p id="mrs_01_24299__p181932049153315"><strong id="mrs_01_24299__b162239437212">./kafka-reassign-partitions.sh --zookeeper </strong><em id="mrs_01_24299__i1815703717236">172.16.0.119:</em><strong id="mrs_01_24299__b15163153810237">2181</strong><strong id="mrs_01_24299__b72140521376">/kafka --reassignment-json-file ../reassignment.json --execute --throttle </strong><em id="mrs_01_24299__i196381852143719">50000000</em></p>
<div class="note" id="mrs_01_24299__note186451665512"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24299__p186601615554"><strong id="mrs_01_24299__b137463113253">--throttle 50000000</strong>: The maximum bandwidth is 50 MB/s. You can change the bandwidth based on the data volume and the customer's requirements on the balancing time. If the data volume is 5 TB, the bandwidth is 50 MB/s and the data balancing takes about 8 hours.</p>
</div></div>
<p id="mrs_01_24299__p119051257121414"><span><img id="mrs_01_24299__image89041157141410" src="en-us_image_0000001349170305.png"></span></p>
</p></li><li id="mrs_01_24299__li1030051193412"><span>Run the following command to check the data migration status:</span><p><p id="mrs_01_24299__p822664173515"><strong id="mrs_01_24299__b201413291125">./kafka-reassign-partitions.sh --zookeeper </strong><em id="mrs_01_24299__i630584312239">172.16.0.119:</em><strong id="mrs_01_24299__b14309043132314">2181</strong><strong id="mrs_01_24299__b214152912217">/kafka --reassignment-json-file ../reassignment.json --verify</strong></p>
<p id="mrs_01_24299__p115391849125612"><span><img id="mrs_01_24299__image14167175145619" src="en-us_image_0000001438962057.png"></span></p>
</p></li></ol>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0375.html">Using Kafka</a></div>
</div>
</div>