forked from docs/doc-exports
Yang, Tong
48706b7552
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
67 lines
17 KiB
HTML
67 lines
17 KiB
HTML
<a name="mrs_01_24053"></a><a name="mrs_01_24053"></a>
|
|
|
|
<h1 class="topictitle1">Using the ClickHouse Data Migration Tool</h1>
|
|
<div id="body32001227"><p id="mrs_01_24053__en-us_topic_0000001173789598_p176811527111117">The ClickHouse data migration tool can migrate some partitions of one or more partitioned MergeTree tables on several ClickHouseServer nodes to the same tables on other ClickHouseServer nodes. In the capacity expansion scenario, you can use this tool to migrate data from an original node to a new node to balance data after capacity expansion.</p>
|
|
<div class="section" id="mrs_01_24053__en-us_topic_0000001173789598_section135233106402"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24053__en-us_topic_0000001173789598_ul1366411569378"><li id="mrs_01_24053__en-us_topic_0000001173789598_li16664115614371">The ClickHouse and Zookeeper services are running properly. The ClickHouseServer instances on the source and destination nodes are normal.</li><li id="mrs_01_24053__en-us_topic_0000001173789598_li13180716392">The destination node has the data table to be migrated and the table is a partitioned MergeTree table.</li><li id="mrs_01_24053__en-us_topic_0000001173789598_li68491131104015">Before creating a migration task, ensure that all tasks for writing data to a table to be migrated have been stopped. After the task is started, you can only query the table to be migrated and cannot write data to or delete data from the table. Otherwise, data may be inconsistent before and after the migration.</li><li id="mrs_01_24053__en-us_topic_0000001173789598_li99501938124016">If automatic balancing is enabled, only a partitioned ReplicatedMergeTree table is migrated, and the partitioned table must have a corresponding distributed table.</li><li id="mrs_01_24053__en-us_topic_0000001173789598_li810881410398">The ClickHouse data directory on the destination node has sufficient space.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_24053__en-us_topic_0000001173789598_section16541460415"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_24053__en-us_topic_0000001173789598_ol466517515449"><li id="mrs_01_24053__en-us_topic_0000001173789598_li17664185174415"><span>Log in to Manager and choose <strong id="mrs_01_24053__en-us_topic_0000001173789598_b17284112875010">Cluster</strong> > <strong id="mrs_01_24053__en-us_topic_0000001173789598_b6219031125018">Services</strong> > <strong id="mrs_01_24053__en-us_topic_0000001173789598_b2909123425018">ClickHouse</strong> > <strong id="mrs_01_24053__en-us_topic_0000001173789598_b1515693710504">Data Migration</strong>. On the displayed page, click <strong id="mrs_01_24053__en-us_topic_0000001173789598_b9556533507">Add Task</strong>.</span><p><div class="note" id="mrs_01_24053__en-us_topic_0000001173789598_note107344310133"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24053__en-us_topic_0000001173789598_ul824216269149"><li id="mrs_01_24053__en-us_topic_0000001173789598_li6989182721419">The number of created migration tasks is limited. By default, a maximum of 20 migration tasks can be created. You can modify the number of migration tasks allowed by modifying the <strong id="mrs_01_24053__en-us_topic_0000001173789598_b215384813593">max_migration_task_number</strong> configuration item on the ClickHouse configuration page of Manager. A migration task occupies a certain number of Znodes on ZooKeeper. Therefore, you are not advised to set the maximum number of migration tasks allowed to a large value.</li><li id="mrs_01_24053__en-us_topic_0000001173789598_li0242926191411">If the number of existing migration tasks exceeds the upper limit, no more migration tasks can be created. The system automatically deletes the earliest migration tasks that have been successfully executed. If no historical migration task is successfully executed, perform <a href="#mrs_01_24053__en-us_topic_0000001173789598_li246411286161">10</a> based on the site requirements and manually delete historical migration tasks.</li></ul>
|
|
</div></div>
|
|
</p></li></ol><ol start="2" id="mrs_01_24053__en-us_topic_0000001173789598_ol266495154417"><li id="mrs_01_24053__en-us_topic_0000001173789598_li18211183015203"><span>On the page for creating a migration task, set the migration task parameters. For details, see <a href="#mrs_01_24053__en-us_topic_0000001173789598_table1724256152117">Table 1</a>. After configuring the parameters, click <strong id="mrs_01_24053__en-us_topic_0000001173789598_b1307182897114539">Next</strong>. If <strong id="mrs_01_24053__en-us_topic_0000001173789598_b14134476143">Automatic balancing</strong> is not enabled, go to <a href="#mrs_01_24053__en-us_topic_0000001173789598_li117062415510">3</a>. If <strong id="mrs_01_24053__en-us_topic_0000001173789598_b14220230121513">Automatic balancing</strong> is enabled, go to <a href="#mrs_01_24053__en-us_topic_0000001173789598_li16513141410554">5</a>.</span><p>
|
|
<div class="tablenoborder"><a name="mrs_01_24053__en-us_topic_0000001173789598_table1724256152117"></a><a name="en-us_topic_0000001173789598_table1724256152117"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24053__en-us_topic_0000001173789598_table1724256152117" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Migration task parameters</caption><thead align="left"><tr id="mrs_01_24053__en-us_topic_0000001173789598_row137242561217"><th align="left" class="cellrowborder" valign="top" width="16.08%" id="mcps1.3.3.3.1.2.1.2.3.1.1"><p id="mrs_01_24053__en-us_topic_0000001173789598_p9724956122115">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="83.91999999999999%" id="mcps1.3.3.3.1.2.1.2.3.1.2"><p id="mrs_01_24053__en-us_topic_0000001173789598_p472413569211">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24053__en-us_topic_0000001173789598_row1272412569214"><td class="cellrowborder" valign="top" width="16.08%" headers="mcps1.3.3.3.1.2.1.2.3.1.1 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p127241256132114">Task Name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.91999999999999%" headers="mcps1.3.3.3.1.2.1.2.3.1.2 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p16903162412233">Enter a specific task name. The value can contain 1 to 50 characters, including letters, digits, and underscores (_), and cannot be the same as that of an existing migration task.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24053__en-us_topic_0000001173789598_row23441224114717"><td class="cellrowborder" valign="top" width="16.08%" headers="mcps1.3.3.3.1.2.1.2.3.1.1 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p14344112412475">Automatic balancing</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.91999999999999%" headers="mcps1.3.3.3.1.2.1.2.3.1.2 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p42052383213">Choose whether to enable <strong id="mrs_01_24053__en-us_topic_0000001173789598_b1465124616165">Automatic balancing</strong>.</p>
|
|
<ul id="mrs_01_24053__en-us_topic_0000001173789598_ul18225812215"><li id="mrs_01_24053__en-us_topic_0000001173789598_li178220822212">If it is enabled, the system automatically selects the source and destination nodes to ensure that data in partitioned ReplicatedMergeTree tables corresponding to the distributed tables is evenly distributed on each node in the cluster.</li><li id="mrs_01_24053__en-us_topic_0000001173789598_li782211812218">If it is not enabled, you need to manually select the source and destination nodes.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24053__en-us_topic_0000001173789598_row201022590231"><td class="cellrowborder" valign="top" width="16.08%" headers="mcps1.3.3.3.1.2.1.2.3.1.1 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p9103159122314">Task Type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.91999999999999%" headers="mcps1.3.3.3.1.2.1.2.3.1.2 "><ul id="mrs_01_24053__en-us_topic_0000001173789598_ul197102410243"><li id="mrs_01_24053__en-us_topic_0000001173789598_li3719245244"><strong id="mrs_01_24053__en-us_topic_0000001173789598_b1215711143313">Scheduled Task</strong>: When the scheduled task is selected, you can set <strong id="mrs_01_24053__en-us_topic_0000001173789598_b14695294346">Started</strong> to specify a time point later than the current time to execute the task.</li><li id="mrs_01_24053__en-us_topic_0000001173789598_li1540830182410"><strong id="mrs_01_24053__en-us_topic_0000001173789598_b1446112333311">Immediate task</strong>: The task is executed immediately after it is started.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24053__en-us_topic_0000001173789598_row139217266276"><td class="cellrowborder" valign="top" width="16.08%" headers="mcps1.3.3.3.1.2.1.2.3.1.1 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p129352622714">Started</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.91999999999999%" headers="mcps1.3.3.3.1.2.1.2.3.1.2 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p89521736192719">Set this parameter when <strong id="mrs_01_24053__en-us_topic_0000001173789598_b2046884013359">Task Type</strong> is set to <strong id="mrs_01_24053__en-us_topic_0000001173789598_b4950243143517">Scheduled Task</strong>. The valid value is a time point within 90 days from now.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24053__en-us_topic_0000001173789598_row74391442144714"><td class="cellrowborder" valign="top" width="16.08%" headers="mcps1.3.3.3.1.2.1.2.3.1.1 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p1843918425477">Maximum Bandwidth</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.91999999999999%" headers="mcps1.3.3.3.1.2.1.2.3.1.2 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p660923363">Bandwidth upper limit of each ClickHouseServer node. The value ranges from 1 MB/s to 10,000 MB/s. In automatic balancing scenarios, increase the bandwidth as much as possible. Flow control is disabled by default.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24053__en-us_topic_0000001173789598_row243954214472"><td class="cellrowborder" valign="top" width="16.08%" headers="mcps1.3.3.3.1.2.1.2.3.1.1 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p19440174215479">Data Amount Migrated</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="83.91999999999999%" headers="mcps1.3.3.3.1.2.1.2.3.1.2 "><p id="mrs_01_24053__en-us_topic_0000001173789598_p104400425475">Percentage of the amount of data migrated in each table to the total amount of data in the table. The value ranges from 0 to 100%. If this parameter is left blank, the value is set to 50% by default. This parameter is valid only when <strong id="mrs_01_24053__en-us_topic_0000001173789598_b12673151632413">Automatic balancing</strong> is disabled.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_24053__en-us_topic_0000001173789598_li117062415510"><a name="mrs_01_24053__en-us_topic_0000001173789598_li117062415510"></a><a name="en-us_topic_0000001173789598_li117062415510"></a><span>On the <strong id="mrs_01_24053__en-us_topic_0000001173789598_b1214899192517">Select Node</strong> page, select the data table to be migrated from the list on the left and click <strong id="mrs_01_24053__en-us_topic_0000001173789598_b12268428122518">OK</strong>. In the list on the right, select the source node of the selected data table and click <strong id="mrs_01_24053__en-us_topic_0000001173789598_b963681682611">Next</strong>.</span><p><div class="note" id="mrs_01_24053__en-us_topic_0000001173789598_note1360220561254"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24053__en-us_topic_0000001173789598_p11603205692517">After a node is selected, other nodes that are replicas of the selected node are automatically selected as the source nodes.</p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_24053__en-us_topic_0000001173789598_li962162204719"><span>On the <strong id="mrs_01_24053__en-us_topic_0000001173789598_b1089472372819">Select Data Table</strong> page, select the host name of the destination node and click <strong id="mrs_01_24053__en-us_topic_0000001173789598_b557756132816">Next</strong>.</span><p><div class="note" id="mrs_01_24053__en-us_topic_0000001173789598_note131789372488"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24053__en-us_topic_0000001173789598_p256119347315">The destination node must be different from the source nodes. The selected source nodes are not displayed on the page.</p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_24053__en-us_topic_0000001173789598_li16513141410554"><a name="mrs_01_24053__en-us_topic_0000001173789598_li16513141410554"></a><a name="en-us_topic_0000001173789598_li16513141410554"></a><span>Confirm the task information and click <strong id="mrs_01_24053__en-us_topic_0000001173789598_b8422124224619">Submit</strong>.</span><p><p id="mrs_01_24053__en-us_topic_0000001173789598_p41571131164719">The data migration tool automatically calculates the partitions to be migrated based on the size of the data table to be migrated and the value of <strong id="mrs_01_24053__en-us_topic_0000001173789598_b19782128143118">Data Amount Migrated</strong> set on the <strong id="mrs_01_24053__en-us_topic_0000001173789598_b19590356173018">Add Task</strong> page.</p>
|
|
</p></li><li id="mrs_01_24053__en-us_topic_0000001173789598_li13031048164711"><span>After the migration task is submitted, click <strong id="mrs_01_24053__en-us_topic_0000001173789598_b693332315474">Start</strong> in the <strong id="mrs_01_24053__en-us_topic_0000001173789598_b1555362514479">Operation </strong>column. If the task is an immediate task, the task starts to be executed. If the task is a scheduled task, the countdown starts.</span></li><li id="mrs_01_24053__en-us_topic_0000001173789598_li4668164114010"><span>During the migration task execution, you can click <strong id="mrs_01_24053__en-us_topic_0000001173789598_b3887268022369">Cancel</strong> to cancel the migration task. If the migration task is canceled, the migrated data on the destination node will not be rolled back.</span><p><div class="caution" id="mrs_01_24053__en-us_topic_0000001173789598_note257622204010"><span class="cautiontitle"><img src="public_sys-resources/caution_3.0-en-us.png"> </span><div class="cautionbody"><p id="mrs_01_24053__en-us_topic_0000001173789598_p3576322104015">After a task with automatic balancing enabled is canceled, it will not stop immediately. It will stop after the table migration is complete. After a task with automatic balancing disabled is canceled, the task stops immediately. After the task stops, a partition may have been migrated to the destination node, but it is not deleted from the source node. In this case, duplicate data exists. Manually check whether the migrated partition still exists on the source node. If it still exists, check that the total data volume of the partition on the destination node is the same as that on the source node, and then delete the partition from the source node.</p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_24053__en-us_topic_0000001173789598_li131211658171517"><span>Choose <strong id="mrs_01_24053__en-us_topic_0000001173789598_b56728502134">More</strong> > <strong id="mrs_01_24053__en-us_topic_0000001173789598_b8504652111319">Details</strong> to view details about the migration task.</span></li><li id="mrs_01_24053__en-us_topic_0000001173789598_li4288165012311"><span>After the migration is complete, choose <strong id="mrs_01_24053__en-us_topic_0000001173789598_b17403139145">More</strong> > <strong id="mrs_01_24053__en-us_topic_0000001173789598_b34095341413">Results</strong> to view the migration result.</span><p><p id="mrs_01_24053__en-us_topic_0000001173789598_p134414516234">In a non-automatic balancing task, you can view the migrated partitions of each table and the partition migration result. If the partition migration is not finished, the partition has been copied to the destination node but not deleted from the source node because the data volume of the partition on the source node is inconsistent with that of the partition on the destination node. In this case, check whether the data volume of the partition on the source node is consistent with that of the partition on the destination node, and then delete the partition from the source node.</p>
|
|
</p></li><li id="mrs_01_24053__en-us_topic_0000001173789598_li246411286161"><a name="mrs_01_24053__en-us_topic_0000001173789598_li246411286161"></a><a name="en-us_topic_0000001173789598_li246411286161"></a><span>After the migration is complete, choose <strong id="mrs_01_24053__en-us_topic_0000001173789598_b147221042181411">More</strong> > <strong id="mrs_01_24053__en-us_topic_0000001173789598_b9560144121419">Delete</strong> to delete the directories related to the migration task on ZooKeeper and the destination node.</span></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24250.html">Migrating ClickHouse Data</a></div>
|
|
</div>
|
|
</div>
|
|
|