forked from docs/doc-exports
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
435 lines
67 KiB
HTML
435 lines
67 KiB
HTML
<a name="mrs_01_24508"></a><a name="mrs_01_24508"></a>
|
|
|
|
<h1 class="topictitle1">Using the Migration Tool to Quickly Migrate ClickHouse Cluster Data</h1>
|
|
<div id="body0000001583195165"><div class="note" id="mrs_01_24508__note1382418482123"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24508__p1682554813120">This section applies only to MRS 3.2.0 or later.</p>
|
|
</div></div>
|
|
<div class="section" id="mrs_01_24508__section13443205512355"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24508__p0790178365">Scenario 1: As the number of MRS ClickHouse services increases, the storage and compute resources of clusters cannot meet service requirements. Thus, the clusters need to be split so that part of user service and database data can be migrated to new clusters.</p>
|
|
<p id="mrs_01_24508__p10453719144218">Scenario 2: The data center where the backend hosts of MRS ClickHouse clusters are located needs to be migrated, so does the ClickHouse cluster data.</p>
|
|
<p id="mrs_01_24508__p159343107447">To meet the migration requirements, MRS provides a one-click data migration tool for migrating the databases, table objects (DDL), and service data of ClickHouse from a source cluster to the new cluster.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24508__section191591139175"><h4 class="sectiontitle">Migration Mechanism</h4><ul id="mrs_01_24508__ul1148114459229"><li id="mrs_01_24508__li13481114513224">Replicated*MergeTree table migration<p id="mrs_01_24508__p62931399228"><a name="mrs_01_24508__li13481114513224"></a><a name="li13481114513224"></a>In this migration solution, ClickHouse uses ZooKeeper to automatically synchronize the data of Replicated*MergeTree tables of different replicas in the same shard. The logical procedure is as follows:</p>
|
|
<div class="p" id="mrs_01_24508__p176121047112719">First, add the ZooKeeper information of the source cluster to the configuration file of the destination cluster as the auxiliary ZooKeeper. Then, in the destination cluster, create a temporary table that has the same ZooKeeper path and structure as the source cluster but different replicas from it. After the temporary table is created, data in the source cluster will be automatically synchronized to the temporary table. Once data synchronization is complete, copy data from the temporary table to the formal table.<div class="fignone" id="mrs_01_24508__fig1185145715303"><span class="figcap"><b>Figure 1 </b>Replicated*MergeTree table migration architecture</span><br><span><img id="mrs_01_24508__image5702356191015" src="en-us_image_0000001587840761.png"></span></div>
|
|
</div>
|
|
</li><li id="mrs_01_24508__li15710811184">Distributed table migration<p id="mrs_01_24508__p196071158121816"><a name="mrs_01_24508__li15710811184"></a><a name="li15710811184"></a>During the migration of a replicated table, its metadata in the source cluster is exported and changed to the ZooKeeper path and replica of the destination cluster. Then, you can create a table in the destination cluster based on the modified metadata.</p>
|
|
</li><li id="mrs_01_24508__li98089922320">Non-replicated table and materialized view migration<p id="mrs_01_24508__p226673811255"><a name="mrs_01_24508__li98089922320"></a><a name="li98089922320"></a>To migrate data in the non-replicated tables and materialized views, you can call the <strong id="mrs_01_24508__b149619253248">remote</strong> function.</p>
|
|
</li></ul>
|
|
<p id="mrs_01_24508__p123715209273">The preceding migration operations are encapsulated using the migration tool script. In this way, you can modify the related configuration files and run the migration scripts to complete the migration by one click. For details, see the procedure description.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24508__section89165113118"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_24508__ul19410750"><li id="mrs_01_24508__li89275218511">The status of the source ClickHouse cluster to be migrated is normal and the cluster in the security mode.</li><li id="mrs_01_24508__li1894005516">The destination ClickHouse cluster of MRS 3.1.3 or later has been created for data migration. The cluster must be in the security mode. The number of ClickHouserver instances in the ClickHouse cluster is greater than or equal to that of the source cluster.</li><li id="mrs_01_24508__li132656331129">Currently, logical clusters support only data migration to clusters with the same number of replicas.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_24508__section1954132835511"><h4 class="sectiontitle">Constraints</h4><ul id="mrs_01_24508__ul238871365117"><li id="mrs_01_24508__li567331633115">Only table data and table object metadata (DDL) can be migrated. SQL statements like ETL need to be manually migrated.</li><li id="mrs_01_24508__li13967101475114">To ensure data consistency before and after the migration, stop the ClickHouse service of the source cluster before the migration. For details, see the procedure description.</li><li id="mrs_01_24508__li11973185501713">If the table of the source cluster is deleted during the migration, this issue can only be solved manually.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_24508__section1517535314162"><h4 class="sectiontitle">Procedure</h4><div class="p" id="mrs_01_24508__p15345815168">The overall migration procedure is as follows:<div class="fignone" id="mrs_01_24508__fig954781195814"><span class="figcap"><b>Figure 2 </b>Migration flowchart</span><br><span><img id="mrs_01_24508__image15470119582" src="en-us_image_0000001532516862.png"></span></div>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24508__table114913502178" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Migration description</caption><thead align="left"><tr id="mrs_01_24508__row111503505172"><th align="left" class="cellrowborder" valign="top" width="34.28%" id="mcps1.3.6.2.2.2.3.1.1"><p id="mrs_01_24508__p141506507171">Step</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="65.72%" id="mcps1.3.6.2.2.2.3.1.2"><p id="mrs_01_24508__p10150195011179">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24508__row415014502177"><td class="cellrowborder" valign="top" width="34.28%" headers="mcps1.3.6.2.2.2.3.1.1 "><p id="mrs_01_24508__p11150155081717"><a href="#mrs_01_24508__section132661749155412">Step 1: Connect the source cluster to the destination cluster.</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.72%" headers="mcps1.3.6.2.2.2.3.1.2 "><p id="mrs_01_24508__p181502509173">This step ensures that the source and target ClickHouse clusters as well as their nodes can communicate with each other.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row8150115051720"><td class="cellrowborder" valign="top" width="34.28%" headers="mcps1.3.6.2.2.2.3.1.1 "><p id="mrs_01_24508__p1150150151714"><a href="#mrs_01_24508__section1669382893718">Step 2: Add the ZooKeeper information of the source cluster to the configuration file of the destination cluster.</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.72%" headers="mcps1.3.6.2.2.2.3.1.2 "><p id="mrs_01_24508__p1115055019175">By doing so, ZooKeeper in the source cluster can be used as the auxiliary ZooKeeper during data migration.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row18150250161715"><td class="cellrowborder" valign="top" width="34.28%" headers="mcps1.3.6.2.2.2.3.1.1 "><p id="mrs_01_24508__p24841126152113"><a href="#mrs_01_24508__section16918183315128">Step 3: Migrate metadata of databases and tables in the source ClickHouse cluster to the destination cluster.</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.72%" headers="mcps1.3.6.2.2.2.3.1.2 "><p id="mrs_01_24508__p8150165010175">You can run the corresponding script to migrate metadata such as the database name, table name, and table structure of the ClickHouse database and tables in the source cluster to the destination cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row19150125017173"><td class="cellrowborder" valign="top" width="34.28%" headers="mcps1.3.6.2.2.2.3.1.1 "><p id="mrs_01_24508__p151504508179"><a href="#mrs_01_24508__section18047266428">Step 4: Migrate data of the databases and tables in the source ClickHouse cluster to the destination cluster.</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.72%" headers="mcps1.3.6.2.2.2.3.1.2 "><p id="mrs_01_24508__p815095021713">You can run the corresponding script to migrate the ClickHouse database and table data from the source cluster to the destination cluster.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="mrs_01_24508__section132661749155412"><a name="mrs_01_24508__section132661749155412"></a><a name="section132661749155412"></a><h4 class="sectiontitle">Connecting the Source Cluster to the Destination Cluster</h4><ol id="mrs_01_24508__ol59721021175813"><li id="mrs_01_24508__li797292155819">Connect the source cluster to the destination cluster so that the ClickHouse instance nodes in the two clusters can communicate with each other.</li><li id="mrs_01_24508__li18306159121">Add the host configurations of the source cluster to all nodes in the destination cluster.<ol type="a" id="mrs_01_24508__ol63311242171212"><li id="mrs_01_24508__li7625172962713">Log in to FusionInsight Manager of the source ClickHouse cluster, choose <strong id="mrs_01_24508__b1057418240105">Cluster</strong> > <strong id="mrs_01_24508__b15427143211010">ClickHouse</strong>, click the <strong id="mrs_01_24508__b572317475107">Instance</strong> tab, and view the service IP address of the ClickHouseServer instance node.</li><li id="mrs_01_24508__li1990635315277"><a name="mrs_01_24508__li1990635315277"></a><a name="li1990635315277"></a>Log in to any ClickHouseServer node using SSH and run the following command to check the host configurations of the ClickHouse instance in the source cluster:<p id="mrs_01_24508__p8861153682915"><a name="mrs_01_24508__li1990635315277"></a><a name="li1990635315277"></a><strong id="mrs_01_24508__b1621249132916">cat /etc/hosts</strong></p>
|
|
<p id="mrs_01_24508__p07631025183013">The following figure shows the host configurations of the ClickHouse instance:</p>
|
|
<p id="mrs_01_24508__p185361063334"><span><img id="mrs_01_24508__image16536146103315" src="en-us_image_0000001532836098.png"></span></p>
|
|
</li><li id="mrs_01_24508__li17786173195919">Log in to FusionInsight Manager of the destination ClickHouse cluster, choose <strong id="mrs_01_24508__b113411452151316">Cluster</strong> > <strong id="mrs_01_24508__b1334115218133">ClickHouse</strong>, click the <strong id="mrs_01_24508__b134175221313">Instance</strong> tab, and view the service IP address of the ClickHouseServer instance node in the destination cluster.</li><li id="mrs_01_24508__li3155143763511">Log in to all ClickHouse nodes in the destination cluster as user <strong id="mrs_01_24508__b547315111145">root</strong> and run the following command to modify the <strong id="mrs_01_24508__b368429181510">/etc/hosts</strong> configuration of the nodes:<p id="mrs_01_24508__p627015142363"><strong id="mrs_01_24508__b4508181293610">vi </strong><strong id="mrs_01_24508__b12469133617">/etc/hosts</strong></p>
|
|
<p id="mrs_01_24508__p177981416193614">Copy the host information of the ClickHouse instance of the source cluster obtained in <a href="#mrs_01_24508__li1990635315277">2.b</a> to the <strong id="mrs_01_24508__b133517257163">hosts</strong> file.</p>
|
|
</li></ol>
|
|
</li><li id="mrs_01_24508__li6983193663413">Configure mutual trust between the source and destination clusters.</li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_24508__section1669382893718"><a name="mrs_01_24508__section1669382893718"></a><a name="section1669382893718"></a><h4 class="sectiontitle">Adding the ZooKeeper Information of the Source Cluster to the Configuration File of the Destination Cluster</h4><ol id="mrs_01_24508__ol5301134793814"><li id="mrs_01_24508__li13474131194310"><a name="mrs_01_24508__li13474131194310"></a><a name="li13474131194310"></a>Log in to FusionInsight Manager of the source cluster, choose <strong id="mrs_01_24508__b112394618437">Cluster</strong> > <strong id="mrs_01_24508__b686361019436">Services</strong> > <strong id="mrs_01_24508__b18298514184310">ZooKeeper</strong>, and click the <strong id="mrs_01_24508__b203061124174315">Instance</strong> tab. On the displayed page, view the service IP addresses of the ZooKeeper quorumpeer instance , as shown in <a href="#mrs_01_24508__fig39818132540">Figure 3</a>.<p id="mrs_01_24508__p951964555517"></p>
|
|
<div class="fignone" id="mrs_01_24508__fig39818132540"><a name="mrs_01_24508__fig39818132540"></a><a name="fig39818132540"></a><span class="figcap"><b>Figure 3 </b>Addresses of the source Zookeeper quorumpeer instance</span><br><span><img id="mrs_01_24508__image670410377428" src="en-us_image_0000001537090654.png"></span></div>
|
|
</li><li id="mrs_01_24508__li1315311694620"><a name="mrs_01_24508__li1315311694620"></a><a name="li1315311694620"></a>Log in to FusionInsight Manager of the destination cluster, choose <strong id="mrs_01_24508__b9905123075015">Cluster</strong> > <strong id="mrs_01_24508__b132151234155010">Services</strong> > <strong id="mrs_01_24508__b12312337155018">ClickHouse</strong> and click the <strong id="mrs_01_24508__b142766018309">Configurations</strong> tab and then <strong id="mrs_01_24508__b5400351115014">All Configurations</strong>. On the displayed page, search for the <strong id="mrs_01_24508__b1524881418519">clickhouse-config-customize</strong> parameter.</li><li id="mrs_01_24508__li16763133318475"><a name="mrs_01_24508__li16763133318475"></a><a name="li16763133318475"></a>Add ZooKeeper instance information of the source cluster to the <strong id="mrs_01_24508__b19903151816534">clickhouse-config-customize</strong> parameter by referring to <a href="#mrs_01_24508__table6772947124819">Table 2</a>.
|
|
<div class="tablenoborder"><a name="mrs_01_24508__table6772947124819"></a><a name="table6772947124819"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24508__table6772947124819" width="90%" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Configurations of <strong id="mrs_01_24508__b149421468533">clickhouse-config-customize</strong></caption><thead align="left"><tr id="mrs_01_24508__row47724477484"><th align="left" class="cellrowborder" valign="top" width="33.879999999999995%" id="mcps1.3.8.2.3.3.2.3.1.1"><p id="mrs_01_24508__p2771174713483">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="66.12%" id="mcps1.3.8.2.3.3.2.3.1.2"><p id="mrs_01_24508__p57721647144810">Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24508__row1277216472486"><td class="cellrowborder" valign="top" width="33.879999999999995%" headers="mcps1.3.8.2.3.3.2.3.1.1 "><p id="mrs_01_24508__p1077284714813">auxiliary_zookeepers.zookeeper2.node[1].host</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.12%" headers="mcps1.3.8.2.3.3.2.3.1.2 "><p id="mrs_01_24508__p147726473482">Service IP address of the first ZooKeeper quorumpeer instance in the source cluster obtained in <a href="#mrs_01_24508__li13474131194310">1</a>. Currently, only the IP address of the ZooKeeper instance can be configured.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row3772134712489"><td class="cellrowborder" valign="top" width="33.879999999999995%" headers="mcps1.3.8.2.3.3.2.3.1.1 "><p id="mrs_01_24508__p1777214717485">auxiliary_zookeepers.zookeeper2.node[1].port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.12%" headers="mcps1.3.8.2.3.3.2.3.1.2 "><p id="mrs_01_24508__p17772194710481">2181</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row1377284714819"><td class="cellrowborder" valign="top" width="33.879999999999995%" headers="mcps1.3.8.2.3.3.2.3.1.1 "><p id="mrs_01_24508__p377254774819">auxiliary_zookeepers.zookeeper2.node[2].host</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.12%" headers="mcps1.3.8.2.3.3.2.3.1.2 "><p id="mrs_01_24508__p4177153612526">Service IP address of the second ZooKeeper quorumpeer instance in the source cluster obtained in <a href="#mrs_01_24508__li13474131194310">1</a>. Currently, only the IP address of the ZooKeeper instance can be configured.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row7772104718487"><td class="cellrowborder" valign="top" width="33.879999999999995%" headers="mcps1.3.8.2.3.3.2.3.1.1 "><p id="mrs_01_24508__p8772747154816">auxiliary_zookeepers.zookeeper2.node[2].port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.12%" headers="mcps1.3.8.2.3.3.2.3.1.2 "><p id="mrs_01_24508__p181369565115">2181</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row177219479484"><td class="cellrowborder" valign="top" width="33.879999999999995%" headers="mcps1.3.8.2.3.3.2.3.1.1 "><p id="mrs_01_24508__p5772134715487">auxiliary_zookeepers.zookeeper2.node[3].host</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.12%" headers="mcps1.3.8.2.3.3.2.3.1.2 "><p id="mrs_01_24508__p3760105035218">Service IP address of the third ZooKeeper quorumpeer instance in the source cluster obtained in <a href="#mrs_01_24508__li13474131194310">1</a>. Currently, only the IP address of the ZooKeeper instance can be configured.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row17772647174812"><td class="cellrowborder" valign="top" width="33.879999999999995%" headers="mcps1.3.8.2.3.3.2.3.1.1 "><p id="mrs_01_24508__p1977274711485">auxiliary_zookeepers.zookeeper2.node[3].port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.12%" headers="mcps1.3.8.2.3.3.2.3.1.2 "><p id="mrs_01_24508__p35212665111">2181</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_24508__li11302747173816">After the configuration is complete, click <strong id="mrs_01_24508__b108680546577">Save</strong>. In the displayed dialog box, click <strong id="mrs_01_24508__b893071615810">OK</strong>.</li><li id="mrs_01_24508__li17679261131"><a name="mrs_01_24508__li17679261131"></a><a name="li17679261131"></a>Log in to any ClickHouseServer node of the destination cluster as user <strong id="mrs_01_24508__b834185118588">root</strong>. Run the following command to view the ClickHouseServer instance information:<p id="mrs_01_24508__p17677217543"><strong id="mrs_01_24508__b44701231246">ps -ef |grep clickhouse</strong></p>
|
|
<div class="p" id="mrs_01_24508__p8464134917412">Obtain the value of <strong id="mrs_01_24508__b7321397593">--config-file</strong>, that is, the configuration file directory of the ClickHouseServer, from the query result.<div class="fignone" id="mrs_01_24508__fig83055442712"><span class="figcap"><b>Figure 4 </b>Obtaining the configuration file directory of ClickHouseServer</span><br><span><img id="mrs_01_24508__image3308923194617" src="en-us_image_0000001537413022.png"></span></div>
|
|
</div>
|
|
</li><li id="mrs_01_24508__li1248013418211">Run the corresponding command to check whether the information about <strong id="mrs_01_24508__b06191356713"><auxiliary_zookeepers></strong> is added in the ClickHouse configuration file <strong id="mrs_01_24508__b138013201317">config.xml</strong>.<div class="p" id="mrs_01_24508__p20604641380"><strong id="mrs_01_24508__b1459313281890">cat </strong><em id="mrs_01_24508__i593010328918">Directory of the config.xml file obtained in <a href="#mrs_01_24508__li17679261131">5</a></em><div class="fignone" id="mrs_01_24508__fig077220599717"><span class="figcap"><b>Figure 5 </b>Viewing the added ZooKeeper information of the source cluster</span><br><span><img id="mrs_01_24508__image47721159775" src="en-us_image_0000001583195985.png"></span></div>
|
|
</div>
|
|
</li><li id="mrs_01_24508__li11758429194015"><a name="mrs_01_24508__li11758429194015"></a><a name="li11758429194015"></a>In the configuration file directory obtained in <a href="#mrs_01_24508__li17679261131">5</a>, run the following command to obtain the ZooKeeper authentication information of the source cluster:<p id="mrs_01_24508__p892217120416"><strong id="mrs_01_24508__b433819169411">cat ENV_VARS | grep ZK</strong></p>
|
|
<p id="mrs_01_24508__p7253123119456"><span><img id="mrs_01_24508__image725363114454" src="en-us_image_0000001583316321.png"></span></p>
|
|
<p id="mrs_01_24508__p1522724819452">Obtain the values of <strong id="mrs_01_24508__b1381218529178">ZK_SERVER_FQDN</strong>, <strong id="mrs_01_24508__b167121957141715">ZK_USER_PRINCIPAL</strong> and <strong id="mrs_01_24508__b128886591818">ZK_USER_REALM</strong>.</p>
|
|
</li><li id="mrs_01_24508__li1499011385018">Log in to FusionInsight Manager of the destination cluster, choose <strong id="mrs_01_24508__b1757310916229">Cluster</strong> > <strong id="mrs_01_24508__b93201421202211">Services</strong> > <strong id="mrs_01_24508__b151211028102219">ClickHouse</strong>, click the <strong id="mrs_01_24508__b17878155432213">Configurations</strong> tab and then <strong id="mrs_01_24508__b47761291237">All Configurations</strong>. In the navigation pane on the left, choose <strong id="mrs_01_24508__b10247182716249">ClickHouseServer(Role)</strong> > <strong id="mrs_01_24508__b531742982520">Backup</strong> and set the parameters by referring to the following table.
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24508__table24761399518" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Configuring the cluster authentication information</caption><thead align="left"><tr id="mrs_01_24508__row14477133911517"><th align="left" class="cellrowborder" valign="top" width="28.87%" id="mcps1.3.8.2.8.8.2.3.1.1"><p id="mrs_01_24508__p13477163911519">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="71.13000000000001%" id="mcps1.3.8.2.8.8.2.3.1.2"><p id="mrs_01_24508__p1747793905113">Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24508__row747713965118"><td class="cellrowborder" valign="top" width="28.87%" headers="mcps1.3.8.2.8.8.2.3.1.1 "><p id="mrs_01_24508__p13477163905119">AUXILIARY_ZK_SERVER_FQDN</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.13000000000001%" headers="mcps1.3.8.2.8.8.2.3.1.2 "><p id="mrs_01_24508__p154771939195110">Value of <strong id="mrs_01_24508__b3402143814321">ZK_SERVER_FQDN</strong> obtained in <a href="#mrs_01_24508__li11758429194015">7</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row10477239185111"><td class="cellrowborder" valign="top" width="28.87%" headers="mcps1.3.8.2.8.8.2.3.1.1 "><p id="mrs_01_24508__p247793919515">AUXILIARY_ZK_SERVER_PRINCIPAL</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.13000000000001%" headers="mcps1.3.8.2.8.8.2.3.1.2 "><p id="mrs_01_24508__p169981632809">Value of <strong id="mrs_01_24508__b115758544328">ZK_USER_PRINCIPAL</strong> obtained in <a href="#mrs_01_24508__li11758429194015">7</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row174778393519"><td class="cellrowborder" valign="top" width="28.87%" headers="mcps1.3.8.2.8.8.2.3.1.1 "><p id="mrs_01_24508__p144772396514">AUXILIARY_ZK_SERVER_REALM</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.13000000000001%" headers="mcps1.3.8.2.8.8.2.3.1.2 "><p id="mrs_01_24508__p1239919341207">Value of <strong id="mrs_01_24508__b8411103363317">ZK_USER_REALM</strong> obtained in <a href="#mrs_01_24508__li11758429194015">7</a></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row57358592578"><td class="cellrowborder" valign="top" width="28.87%" headers="mcps1.3.8.2.8.8.2.3.1.1 "><p id="mrs_01_24508__p12735145935712">METADATA_COLLECTION_TIMEOUT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="71.13000000000001%" headers="mcps1.3.8.2.8.8.2.3.1.2 "><p id="mrs_01_24508__p1679014231411"><strong id="mrs_01_24508__b15360171620349">180</strong>.</p>
|
|
<p id="mrs_01_24508__p868610351011">This parameter specifies the timeout interval for waiting for the completion of metadata backup on other nodes, in seconds.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="p" id="mrs_01_24508__p7769123514215"><div class="fignone" id="mrs_01_24508__fig220644312212"><span class="figcap"><b>Figure 6 </b>Configuring the cluster authentication information</span><br><span><img id="mrs_01_24508__image95467356184" src="en-us_image_0000001532676354.png"></span></div>
|
|
</div>
|
|
</li><li id="mrs_01_24508__li1247395810242">Click <strong id="mrs_01_24508__b0873737184311">Save</strong>. In the dialog box that is displayed, click <strong id="mrs_01_24508__b1687916375436">OK</strong>.</li><li id="mrs_01_24508__li12543035356">On the ClickHouse service page, click the <strong id="mrs_01_24508__b192601030164411">Instance</strong> tab. On this tab page, select the ClickHouseServer instance from the instance list, and choose <strong id="mrs_01_24508__b18518202154514">More</strong> > <strong id="mrs_01_24508__b274967456">Restart Instance</strong> in the <strong id="mrs_01_24508__b3694715104519">Operation</strong> column to restart the ClickHouseServer instance.</li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_24508__section16918183315128"><a name="mrs_01_24508__section16918183315128"></a><a name="section16918183315128"></a><h4 class="sectiontitle">Migrating Metadata of Databases and Tables in the Source ClickHouse Cluster to the Destination Cluster</h4><ol id="mrs_01_24508__ol11914102213419"><li id="mrs_01_24508__li1162843314242"><a name="mrs_01_24508__li1162843314242"></a><a name="li1162843314242"></a>Log in to FusionInsight Manager of the source and destination clusters, and create the username and password required for the migration. The procedure is as follows:<ol type="a" id="mrs_01_24508__ol25193992614"><li id="mrs_01_24508__li36771959142510">Log in to Manager and choose <strong id="mrs_01_24508__b42717121575">System</strong> > <strong id="mrs_01_24508__b15271101215716">Permission</strong> > <strong id="mrs_01_24508__b6272112974">Role</strong>. On the displayed page, click <strong id="mrs_01_24508__b152721512875">Create Role</strong>.</li><li id="mrs_01_24508__li11670123433010"><a name="mrs_01_24508__li11670123433010"></a><a name="li11670123433010"></a>Specify <strong id="mrs_01_24508__b77423214991517">Role Name</strong>, for example, <strong id="mrs_01_24508__b2067233315712">ckrole</strong>. In the <strong id="mrs_01_24508__b191103569791517">Configure Resource Permission</strong> area, click the cluster name. On the displayed service list page, click the ClickHouse service.</li><li id="mrs_01_24508__li18942182915278">Select <strong id="mrs_01_24508__b95422658791517">SUPER_USER_GROUP</strong> and click <strong id="mrs_01_24508__b168896055891517">OK</strong>.</li><li id="mrs_01_24508__li1183214191540">Choose <strong id="mrs_01_24508__b173382918491517">System</strong>. On the navigation pane on the left, choose <strong id="mrs_01_24508__b58469453291517">Permission</strong> > <strong id="mrs_01_24508__b91903345591517">User</strong> and click <strong id="mrs_01_24508__b98300858991517">Create</strong>.</li><li id="mrs_01_24508__li0521154115455">Select <strong id="mrs_01_24508__b190303448791517">Human-Machine</strong> for <strong id="mrs_01_24508__b111348432291517">User Type</strong> and set <strong id="mrs_01_24508__b144645349991517">Password</strong> and <strong id="mrs_01_24508__b150472699991517">Confirm Password</strong> to the password of the user.<div class="note" id="mrs_01_24508__note1454394019318"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_24508__ul9156135863113"><li id="mrs_01_24508__li615715818314">Username: The username cannot contain hyphens (-). Otherwise, the authentication will fail.</li><li id="mrs_01_24508__li7693437163212">Password: The password cannot contain special characters $, ., and #. Otherwise, the authentication will fail.</li></ul>
|
|
</div></div>
|
|
</li><li id="mrs_01_24508__li766510438542">In the <strong id="mrs_01_24508__b165634514091517">Role</strong> area, click <strong id="mrs_01_24508__b7893448391517">Add</strong> . In the displayed dialog box, select the role name in <a href="#mrs_01_24508__li11670123433010">1.b</a> and click <strong id="mrs_01_24508__b172897146991517">OK</strong> to add the role. Then, click <strong id="mrs_01_24508__b157685528191517">OK</strong>.</li><li id="mrs_01_24508__li8313346183016">After the user is created, click the user name in the upper right corner to log out of the system. Log in to FusionInsight Manager as the new user and change its password as prompted.</li></ol>
|
|
</li><li id="mrs_01_24508__li191413227416">Download the ClickHouse client and install it as user <strong id="mrs_01_24508__b14577259183218">omm</strong> to the destination cluster.</li><li id="mrs_01_24508__li18579227455">Log in to the client node as user <strong id="mrs_01_24508__b115814913171">omm</strong>, go to the <em id="mrs_01_24508__i14985122616349">Client installation directory</em><strong id="mrs_01_24508__b106281547153413">/ClickHouse/clickhouse_migration_tool/clickhouse-metadata-migration</strong> directory, and configure migration information. Run the following command to modify the <strong id="mrs_01_24508__b16591318163510">example_config.yaml</strong> configuration file by referring to <a href="#mrs_01_24508__table16816744680">Table 4</a>:<p id="mrs_01_24508__p1417112338111"><strong id="mrs_01_24508__b1811455020119">cd</strong> <em id="mrs_01_24508__i193716351359">Client installation directory</em><strong id="mrs_01_24508__b1489982001210">/ClickHouse/clickhouse_migration_tool/clickhouse-metadata-migration</strong></p>
|
|
<p id="mrs_01_24508__p147521451063"><strong id="mrs_01_24508__b196618104616">vi example_config.yaml</strong></p>
|
|
<p id="mrs_01_24508__p686566628">After the configuration is modified, you must delete all comment with number sign(#) and retain only valid configurations. Otherwise, an error may occur during script migration.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_24508__table16816744680"></a><a name="table16816744680"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24508__table16816744680" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Parameters in <strong id="mrs_01_24508__b4875183263413">the example_config.yaml</strong> file</caption><thead align="left"><tr id="mrs_01_24508__row11816344485"><th align="left" class="cellrowborder" valign="top" width="14.788521147885211%" id="mcps1.3.9.2.3.9.2.4.1.1"><p id="mrs_01_24508__p98161044286">Configuration Item</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="19.538046195380463%" id="mcps1.3.9.2.3.9.2.4.1.2"><p id="mrs_01_24508__p1281613441583">Sub-item</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="65.67343265673433%" id="mcps1.3.9.2.3.9.2.4.1.3"><p id="mrs_01_24508__p168167444818">Value and Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24508__row208160441684"><td class="cellrowborder" rowspan="7" valign="top" width="14.788521147885211%" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p178161443811">source_cluster</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.538046195380463%" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p1581611447814">host</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.67343265673433%" headers="mcps1.3.9.2.3.9.2.4.1.3 "><p id="mrs_01_24508__p9816544582">IP address of any ClickHouseServer node in the source cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row281719441880"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p1181712441486">cluster_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p13817134413818">Name of the source ClickHouse cluster. You can log in to the ClickHouse client by referring to <a href="mrs_01_2345.html">Using ClickHouse from Scratch</a> and run the following command to obtain the value. If the source cluster name has not been changed, the default value is <strong id="mrs_01_24508__b873169512">default_cluster</strong>.</p>
|
|
<p id="mrs_01_24508__p17227339182215"><strong id="mrs_01_24508__b1448965432216">select cluster,shard_num,replica_num,host_name from system.clusters;</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row564361451215"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p1764321431214">https_port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p16643114101218">To obtain the port number, log in to FusionInsight Manager of the source cluster, choose <strong id="mrs_01_24508__b5683338563">Cluster</strong> > <strong id="mrs_01_24508__b12771134617619">Services</strong> > <strong id="mrs_01_24508__b14682501263">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b5861162773">Configurations</strong> tab and then <strong id="mrs_01_24508__b9421696714">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b8591034374">https_port</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row6332218161211"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p4333718181218">zookeeper_root_path</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p5333118151216">To obtain the value, log in to FusionInsight Manager of the source cluster, choose <strong id="mrs_01_24508__b6613944571">Cluster</strong> > <strong id="mrs_01_24508__b13615184418712">Services</strong> > <strong id="mrs_01_24508__b1361664419712">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b96181944774">Configurations</strong> tab and then <strong id="mrs_01_24508__b1961912441878">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b66207441879">clickhouse.zookeeper.root.path</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row16686819377"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p166811818377">system</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p230054412370">System parameter. Retain the default value.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row1141194743719"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p5543174863714">databases</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p6543174813375">Optional.</p>
|
|
<ul id="mrs_01_24508__ul1954364815374"><li id="mrs_01_24508__li3543114815376">If this parameter is specified, data in the specified database of the source ClickHouse cluster is migrated. You can specify multiple databases. The following configuration is for your reference:<pre class="screen" id="mrs_01_24508__screen13543164863715">databases:
|
|
- "database"
|
|
- "database_1"</pre>
|
|
<p id="mrs_01_24508__p1654344816370">Data in the <strong id="mrs_01_24508__b115462018486">database</strong> and <strong id="mrs_01_24508__b57366227487">database_1</strong> databases of the source cluster is migrated.</p>
|
|
</li><li id="mrs_01_24508__li11543144810372">If this parameter is not specified, table data of all databases in the source ClickHouse cluster is migrated. Leave the <strong id="mrs_01_24508__b188586102498">databases</strong> parameter empty. The following is an example:<pre class="screen" id="mrs_01_24508__screen1754315485371">databases:</pre>
|
|
<p id="mrs_01_24508__p145441548183715">Table information of all databases in the source ClickHouse cluster is migrated.</p>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row55794441378"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p3547348133710">tables</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p55471748173718">Optional. The value is in the format of <em id="mrs_01_24508__i1733175619499">Database name.Table name</em>. The database name must be in the databases parameter list.</p>
|
|
<ul id="mrs_01_24508__ul954784816371"><li id="mrs_01_24508__li9547174883717">If this parameter is specified, data in specified tables in the source ClickHouse cluster database is migrated. You can configure multiple tables. The following configuration is for your reference:<pre class="screen" id="mrs_01_24508__screen10547548123713">tables:
|
|
- "database.table_1"
|
|
- "database_1.table_2"</pre>
|
|
<p id="mrs_01_24508__p175470489373">Data in <strong id="mrs_01_24508__b67702394533">table_1</strong> of <strong id="mrs_01_24508__b453922812540">database</strong> and <strong id="mrs_01_24508__b173184317532">table_2</strong> of database_1 of the source cluster is migrated.</p>
|
|
</li><li id="mrs_01_24508__li1254704843719">If this parameter is not specified and the <strong id="mrs_01_24508__b16600210145511">databases</strong> parameter is specified, all table data in the <strong id="mrs_01_24508__b1516335885510">databases</strong> database is migrated. If the <strong id="mrs_01_24508__b1178133135617">databases</strong> parameter is not specified, all table data in all databases of the source ClickHouse cluster is migrated. The following configuration is for your reference:<pre class="screen" id="mrs_01_24508__screen85481248183711">tables:</pre>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row1280720134114"><td class="cellrowborder" rowspan="6" valign="top" width="14.788521147885211%" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p8807013111116">destination_cluster</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.538046195380463%" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p1080716131112">host</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.67343265673433%" headers="mcps1.3.9.2.3.9.2.4.1.3 "><p id="mrs_01_24508__p17460154533612">IP address of any ClickHouseServer node in the destination cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row12302181610115"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p33021816181111">cluster_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p12460045103619">Name of the destination ClickHouse cluster. You can log in to the ClickHouse client by referring to <a href="mrs_01_2345.html">Using ClickHouse from Scratch</a> and run the following command to obtain the value. If the destination cluster name has not been changed, the default value is <strong id="mrs_01_24508__b1181312152427">default_cluster</strong>.</p>
|
|
<p id="mrs_01_24508__p1946054513619"><strong id="mrs_01_24508__b746012455361">select cluster,shard_num,replica_num,host_name from system.clusters;</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row15954161881110"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p0954111812110">user</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p1190783017375">Username created in <a href="#mrs_01_24508__li1162843314242">1</a> for logging in to FusionInsight Manager of the destination ClickHouse cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row147021736141211"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p1722074712129">https_port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p1460114516368">To obtain the port number, log in to FusionInsight Manager of the destination cluster, choose <strong id="mrs_01_24508__b87811324193112">Cluster</strong> > <strong id="mrs_01_24508__b18781192419319">Services</strong> > <strong id="mrs_01_24508__b13781162414317">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b14782024123112">Configurations</strong> tab and then <strong id="mrs_01_24508__b2782724153118">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b1378232423110">https_port</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row521693941210"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p17220144716124">zookeeper_root_path</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p8460144513614">To obtain the value, log in to FusionInsight Manager of the destination cluster, choose <strong id="mrs_01_24508__b47561714358">Cluster</strong> > <strong id="mrs_01_24508__b1475517153515">Services</strong> > <strong id="mrs_01_24508__b975817113519">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b207611711355">Configurations</strong> tab and then <strong id="mrs_01_24508__b3761917163513">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b8761217103516">clickhouse.zookeeper.root.path</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row1211782613376"><td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.1 "><p id="mrs_01_24508__p611792643710">system</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.9.2.3.9.2.4.1.2 "><p id="mrs_01_24508__p4277124210376">System parameter. Retain the default value.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_24508__li1733154810406">Run the following command to migrate data and wait until the script execution is complete:<p id="mrs_01_24508__p1934784611420"><a name="mrs_01_24508__li1733154810406"></a><a name="li1733154810406"></a><strong id="mrs_01_24508__b52111423124410">./clickhouse_migrate_metadata.sh -f yaml_file</strong></p>
|
|
<p id="mrs_01_24508__p750813512434">Enter the usernames and passwords of the source and destination clusters.</p>
|
|
<p id="mrs_01_24508__p14721172012610"><span><img id="mrs_01_24508__image77214207611" src="en-us_image_0000001537269552.png"></span></p>
|
|
</li></ol>
|
|
<div class="note" id="mrs_01_24508__note5308184043517"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24508__p85431041183513">If metadata migration fails, perform the following steps:</p>
|
|
<ol id="mrs_01_24508__ol1276318107364"><li id="mrs_01_24508__li5974112445010">Locate the failure cause. Specifically, check whether any parameters in the configuration file are incorrectly configured.<ul id="mrs_01_24508__ul4502145511501"><li id="mrs_01_24508__li12502155514500">If yes, reconfigure the parameters and perform metadata migration.</li><li id="mrs_01_24508__li115021955155015">If no, go to <a href="#mrs_01_24508__li19763810133612">2</a>.</li></ul>
|
|
</li><li id="mrs_01_24508__li19763810133612"><a name="mrs_01_24508__li19763810133612"></a><a name="li19763810133612"></a>Set the names of the tables that fail to be migrated in the metadata migration configuration file based on the <strong id="mrs_01_24508__b19646554191815">databases</strong> and <strong id="mrs_01_24508__b1313635981811">tables</strong> parameters in <a href="#mrs_01_24508__table16816744680">Table 4</a> and run the metadata migration command again. If the migration fails, contact O&M personnel.</li></ol>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_24508__section18047266428"><a name="mrs_01_24508__section18047266428"></a><a name="section18047266428"></a><h4 class="sectiontitle">Migrating Data of the Databases and Tables in the Source ClickHouse Cluster to the Destination Cluster</h4><ol id="mrs_01_24508__ol166042470428"><li id="mrs_01_24508__li16604194711421">Log in to the ClickHouse client node in the destination cluster as user <strong id="mrs_01_24508__b1063325319177">omm</strong> and go to <em id="mrs_01_24508__i1467142216369">Client installation directory</em><strong id="mrs_01_24508__b165651302368">/ClickHouse/clickhouse_migration_tool/clickhouse-data-migration</strong>.<p id="mrs_01_24508__p4629204159"><strong id="mrs_01_24508__b15322535141510">cd</strong> <em id="mrs_01_24508__i456665016368">Client installation directory</em><strong id="mrs_01_24508__b5323193501511">/ClickHouse/clickhouse_migration_tool/clickhouse-data-migration</strong></p>
|
|
</li><li id="mrs_01_24508__li15540162313445">Run the following command to modify the <strong id="mrs_01_24508__b19596345133916">example_config.yaml</strong> configuration file by referring to <a href="#mrs_01_24508__table993412163478">Table 5</a>:<p id="mrs_01_24508__p14506102320478"><strong id="mrs_01_24508__b17537526174718">vi example_config.yaml</strong></p>
|
|
<div class="p" id="mrs_01_24508__p62493391516">After the configuration is modified, you must delete all comment with number sign(#) and retain only valid configurations. Otherwise, an error may occur during script migration.
|
|
<div class="tablenoborder"><a name="mrs_01_24508__table993412163478"></a><a name="table993412163478"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24508__table993412163478" frame="border" border="1" rules="all"><caption><b>Table 5 </b>Parameters in <strong id="mrs_01_24508__b993371612472">example_config.yaml</strong></caption><thead align="left"><tr id="mrs_01_24508__row89331416204714"><th align="left" class="cellrowborder" valign="top" width="14.788521147885211%" id="mcps1.3.10.2.2.4.1.2.4.1.1"><p id="mrs_01_24508__p2933816164711">Configuration Item</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="19.538046195380463%" id="mcps1.3.10.2.2.4.1.2.4.1.2"><p id="mrs_01_24508__p1993321604719">Sub-item</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="65.67343265673433%" id="mcps1.3.10.2.2.4.1.2.4.1.3"><p id="mrs_01_24508__p49331916184714">Value and Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24508__row99331116164718"><td class="cellrowborder" rowspan="9" valign="top" width="14.788521147885211%" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p1993371618479">source_cluster</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.538046195380463%" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p119331716144715">host</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.67343265673433%" headers="mcps1.3.10.2.2.4.1.2.4.1.3 "><p id="mrs_01_24508__p4933216164713">IP address of any ClickHouseServer node in the source cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row493318161477"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p18933516104710">cluster_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p893312160472">Name of the source ClickHouse cluster. You can log in to the ClickHouse client by referring to <a href="mrs_01_2345.html">Using ClickHouse from Scratch</a> and run the following command to obtain the value. If the source cluster name has not been changed, the default value is <strong id="mrs_01_24508__b9316724154316">default_cluster</strong>.</p>
|
|
<p id="mrs_01_24508__p193310164472"><strong id="mrs_01_24508__b16933141684713">select cluster,shard_num,replica_num,host_name from system.clusters;</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row393361611473"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p1193318162473">user</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p293315163473">Username created in <a href="#mrs_01_24508__li1162843314242">1</a> for logging in to FusionInsight Manager of the source ClickHouse cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row129341816124711"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p119346161470">https_port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p193491610471">To obtain the port number, log in to FusionInsight Manager of the source cluster, choose <strong id="mrs_01_24508__b422205319436">Cluster</strong> > <strong id="mrs_01_24508__b14222115316434">Services</strong> > <strong id="mrs_01_24508__b19222115314312">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b1922216539437">Configurations</strong> tab and then <strong id="mrs_01_24508__b192221453164320">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b2223135311433">https_port</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row493412166471"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p1093491616476">tcp_port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p193441611478">To obtain the value, log in to FusionInsight Manager of the source cluster, choose <strong id="mrs_01_24508__b425721794519">Cluster</strong> > <strong id="mrs_01_24508__b10235182316452">Services</strong> > <strong id="mrs_01_24508__b979018267455">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b62171029114512">Configurations</strong> tab and then <strong id="mrs_01_24508__b16271145134510">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b1239501354610">tcp_port_secure</strong> if the cluster is in security mode. Otherwise, search for <strong id="mrs_01_24508__b1622614387466">tcp_port</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row139341516174713"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p1093401694710">zookeeper_root_path</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p1693451611473">To obtain the value, log in to FusionInsight Manager of the source cluster, choose <strong id="mrs_01_24508__b950117232470">Cluster</strong> > <strong id="mrs_01_24508__b185011923204715">Services</strong> > <strong id="mrs_01_24508__b18501823104711">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b450112312471">Configurations</strong> tab and then <strong id="mrs_01_24508__b85011223144715">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b7501723124718">clickhouse.zookeeper.root.path</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row148007448332"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p158031144143311">system</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p180364413337">System parameter. Retain the default value.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row1121583811617"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p11215133811614">databases</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p2215193817168">Optional.</p>
|
|
<ul id="mrs_01_24508__ul103536133183"><li id="mrs_01_24508__li735315134185">If this parameter is specified, data in the specified database of the source ClickHouse cluster is migrated. You can specify multiple databases. The following configuration is for your reference:<pre class="screen" id="mrs_01_24508__screen1714983273810">databases:
|
|
- "database"
|
|
- "database_1"</pre>
|
|
<p id="mrs_01_24508__p556504692113">Data in the <strong id="mrs_01_24508__b1068846755">database</strong> and <strong id="mrs_01_24508__b48424937">database_1</strong> databases of the source cluster is migrated.</p>
|
|
</li><li id="mrs_01_24508__li97097512224">If this parameter is not specified, table data of all databases in the source ClickHouse cluster is migrated. Leave the <strong id="mrs_01_24508__b1286101951">databases</strong> parameter empty. The following is an example:<pre class="screen" id="mrs_01_24508__screen430723913812">databases:</pre>
|
|
<p id="mrs_01_24508__p1161751622414">Table information of all databases in the source ClickHouse cluster is migrated.</p>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row12781195701610"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p12781115771617">tables</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p04891434202414">Optional. The value is in the format of <em id="mrs_01_24508__i194178709">Database name.Table name</em>. The database name must be in the databases parameter list.</p>
|
|
<ul id="mrs_01_24508__ul154891347245"><li id="mrs_01_24508__li20490434102411">If this parameter is specified, data in specified tables in the source ClickHouse cluster database is migrated. You can configure multiple tables. The following configuration is for your reference:<pre class="screen" id="mrs_01_24508__screen357804612383">tables:
|
|
- "database.table_1"
|
|
- "database_1.table_2"</pre>
|
|
<p id="mrs_01_24508__p24902034142414">Data in <strong id="mrs_01_24508__b1270462522">table_1</strong> of <strong id="mrs_01_24508__b587771750">database</strong> and <strong id="mrs_01_24508__b1355580378">table_2</strong> of database_1 of the source cluster is migrated.</p>
|
|
</li><li id="mrs_01_24508__li34901834162410">If this parameter is not specified and the <strong id="mrs_01_24508__b785500435">databases</strong> parameter is specified, all table data in the <strong id="mrs_01_24508__b575688419">databases</strong> database is migrated. If the <strong id="mrs_01_24508__b472580025">databases</strong> parameter is not specified, all table data in all databases of the source ClickHouse cluster is migrated. The following configuration is for your reference:<pre class="screen" id="mrs_01_24508__screen414119540384">tables:</pre>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row993431616475"><td class="cellrowborder" rowspan="7" valign="top" width="14.788521147885211%" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p2093415163473">destination_cluster</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.538046195380463%" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p1693421620474">host</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.67343265673433%" headers="mcps1.3.10.2.2.4.1.2.4.1.3 "><p id="mrs_01_24508__p5934916184710">IP address of any ClickHouseServer node in the destination cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row19934516114711"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p1493431614718">cluster_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p793451614471">Name of the destination ClickHouse cluster. You can log in to the ClickHouse client by referring to <a href="mrs_01_2345.html">Using ClickHouse from Scratch</a> and run the following command to obtain the value. If the destination cluster name has not been changed, the default value is <strong id="mrs_01_24508__b3968246105617">default_cluster</strong>.</p>
|
|
<p id="mrs_01_24508__p4934111615474"><strong id="mrs_01_24508__b1993481616477">select cluster,shard_num,replica_num,host_name from system.clusters;</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row199342164479"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p893461612470">user</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p39342016104713">Username created in <a href="#mrs_01_24508__li1162843314242">1</a> for logging in to FusionInsight Manager of the destination ClickHouse cluster.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row15934916194716"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p119341616194718">https_port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p293413164472">To obtain the port number, log in to FusionInsight Manager of the destination cluster, choose <strong id="mrs_01_24508__b1647111145710">Cluster</strong> > <strong id="mrs_01_24508__b748511145719">Services</strong> > <strong id="mrs_01_24508__b16481811165718">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b3481911125713">Configurations</strong> tab and then <strong id="mrs_01_24508__b348141110573">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b448141119574">https_port</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row1593421664719"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p893416165471">tcp_port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p2093481616471"> To obtain the value, log in to FusionInsight Manager of the destination cluster, choose <strong id="mrs_01_24508__b01479396578">Cluster</strong> > <strong id="mrs_01_24508__b1315353915577">Services</strong> > <strong id="mrs_01_24508__b41535395573">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b20154939125719">Configurations</strong> tab and then <strong id="mrs_01_24508__b715411397576">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b14154539115711">tcp_port_secure</strong> if the cluster is in security mode. Otherwise, search for <strong id="mrs_01_24508__b4155539135717">tcp_port</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row18934121674712"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p593481618470">zookeeper_root_path</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p18934121644716">To obtain the value, log in to FusionInsight Manager of the destination cluster, choose <strong id="mrs_01_24508__b69379237589">Cluster</strong> > <strong id="mrs_01_24508__b0937152312580">Services</strong> > <strong id="mrs_01_24508__b14937423105814">ClickHouse</strong>, and click the <strong id="mrs_01_24508__b1193711231581">Configurations</strong> tab and then <strong id="mrs_01_24508__b2938523105812">All Configurations</strong>. In the displayed page, search for <strong id="mrs_01_24508__b14938122335816">clickhouse.zookeeper.root.path</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row19599022143319"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p185999228332">system</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p1359918227335">System parameter. Retain the default value.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row1093418165470"><td class="cellrowborder" rowspan="3" valign="top" width="14.788521147885211%" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p39341316104712">auxiliary_zookeepers</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.538046195380463%" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p129342016204712">name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.67343265673433%" headers="mcps1.3.10.2.2.4.1.2.4.1.3 "><p id="mrs_01_24508__p10934916154719">ZooKeeper name of the source ClickHouse cluster configured in <a href="#mrs_01_24508__li16763133318475">3</a>, for example, <strong id="mrs_01_24508__b94026729591517">zookeeper2</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row99342169473"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p169341316114718">hosts</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p7934171611477">IP address of the ZooKeeper instance of the source ClickHouse. To obtain the IP address, log in to FusionInsight Manager of the source cluster, choose <strong id="mrs_01_24508__b070774335919">Cluster</strong> > <strong id="mrs_01_24508__b5707174305913">Services</strong> > <strong id="mrs_01_24508__b3708144365913">ZooKeeper</strong>, and click the <strong id="mrs_01_24508__b5708194375915">Instance</strong> tab. On the displayed page, view the service IP addresses of the ZooKeeper quorumpeer instance , as shown in <a href="#mrs_01_24508__fig39818132540">Figure 3</a>.</p>
|
|
<div class="p" id="mrs_01_24508__p79581452153515">The format is as follows:<pre class="screen" id="mrs_01_24508__screen461715916384">hosts:
|
|
- "192.168.1.2"
|
|
- "192.168.1.3"
|
|
- "192.168.1.4"</pre>
|
|
</div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row9934416124716"><td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p393451612472">port</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p129341316124717">2181</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row166923891013"><td class="cellrowborder" valign="top" width="14.788521147885211%" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p15305214151019">execution_procedure</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.538046195380463%" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p19305014151017">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.67343265673433%" headers="mcps1.3.10.2.2.4.1.2.4.1.3 "><p id="mrs_01_24508__p13305181441018">This parameter is left blank by default, indicating that the script is executed once to synchronize service data. Value options are <strong id="mrs_01_24508__b579416311303">firststep</strong> and <strong id="mrs_01_24508__b133930342020">secondstep</strong>.</p>
|
|
<ul id="mrs_01_24508__ul3305121451019"><li id="mrs_01_24508__li1030511149103"><strong id="mrs_01_24508__b125911108012">firststep</strong>: Only the temporary replication table is created. The auxiliary ZooKeeper can synchronize data from the original cluster to the temporary table in real time.</li><li id="mrs_01_24508__li1130581420104"><strong id="mrs_01_24508__b161841151011">secondstep</strong>: data in the temporary replication table is attached to the local table of the destination cluster.<div class="caution" id="mrs_01_24508__note730561441019"><span class="cautiontitle"> CAUTION: </span><div class="cautionbody"><p id="mrs_01_24508__p19305201410109">If this parameter is set to <strong id="mrs_01_24508__b112803015111">secondstep</strong>, O&M personnel and users need to confirm that ClickHouse-related services have been stopped before script execution.</p>
|
|
</div></div>
|
|
</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24508__row9742161231015"><td class="cellrowborder" valign="top" width="14.788521147885211%" headers="mcps1.3.10.2.2.4.1.2.4.1.1 "><p id="mrs_01_24508__p330541411013">onereplica_use_auxiliaryzookeeper</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="19.538046195380463%" headers="mcps1.3.10.2.2.4.1.2.4.1.2 "><p id="mrs_01_24508__p530531414109">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.67343265673433%" headers="mcps1.3.10.2.2.4.1.2.4.1.3 "><ul id="mrs_01_24508__ul1630511412106"><li id="mrs_01_24508__li13063141106">If this parameter is set to <strong id="mrs_01_24508__b1739614552113">1</strong>, temporary tables are created for only one replica of each shard.</li><li id="mrs_01_24508__li53064144109">If this parameter is set to <strong id="mrs_01_24508__b1479314221814">0</strong>, temporary tables are created for two replicas of each shard.</li></ul>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</li><li id="mrs_01_24508__li6900232135719">Stop the ClickHouse service of the source cluster.</li><li id="mrs_01_24508__li125756574573">Run the following command to migrate data and wait until the script execution is complete:<p id="mrs_01_24508__p1972675710583"><a name="mrs_01_24508__li125756574573"></a><a name="li125756574573"></a><strong id="mrs_01_24508__b4972190165911">./clickhouse_migrate_data.sh -f yaml_file</strong></p>
|
|
<p id="mrs_01_24508__p12386339202718">Enter the usernames and passwords of the source and destination clusters.</p>
|
|
</li><li id="mrs_01_24508__li285414242020">After the script is executed successfully, perform the following steps to check whether the migrated data in the source cluster is consistent with that in the destination cluster based on the migration result logs:<p id="mrs_01_24508__p621615267019"><a name="mrs_01_24508__li285414242020"></a><a name="li285414242020"></a>Log in to the ClickHouse client node in the destination cluster and go to the <em id="mrs_01_24508__i2332142017378">Client installation directory</em><strong id="mrs_01_24508__b1389093319371">/ClickHouse/clickhouse_migration_tool/clickhouse-data-migration/comparison_result</strong> directory.</p>
|
|
<p id="mrs_01_24508__p15568142155510">Compare the following result file information to check the data consistency between the source cluster and the destination cluster:</p>
|
|
<ul id="mrs_01_24508__ul178031548185620"><li id="mrs_01_24508__li980384825619"><strong id="mrs_01_24508__b335694913359">source_cluster_table_info</strong>: statistics of data migrated from the source cluster</li><li id="mrs_01_24508__li25756175719"><strong id="mrs_01_24508__b67539483620">destination_cluster_table_info</strong>: statistics of data migrated to the destination cluster</li><li id="mrs_01_24508__li18390031115711"><strong id="mrs_01_24508__b03491345363">compare_result_file.txt</strong>: data consistency comparison result before and after migration</li></ul>
|
|
<p id="mrs_01_24508__p255719315596">If the data is inconsistent before and after the migration, clear the data in the table of the destination cluster and migrate the data in the table separately or manually.</p>
|
|
<p id="mrs_01_24508__p129313100015">In addition, you can log in to the ClickHouse databases of the source and destination clusters to manually check whether the number of table data records and partitions are consistent.</p>
|
|
</li><li id="mrs_01_24508__li44971547201318">Log in to FusionInsight Manager of the destination cluster and delete the ZooKeeper information added to <strong id="mrs_01_24508__b109452261224">clickhouse-config-customize</strong> in <a href="#mrs_01_24508__li1315311694620">2</a>.<p id="mrs_01_24508__p76801446193918">Click <strong id="mrs_01_24508__b1952018200317">Save</strong>. In the displayed dialog box, click <strong id="mrs_01_24508__b79851291033">OK</strong>.</p>
|
|
</li><li id="mrs_01_24508__li932564911317">After data migration is complete, switch services to the target ClickHouse cluster.</li><li id="mrs_01_24508__li2275135019572">Go to <em id="mrs_01_24508__i9921659173717">Client installation directory</em><strong id="mrs_01_24508__b997514513810">/ClickHouse/clickhouse_migration_tool/clickhouse-data-migration</strong> and <em id="mrs_01_24508__i11623137385">Client installation directory</em><strong id="mrs_01_24508__b220103516388">/ClickHouse/clickhouse_migration_tool/clickhouse-metadata-migration</strong> on the ClickHouse node in the destination cluster.<p id="mrs_01_24508__p876019525597"><strong id="mrs_01_24508__b1325195015591">vi example_config.yaml</strong></p>
|
|
<p id="mrs_01_24508__p185821949165913">Delete the password from the configuration file to prevent password leakage.</p>
|
|
</li></ol>
|
|
<div class="note" id="mrs_01_24508__note1385975914020"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_24508__p710518114118">If service data migration fails, perform the following steps:</p>
|
|
<ol id="mrs_01_24508__ol8481355124117"><li id="mrs_01_24508__li819555175218">Locate the failure cause. Specifically, check whether any parameters in the configuration file are incorrectly configured.<ul id="mrs_01_24508__ul3195105175213"><li id="mrs_01_24508__li619585175220">If yes, reconfigure the parameters and perform service data migration.</li><li id="mrs_01_24508__li9195115165219">If no, go to <a href="#mrs_01_24508__li2481155164115">2</a>.</li></ul>
|
|
</li><li id="mrs_01_24508__li2481155164115"><a name="mrs_01_24508__li2481155164115"></a><a name="li2481155164115"></a>Run the <strong id="mrs_01_24508__b16450715184511">drop table </strong><em id="mrs_01_24508__i12481134114212">table_name</em> command to delete the data tables related to the table from the node that fails to be migrated in the destination cluster.</li><li id="mrs_01_24508__li19481145544113">Run the <strong id="mrs_01_24508__b189173454510">show create table </strong><em id="mrs_01_24508__i89185417455">table_name</em> command to query the table creation statements related to the table in the source cluster and create the table in the destination cluster again.</li><li id="mrs_01_24508__li1232803613463">Set the names of the tables that fail to be migrated in the service data migration configuration file based on the <strong id="mrs_01_24508__b16749163894210">databases</strong> and <strong id="mrs_01_24508__b0750203834216">tables</strong> parameters in <a href="#mrs_01_24508__table993412163478">Table 5</a> and run the service data migration command again. If the command fails to execute, contact O&M personnel.</li></ol>
|
|
</div></div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24250.html">Migrating ClickHouse Data</a></div>
|
|
</div>
|
|
</div>
|
|
|