forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
34 lines
6.1 KiB
HTML
34 lines
6.1 KiB
HTML
<a name="mrs_01_2301"></a><a name="mrs_01_2301"></a>
|
|
|
|
<h1 class="topictitle1">Migrating Data on CarbonData from Spark 1.5 to Spark2x</h1>
|
|
<div id="body1600760662040"><div class="section" id="mrs_01_2301__section833453015315"><h4 class="sectiontitle">Migration Solution Overview</h4><p id="mrs_01_2301__p16765153113110">This migration guides you to migrate the CarbonData table data of Spark 1.5 to that of Spark2x.</p>
|
|
<div class="note" id="mrs_01_2301__note73802029132816"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_2301__p183816295281">Before performing this operation, you need to stop the data import service of the CarbonData table in Spark 1.5 and migrate data to the CarbonData table of Spark2x at a time. After the migration is complete, use Spark2x to perform service operations.</p>
|
|
</div></div>
|
|
<p id="mrs_01_2301__p15758191471217">Migration roadmap:</p>
|
|
<ol id="mrs_01_2301__ol158212224124"><li id="mrs_01_2301__li1482102218126">Use Spark 1.5 to migrate historical data to the intermediate table.</li><li id="mrs_01_2301__li982112291216">Use Spark2x to migrate data from the intermediate table to the target table and change the target table name to the original table name.</li><li id="mrs_01_2301__li5821822121220">After the migration is complete, use Spark2x to operate data in the CarbonData table.</li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_2301__section65812925414"><h4 class="sectiontitle">Migration Solution and Commands</h4><p id="mrs_01_2301__p61011930195412"><strong id="mrs_01_2301__b66711197373">Migrating Historical Data</strong></p>
|
|
</div>
|
|
<ol id="mrs_01_2301__ol189205312114"><li id="mrs_01_2301__li1892013318114"><span>Stop the CarbonData data import service, use spark-beeline of Spark 1.5 to view the ID and time of the latest segment in the CarbonData table, and record the segment ID.</span><p><p id="mrs_01_2301__p98421913165519"><strong id="mrs_01_2301__b334122419552">show segments for table dbname.tablename;</strong></p>
|
|
</p></li><li id="mrs_01_2301__li1092003116117"><a name="mrs_01_2301__li1092003116117"></a><a name="li1092003116117"></a><span>Run spark-beeline of Spark 1.5 as the user who has created the original CarbonData table to create an intermediate table in ORC or Parquet format. Then import the data in the original CarbonData table to the intermediate table. After the import is complete, the services of the CarbonData table can be restored.</span><p><p id="mrs_01_2301__p104110183564">Create an ORC table.</p>
|
|
<p id="mrs_01_2301__p17788131205610"><strong id="mrs_01_2301__b1497222318568">CREATE TABLE dbname.mid_tablename_orc STORED AS ORC as select * from dbname.tablename;</strong></p>
|
|
<p id="mrs_01_2301__p149741819115620">Create a Parquet table.</p>
|
|
<p id="mrs_01_2301__p19510330125612"><strong id="mrs_01_2301__b415973416564">CREATE TABLE dbname.mid_tablename_parq STORED AS PARQUET as select * from dbname.tablename;</strong></p>
|
|
<p id="mrs_01_2301__p5166153195516">In the preceding command, <strong id="mrs_01_2301__b1763711363816">dbname</strong> indicates the database name and <strong id="mrs_01_2301__b16716151718384">tablename</strong> indicates the name of the original CarbonData table.</p>
|
|
</p></li><li id="mrs_01_2301__li192112311210"><a name="mrs_01_2301__li192112311210"></a><a name="li192112311210"></a><span>Run spark-beeline of Spark2x as the user who has created the original CarbonData table. Run the table creation statement of the old table to create a CarbonData table.</span><p><div class="note" id="mrs_01_2301__note1972753417620"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_2301__p197286341616">In the statement for creating a new table, the field sequence and type must be the same as those of the old table. In this way, the index column structure of the old table can be retained, which helps avoid errors caused by the use of <strong id="mrs_01_2301__b146868420413">select *</strong> statement during data insertion.</p>
|
|
</div></div>
|
|
<p id="mrs_01_2301__p53836325810">Run the spark-beeline command of Spark 1.5 to view the table creation statement of the old table: <strong id="mrs_01_2301__b19051014288">SHOW CREATE TABLE dbname.tablename;</strong></p>
|
|
<p id="mrs_01_2301__p1243517161991">Create a CarbonData table named <strong id="mrs_01_2301__b8143155220475">dbname.new_tablename</strong>.</p>
|
|
</p></li><li id="mrs_01_2301__li179213311416"><span>Run spark-beeline of Spark2x as the user who has created the original CarbonData table to load the intermediate table data in ORC (or PARQUET) format created in <a href="#mrs_01_2301__li1092003116117">2</a> to the new table created in <a href="#mrs_01_2301__li192112311210">3</a>. This step may take a long time (about 2 hours for 200 GB data). The following uses the ORC intermediate table as an example to describe the command for loading data:</span><p><p id="mrs_01_2301__p181881012913"><strong id="mrs_01_2301__b17492244161317">insert into dbname.new_tablename select *</strong></p>
|
|
<p id="mrs_01_2301__p81888121013"><strong id="mrs_01_2301__b1549418443138">from dbname. mid_tablename_orc;</strong></p>
|
|
</p></li><li id="mrs_01_2301__li67016598344"><span>Run spark-beeline of Spark2x as the user who has created the original CarbonData table to query and verify the data in the new table. If the data is correct, change the name of the original CarbonData table and then change the name of the new CarbonData table to the name of the original one.</span><p><p id="mrs_01_2301__p8701115920344"><strong id="mrs_01_2301__b170115599345">ALTER TABLE dbname.tablename RENAME TO dbname.old_tablename;</strong></p>
|
|
<p id="mrs_01_2301__p870111594348"><strong id="mrs_01_2301__b6701185915349">ALTER TABLE dbname.new_tablename RENAME TO dbname.tablename;</strong></p>
|
|
</p></li><li id="mrs_01_2301__li1712319283252"><span>Complete the migration. In this case, you can use Spark2x to query the new table and rebuild the secondary index.</span></li></ol>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1405.html">CarbonData Operation Guide</a></div>
|
|
</div>
|
|
</div>
|
|
|