forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
87 lines
8.3 KiB
HTML
87 lines
8.3 KiB
HTML
<a name="mrs_01_24093"></a><a name="mrs_01_24093"></a>
|
|
|
|
<h1 class="topictitle1">Write Configuration</h1>
|
|
<div id="body0000001104543692">
|
|
<div class="tablenoborder"><a name="mrs_01_24093__table1815615307121"></a><a name="table1815615307121"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24093__table1815615307121" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Write configuration</caption><thead align="left"><tr id="mrs_01_24093__row9156123018123"><th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.1.2.4.1.1"><p id="mrs_01_24093__p6156133081212">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="40%" id="mcps1.3.1.2.4.1.2"><p id="mrs_01_24093__p541231385612">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="30%" id="mcps1.3.1.2.4.1.3"><p id="mrs_01_24093__p6156830181220">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24093__row9156183010125"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.1 "><p id="mrs_01_24093__p81562030181213">hoodie.datasource.write.table.name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.2.4.1.2 "><p id="mrs_01_24093__p6413513135613">Specifies the name of the Hudi table to be written.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.3 "><p id="mrs_01_24093__p131567308126">None</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24093__row51563309124"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.1 "><p id="mrs_01_24093__p31560300128">hoodie.datasource.write.operation</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.2.4.1.2 "><p id="mrs_01_24093__p202452375618">Specifies the operation type of writing the Hudi table. Currently, <strong id="mrs_01_24093__b236114062103458">upsert</strong>, <strong id="mrs_01_24093__b1661419444103458">delete</strong>, <strong id="mrs_01_24093__b2017857173103458">insert</strong>, and <strong id="mrs_01_24093__b1075950018103458">bulk_insert</strong> are supported.</p>
|
|
<ul id="mrs_01_24093__ul10879549586"><li id="mrs_01_24093__li387915465811"><strong id="mrs_01_24093__b1357015316328">upsert</strong>: updates and inserts data.</li><li id="mrs_01_24093__li158796411585"><strong id="mrs_01_24093__b156814132329">delete</strong>: deletes data.</li><li id="mrs_01_24093__li487917413585"><strong id="mrs_01_24093__b164384371323">insert</strong>: inserts data.</li><li id="mrs_01_24093__li6880174125812"><strong id="mrs_01_24093__b739245436103458">bulk_insert</strong>: imports data during initial table creation. Do not <strong id="mrs_01_24093__b163144838103458">upsert</strong> or <strong id="mrs_01_24093__b181128810103458">insert</strong> during initial table creation.</li><li id="mrs_01_24093__li3405105352211"><strong id="mrs_01_24093__b299953222103458">insert_overwrite</strong>: performs insert and overwrite operations on static partitions.</li><li id="mrs_01_24093__li15566184612242"><strong id="mrs_01_24093__b1984535010323">insert_overwrite_table</strong>: performs insert and overwrite operations on dynamic partitions. It does not immediately delete the entire table or overwrite the table. Instead, it overwrites the metadata of the Hudi table logically, and Hudi deletes useless data through the clean mechanism. Its efficiency is higher than that of the combination of <strong id="mrs_01_24093__b724428004103458">bulk_insert</strong> and <strong id="mrs_01_24093__b1424758111103458">overwrite</strong>.</li></ul>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.3 "><p id="mrs_01_24093__p1156330101219">upsert</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24093__row41561030111215"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.1 "><p id="mrs_01_24093__p1215711300126">hoodie.datasource.write.table.type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.2.4.1.2 "><p id="mrs_01_24093__p132512316568">Specifies the Hudi table type. Once the table type is specified, this parameter cannot be modified. The value can be <strong id="mrs_01_24093__b1978129147103458">MERGE_ON_READ</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.3 "><p id="mrs_01_24093__p51571302122">COPY_ON_WRITE</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24093__row151571030131215"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.1 "><p id="mrs_01_24093__p13157230171211">hoodie.datasource.write.precombine.field</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.2.4.1.2 "><p id="mrs_01_24093__p225142319561">Merges and reduplicates rows with the same key before write.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.3 "><p id="mrs_01_24093__p81571730121213">ts</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24093__row1157173011129"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.1 "><p id="mrs_01_24093__p1515793014128">hoodie.datasource.write.payload.class</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.2.4.1.2 "><p id="mrs_01_24093__p1025423135620">Specifies the class used to merge the records to be updated and the updated records during update. This parameter can be customized. You can compile it yourself to implement your merge logic.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.3 "><p id="mrs_01_24093__p215773014128">org.apache.hudi.OverwriteWithLatestAvroPayload</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24093__row19157133015123"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.1 "><p id="mrs_01_24093__p915715305127">hoodie.datasource.write.recordkey.field</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.2.4.1.2 "><p id="mrs_01_24093__p1125523105611">Specifies the primary key of the Hudi table. The Hudi table must have a unique primary key.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.3 "><p id="mrs_01_24093__p10157230121215">uuid</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24093__row1615773020120"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.1 "><p id="mrs_01_24093__p8157430181212">hoodie.datasource.write.partitionpath.field</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.2.4.1.2 "><p id="mrs_01_24093__p32519232569">Specifies the partition key. This parameter is used together with <strong id="mrs_01_24093__b1061292593103458">hoodie.datasource.write.keygenerator.class</strong> to meet the requirements of different partition scenarios.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.3 "><p id="mrs_01_24093__p181571730181218">partitionpath</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24093__row1683773264116"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.1 "><p id="mrs_01_24093__p1183793264116">hoodie.datasource.write.hive_style_partitioning</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.2.4.1.2 "><p id="mrs_01_24093__p425823155610">Specifies whether the partition mode is the same as that of Hive. You are advised to set this parameter to <strong id="mrs_01_24093__b831853266103458">true</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.3 "><p id="mrs_01_24093__p6837143214412">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24093__row1579185616411"><td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.1 "><p id="mrs_01_24093__p1580256134112">hoodie.datasource.write.keygenerator.class</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="40%" headers="mcps1.3.1.2.4.1.2 "><p id="mrs_01_24093__p1925823135613">Generates the primary key and partition mode when used together with <strong id="mrs_01_24093__b2069708678103458">hoodie.datasource.write.partitionpath.field</strong> and <strong id="mrs_01_24093__b1267890379103458">hoodie.datasource.write.recordkey.field</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="30%" headers="mcps1.3.1.2.4.1.3 "><p id="mrs_01_24093__p1958035610412">org.apache.hudi.keygen.SimpleKeyGenerator</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_24032.html">Configuration Reference</a></div>
|
|
</div>
|
|
</div>
|
|
|