forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
735 lines
79 KiB
HTML
735 lines
79 KiB
HTML
<a name="mrs_01_1404"></a><a name="mrs_01_1404"></a>
|
|
|
|
<h1 class="topictitle1">Configuration Reference</h1>
|
|
<div id="body1595920208049"><p id="mrs_01_1404__aa7b55ea30ddc494792d53ea0c52cb350">This section provides the details of all the configurations required for the CarbonData System.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1404__t3f3d97b04e9c45e9a33ec03b352284af" frame="border" border="1" rules="all"><caption><b>Table 1 </b>System configurations in <strong id="mrs_01_1404__b18838171191616">carbon.properties</strong></caption><thead align="left"><tr id="mrs_01_1404__raf29dd9f05c34b678bac446183b9530a"><th align="left" class="cellrowborder" valign="top" width="18.86%" id="mcps1.3.2.2.4.1.1"><p id="mrs_01_1404__a75456412ccf74b08846e3e0cc5ee8699">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="20.990000000000002%" id="mcps1.3.2.2.4.1.2"><p id="mrs_01_1404__a080775fd9ca344e7a913a203e5f7de9e">Default Value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="60.150000000000006%" id="mcps1.3.2.2.4.1.3"><p id="mrs_01_1404__a1359be66d29440a19e846b8346fcde1e">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1404__rb27369dd024d4ea8830217c2615aef1e"><td class="cellrowborder" valign="top" width="18.86%" headers="mcps1.3.2.2.4.1.1 "><p id="mrs_01_1404__a20c8526ba78243ba9c5f39a6435192ac">carbon.ddl.base.hdfs.url</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.990000000000002%" headers="mcps1.3.2.2.4.1.2 "><p id="mrs_01_1404__a004652e0c0a142ccb6c57265316d4b5a">hdfs://hacluster/opt/data</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.150000000000006%" headers="mcps1.3.2.2.4.1.3 "><p id="mrs_01_1404__a583083d303c748259248cb30bd302369">HDFS relative path from the HDFS base path, which is configured in <span class="parmname" id="mrs_01_1404__p123c96b2864b47fb838383d0e3a5c845"><b>fs.defaultFS</b></span>. The path configured in <span class="parmname" id="mrs_01_1404__pb6e44e047318496490c70d8a572a538b"><b>carbon.ddl.base.hdfs.url</b></span> will be appended to the HDFS path configured in <span class="parmname" id="mrs_01_1404__p8c9946264d4049fe800021831e59bfee"><b>fs.defaultFS</b></span>. If this path is configured, you do not need to pass the complete path while dataload.</p>
|
|
<p id="mrs_01_1404__afdb6a0662ad9430fb512ecb8d60a6fa9">For example, if the absolute path of the CSV file is <span class="filepath" id="mrs_01_1404__f0c6844014ded4064a8d1e2382ebcdb97"><b>hdfs://10.18.101.155:54310/data/cnbc/2016/xyz.csv</b></span>, the path <span class="filepath" id="mrs_01_1404__f67f8e4587a8147018bad526167a826dd"><b>hdfs://10.18.101.155:54310</b></span> will come from property <span class="parmname" id="mrs_01_1404__p90e1223918304a5b8a5a1b54793af58c"><b>fs.defaultFS</b></span> and you can configure <span class="filepath" id="mrs_01_1404__ff2be7a04f96c45a0b2f723ed254e289a"><b>/data/cnbc/</b></span> as <span class="parmname" id="mrs_01_1404__p5dc752d5fbd6474593ea574fef269f67"><b>carbon.ddl.base.hdfs.url</b></span>.</p>
|
|
<p id="mrs_01_1404__af02c9015c59141e58b71f7200f477425">During data loading, you can specify the CSV path as <span class="filepath" id="mrs_01_1404__fed78e6895c3540b09c4642107af424cd"><b>/2016/xyz.csv</b></span>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r343943b8576543fdbc455331441f9226"><td class="cellrowborder" valign="top" width="18.86%" headers="mcps1.3.2.2.4.1.1 "><p id="mrs_01_1404__aad428485216f48f894b28adba7b3bf75">carbon.badRecords.location</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.990000000000002%" headers="mcps1.3.2.2.4.1.2 "><p id="mrs_01_1404__ab5afc60b0f1043df9e502cb89521081d">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.150000000000006%" headers="mcps1.3.2.2.4.1.3 "><p id="mrs_01_1404__a40a5c8d88dd64a32bdbfc332b6a9d1cf">Storage path of bad records. This path is an HDFS path. The default value is <strong id="mrs_01_1404__b628413494104735">Null</strong>. If bad records logging or bad records operation redirection is enabled, the path must be configured by the user.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r45f2da53cc11457a800a07d18af1f945"><td class="cellrowborder" valign="top" width="18.86%" headers="mcps1.3.2.2.4.1.1 "><p id="mrs_01_1404__a9a0d3ff900f9419eb2d4408b5483fcd9">carbon.bad.records.action</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.990000000000002%" headers="mcps1.3.2.2.4.1.2 "><p id="mrs_01_1404__a06c2f0032e5b4030a739958e6e266345">fail</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.150000000000006%" headers="mcps1.3.2.2.4.1.3 "><p id="mrs_01_1404__a06b697c58ac44dbdb6af41f4b0c46178">The following are four types of actions for bad records:</p>
|
|
<p id="mrs_01_1404__a18eb74fb55064d0493b558b9c86140f6"><strong id="mrs_01_1404__b694851784104735">FORCE</strong>: Data is automatically corrected by storing the bad records as NULL.</p>
|
|
<p id="mrs_01_1404__a7d251612a6df408b81217630b132e94b"><strong id="mrs_01_1404__b1046599333104735">REDIRECT</strong>: Bad records are written to the raw CSV instead of being loaded.</p>
|
|
<p id="mrs_01_1404__ab164f0ad4eb841cdbacbac53cc78c51e"><strong id="mrs_01_1404__b766037403104735">IGNORE</strong>: Bad records are neither loaded nor written to the raw CSV.</p>
|
|
<p id="mrs_01_1404__a5cb53d6f5bef4a4d91a211b77d8d1616"><strong id="mrs_01_1404__b342375181104735">FAIL</strong>: Data loading fails if any bad records are found.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__reb89f3ea16a845cf889407653719ec80"><td class="cellrowborder" valign="top" width="18.86%" headers="mcps1.3.2.2.4.1.1 "><p id="mrs_01_1404__ae6b4fceeda91428fa73a4c352156b908">carbon.update.sync.folder</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="20.990000000000002%" headers="mcps1.3.2.2.4.1.2 "><p id="mrs_01_1404__a2438b133d4364191affe83cb55d65a22">/tmp/carbondata</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.150000000000006%" headers="mcps1.3.2.2.4.1.3 "><p id="mrs_01_1404__a4e40e0ff1c0b461a930bd2d98c9a4731">Specifies the <strong id="mrs_01_1404__b615045214104735">modifiedTime.mdt</strong> file path. You can set it to an existing path or a new path.</p>
|
|
<div class="note" id="mrs_01_1404__ned6d012c1dd24afe89596402704888de"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1404__abd4524bb4023413f808e1f162773f86b">If you set this parameter to an existing path, ensure that all users can access the path and the path has the 777 permission.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1404__t197b7a04db3c4f919bd30707c2fdcd1f"></a><a name="t197b7a04db3c4f919bd30707c2fdcd1f"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1404__t197b7a04db3c4f919bd30707c2fdcd1f" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Performance configurations in <strong id="mrs_01_1404__b565845517197">carbon.properties</strong></caption><thead align="left"><tr id="mrs_01_1404__r58d08b2e3ce94fa69bd62bc7039cbff7"><th align="left" class="cellrowborder" valign="top" width="18.67%" id="mcps1.3.3.2.4.1.1"><p id="mrs_01_1404__a2a2146a71a7f4106a1d23347d1867773">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="14.979999999999999%" id="mcps1.3.3.2.4.1.2"><p id="mrs_01_1404__a79c9998b19db413fa887df82de3e1b61">Default Value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="66.35%" id="mcps1.3.3.2.4.1.3"><p id="mrs_01_1404__ac96a875cb8004ba09f213da500e699c4">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1404__r17be0093039a4711831d2632c6e7d367"><td class="cellrowborder" colspan="3" valign="top" headers="mcps1.3.3.2.4.1.1 mcps1.3.3.2.4.1.2 mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__af8df7462293243248e6e94f2866969ec"><strong id="mrs_01_1404__a73f0a30e9b754823a9c144f798bf55b7">Data Loading Configuration</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r30bfd47802b94dbd84fb671c335529e9"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__ad28332cd610f4bcb9047a9961e3faa3b">carbon.sort.file.write.buffer.size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a0f21a9c9ccbe4cbca420df8d595d2334">16384</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a60cf0124893a4eaeb1f4bcca411de84c">CarbonData sorts data and writes it to a temporary file to limit memory usage. This parameter controls the size of the buffer used for reading and writing temporary files. The unit is bytes.</p>
|
|
<p id="mrs_01_1404__p614145212137">The value ranges from 10240 to 10485760.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r44bdcf399951407f9c742591cdf63a57"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__af7d269c3e758440ca29a0f75f1567547">carbon.graph.rowset.size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a28a9bda2ddce46e69801e81d9e812a15">100,000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__ad7f661c75f9e4ac9b824ff6e09cbb090">Rowset size exchanged in data loading graph steps.</p>
|
|
<p id="mrs_01_1404__a5b6dc382e58e44bb967454bfaaeae016">The value ranges from 500 to 1,000,000.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rb674745fe1d041a487f5d2ca94f906e3"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a8ab9a299a5794897876af260854ba69d">carbon.number.of.cores.while.loading</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a736cfb5beb454c7387213119fbabd4c9">6</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a93cc5b9556144321b13f9146ea951bd2">Number of cores used during data loading. The greater the number of cores, the better the compaction performance. If the CPU resources are sufficient, you can increase the value of this parameter.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r69ee4322bee048fe89e256f1fe561ab6"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a0bae07486f4249ff8e6c353db5686038">carbon.sort.size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a1aa52e0b277e499bb7a7dc699ce3b1f2">500000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__ad75af23309004a358fe8e969bed57355">Number of records to be sorted</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r83590bf90376409c874ad799352d9ea7"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__ae292cb01879d4629a6454138d1daa2f3">carbon.enableXXHash</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a466bbd62d52c4057b230916c397a586e">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a602113a21510483eaba5a622a5b33004">Hashmap algorithm used for hashkey calculation</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__re93c2a8ab97a48e68b0bd667e7028b30"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a64e1a9441d7942f68e1e54f9f5e993da">carbon.number.of.cores.block.sort</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a54474d75793b48119721aebc997143dc">7</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a8ec3ac90f5f941369bf98f6457c5effb">Number of cores used for sorting blocks during data loading</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r00223e4ec6e84acfb53571fdbbbd51d2"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__aeea46203d5ee43d8b04912cf8e1fcd5b">carbon.max.driver.lru.cache.size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a4b3e57bc84594eab80959b7b295fc5e4">-1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__ab7371bc9c0cc4fa59ba8a53d1334843f">Maximum size of LRU caching for data loading at the driver side. The unit is MB. The default value is <strong id="mrs_01_1404__b2069919755104735">-1</strong>, indicating that there is no memory limit for the caching. Only integer values greater than 0 are accepted.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r834e60d9bcf7475095dfc975e16cae5f"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a2b707baee72c412b8b9e8d4b43cd54a6">carbon.max.executor.lru.cache.size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a6675779795844fc9bc0cc5a8856fabc6">-1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a2f1b12628b60439b9724a02564406c1f">Maximum size of LRU caching for data loading at the executor side. The unit is MB. The default value is <strong id="mrs_01_1404__b953246543104735">-1</strong>, indicating that there is no memory limit for the caching. Only integer values greater than 0 are accepted. If this parameter is not configured, the value of <span class="parmname" id="mrs_01_1404__pb0d2d156f4a04c5bb4f842aa8d879a61"><b>carbon.max.driver.lru.cache.size</b></span> is used.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r4da8db16467f4a318cd7a4d4251159c4"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__ac4046db0d2b84d82a06040ab234c4371">carbon.merge.sort.prefetch</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a059d0a753b6048c38828a560e6d7e60b">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a73da402b4e7c4d85aeda4f91d18b0448">Whether to enable prefetch of data during merge sort while reading data from sorted temp files in the process of data loading</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r2bab23a1b67945658f15f9e5848aacf6"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__af10de8dd85604fd3b4b9e6589cc4ba8c">carbon.update.persist.enable</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a9063cec9cd5b49928178b1c531e86b85">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a3190df6dbc914370aa65cbb8b23ef806">Configuration to enable the dataset of RDD/dataframe to persist data. Enabling this will reduce the execution time of UPDATE operation.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r5285e71ae13c42378992dd19f44d6fce"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__af1ef394f10004e98b62c992353bb7c60">enable.unsafe.sort</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a64b27b0173f34a5b8321af594bd11798">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__aecb8b7dc29624b08b6ab6a2abdd514ef">Whether to use unsafe sort during data loading. Unsafe sort reduces the garbage collection during data load operation, resulting in better performance. The default value is <span class="parmvalue" id="mrs_01_1404__p0a265c87dde94151987a365eefb979ca"><b>true</b></span>, indicating that unsafe sort is enabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rfedbc2d7edaf4fe98264528fae035a83"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a6e894de483a840a09642f529d5955d01">enable.offheap.sort</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a72897bb0d1644d21b789d6cf06de27e8">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a900efa67621e4adfb75d3725bd791c76">Whether to use off-heap memory for sorting of data during data loading</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r35ea952a21a4499f8460f49650711d70"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a3a68aa16d77d418c824b2e3ae1c934b7">offheap.sort.chunk.size.inmb</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__ab876d5e8a94f4d76a669771ec60cac4b">64</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a32bc8358c8a3435b8e38f9cb11dd0c74">Size of data chunks to be sorted, in MB. The value ranges from 1 to 1024.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r8b24bd5d4ebe40b9bfe2835ce87418bc"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a74ca5dfb09724309983e950bd6f3736f">carbon.unsafe.working.memory.in.mb</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a6876eab1aae543d4af57558821b8afa6">512</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a9482024d6fe749689b027e1a2192dffd">Size of the unsafe working memory. This will be used for sorting data and storing column pages. The unit is MB.</p>
|
|
<p id="mrs_01_1404__en-us_topic_0085589714_p229871059051">Memory required for data loading:</p>
|
|
<p id="mrs_01_1404__en-us_topic_0085589714_p55573589051">carbon.number.of.cores.while.loading [default value is 6] x Number of tables to load in parallel x offheap.sort.chunk.size.inmb [default value is 64 MB] + carbon.blockletgroup.size.in.mb [default value is 64 MB] + Current compaction ratio [64 MB/3.5])</p>
|
|
<p id="mrs_01_1404__en-us_topic_0085589714_p500162269051">= Around 900 MB per table</p>
|
|
<p id="mrs_01_1404__en-us_topic_0085589714_p474928549051">Memory required for data query:</p>
|
|
<p id="mrs_01_1404__en-us_topic_0085589714_p247825079051">(SPARK_EXECUTOR_INSTANCES. [default value is 2] x (carbon.blockletgroup.size.in.mb [default value: 64 MB] + carbon.blockletgroup.size.in.mb [default value = 64 MB x 3.5) x Number of cores per executor [default value: 1])</p>
|
|
<p id="mrs_01_1404__en-us_topic_0085589714_p217159799051">= ~ 600 MB</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__reb4c2aee67934e14a81c8dc3d6b2be25"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a8ff232ac303a4dd49760f03c8c25ecd3">carbon.sort.inmemory.storage.size.in.mb</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a5e20a1e691964b959cf93f5459ae7fe4">512</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a60b37444106447d089ca44704e7a656a">Size of the intermediate sort data to be kept in the memory. Once the specified value is reached, the system writes data to the disk. The unit is MB.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__ref8fdf333d354b55bb3ab09d89e2a063"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__aed00250dbbfa40e1b752d21a158a76e7">sort.inmemory.size.inmb</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a44ea46e2b7e24bd48caf951da239499c">1024</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__acbc805633da944adb3d340cd628ab6f0">Size of the intermediate sort data to be kept in the memory. Once the specified value is reached, the system writes data to the disk. The unit is MB.</p>
|
|
<p id="mrs_01_1404__a6e3accd46daf43d380c429d3f628c515">If <strong id="mrs_01_1404__b1378631789104735">carbon.unsafe.working.memory.in.mb</strong> and <strong id="mrs_01_1404__b1353117785104735">carbon.sort.inmemory.storage.size.in.mb</strong> are configured, you do not need to set this parameter. If this parameter has been configured, 20% of the memory is used for working memory <strong id="mrs_01_1404__b1718415496104735">carbon.unsafe.working.memory.in.mb</strong>, and 80% is used for sort storage memory <strong id="mrs_01_1404__b2126835999104735">carbon.sort.inmemory.storage.size.in.mb</strong>.</p>
|
|
<div class="note" id="mrs_01_1404__n00347865ab0845a88b0034447f8e2205"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1404__ad128315db4454d5c8704de1a79c8945d">The value of <strong id="mrs_01_1404__b1131225770104735">spark.yarn.executor.memoryOverhead</strong> configured for Spark must be greater than the value of <strong id="mrs_01_1404__b39880197104735">sort.inmemory.size.inmb</strong> configured for CarbonData. Otherwise, Yarn might stop the executor if off-heap access exceeds the configured executor memory.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r4a365906130545c68b0da55dd069f0ba"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a2a1af7985e624118a118bbbe39417016">carbon.blockletgroup.size.in.mb</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__aacd0594291b34d58939564c4fb21bca6">64</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__ad5820964b2424856869b76eb94fac0b6">The data is read as a group of blocklets which are called blocklet groups. This parameter specifies the size of each blocklet group. Higher value results in better sequential I/O access.</p>
|
|
<p id="mrs_01_1404__a20d0f5b8399145bcbde74f94f186bd58">The minimum value is 16 MB. Any value less than 16 MB will be reset to the default value (64 MB).</p>
|
|
<p id="mrs_01_1404__afb34029923214021b14e866106e79606">The unit is MB.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rc4cad242e1414ba9a101a0ece58d7a82"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a97e014f51c26458899d1249f1ab7969f">enable.inmemory.merge.sort</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a176b490d43b5496199f488513aa053f3">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a2ed5c2e3958548beb79106fde49005e5">Whether to enable <strong id="mrs_01_1404__b19101193418012">inmemorymerge sort</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r1981c1248dca4c4387b3da5727121cf1"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a4fd658008a514875bb8af3bf53d9b647">use.offheap.in.query.processing</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a8b97752bfe934d60866721b3f28270c0">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__aa557f7d2557c4647861aeb737ea6fbfc">Whether to enable <strong id="mrs_01_1404__b293116398014">offheap</strong> in query processing.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__ree04ad4841ad4ae186f1c4882e41c3ef"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a1ebd61f39a414d2aa73acbce1a842ea9">carbon.load.sort.scope</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__aba5b01e3e5f94ffc9578d4389b79aa91">local_sort</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a4f1533424e334edfb3e58d834faf41da">Sort scope for the load operation. There are two types of sort: <strong id="mrs_01_1404__b12424164718">batch_sort </strong>and <strong id="mrs_01_1404__b18506867111">local_sort</strong>. If <strong id="mrs_01_1404__b119131591715">batch_sort</strong> is selected, the loading performance is improved but the query performance is reduced.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r82aa01df9b0044fd889883baddadaca4"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__af340bfd05e9641f3bbc633bee345367e">carbon.batch.sort.size.inmb</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a61852bd5945e4a5b8cd8e069654df14f">-</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a6b429a2f032b4a728956e45ae170c9fd">Size of data to be considered for batch sorting during data loading. The recommended value is less than 45% of the total sort data. The unit is MB.</p>
|
|
<div class="note" id="mrs_01_1404__n62f3a3dca29144cb9915cb3d4c4c04b2"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1404__a24bf183ec4214250b38d456731b6dd4b">If this parameter is not set, its value is about 45% of the value of <strong id="mrs_01_1404__b1955242215104735">sort.inmemory.size.inmb</strong> by default.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r36f3ca2a56674107b219d71012940372"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a258468ad8e7f4082a652728ea88172d8">enable.unsafe.columnpage</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a80ff77850f9e4f3bb7459823da5934f6">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__aaeeb6f3309734ffa94b26c1bf1ea80ac">Whether to keep page data in heap memory during data loading or query to prevent garbage collection bottleneck.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rb73e1e404f1d4118846e678979a5684f"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__af04c79d9fe2a4b46a33d050afdb80cd2">carbon.use.local.dir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a0f2d143fba2146e6a81baf5a92c4b59b">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__ac5f3f5b2e65f4136947ad12dc790f65b">Whether to use Yarn local directories for multi-disk data loading. If this parameter is set to <strong id="mrs_01_1404__b1224029641104735">true</strong>, Yarn local directories are used to load multi-disk data to improve data loading performance.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r388617122fc546e799542d4577a5ae32"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a5d652352c1f54468811646d69dd099ce">carbon.use.multiple.temp.dir</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a2a5bc1fd7c6a4d2e9714a99da4df8ba0">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__acbcf7fc83c5440658a4e1324761ff3f0">Whether to use multiple temporary directories for storing temporary files to improve data loading performance.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row78301313105914"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__p283071335913">carbon.load.datamaps.parallel.db_name.table_name</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__p108309133596">N/A</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__p683081315916">The value can be <strong id="mrs_01_1404__b1691680626104735">true</strong> or <strong id="mrs_01_1404__b504328795104735">false</strong>. You can set the database name and table name to improve the first query performance of the table.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rf6720f89e9b346df97df0434296eb3c3"><td class="cellrowborder" colspan="3" valign="top" headers="mcps1.3.3.2.4.1.1 mcps1.3.3.2.4.1.2 mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__ad97e82069c1e40ef9d6df51579c17e5f"><strong id="mrs_01_1404__a3d621300754546d09b62afc00cb0eb4b">Compaction Configuration</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r24a818ae5e5c4fad886f327850d3f819"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__adc07e84d6e224e0a85854b6d167eccc9">carbon.number.of.cores.while.compacting</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a545757e6e92d4ce5911e0c535c807888">2</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__ab224d7fe878b40f7af841bd1f874f8f8">Number of cores to be used while compacting data. The greater the number of cores, the better the compaction performance. If the CPU resources are sufficient, you can increase the value of this parameter.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r44ee6d4d66d14bb78e9cf7742e253251"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a3f49e1242d49494684f3ec9bd39ffcc8">carbon.compaction.level.threshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a1270cc9d6ea14db6bb0113343803034c">4,3</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a24d8c16b40904984a288f693f40f7d30">This configuration is for minor compaction which decides how many segments to be merged.</p>
|
|
<p id="mrs_01_1404__ab0b30e1c2be44ab887d6077c7428ba9f">For example, if this parameter is set to <span class="parmvalue" id="mrs_01_1404__pce1d4171df8c4d1bb275220f318614c2"><b>2,3</b></span>, minor compaction is triggered every two segments. <span class="parmvalue" id="mrs_01_1404__pe48558c1acbe473da2c99a7fb6448f6e"><b>3</b></span> is the number of level 1 compacted segments which is further compacted to new segment.</p>
|
|
<p id="mrs_01_1404__ab91dd09d621742c9ac34919c608943f0">The value ranges from 0 to 100.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r5e1b770da1c6488b89f0f854b164007f"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__ae5302f28649a44a2bbf730589ad99a5b">carbon.major.compaction.size</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a7ad0700ab67e450f8767bb983d7504fc">1024</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a5baaa9cd32464881a849d12c10469734">Major compaction size. Sum of the segments which is below this threshold will be merged.</p>
|
|
<p id="mrs_01_1404__ae86d78534fdc42f9995abf27af88bce3">The unit is MB.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__re770324180554f49bfb4c218e95503db"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__ae9ccd44ce0e34ffc8caf9325c58e0fc3">carbon.horizontal.compaction.enable</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a94fac725aed1495a97794a30af5d8621">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a9af1bdc68a30478ab9cc7e79f7bc6cf9">Whether to enable/disable horizontal compaction. After every DELETE and UPDATE statement, horizontal compaction may occur in case the incremental (DELETE/ UPDATE) files becomes more than specified threshold. By default, this parameter is set to <span class="parmvalue" id="mrs_01_1404__p49f94eb9a5e44897909f2abd2d96c1d3"><b>true</b></span>. You can set this parameter to <span class="parmvalue" id="mrs_01_1404__p73ee549a4be040e58aa9f0209df74323"><b>false</b></span> to disable horizontal compaction.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rf9c374f82bac4aeb89681c57468ba055"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a70aeedb4319d44c49e748f06b5b594a0">carbon.horizontal.update.compaction.threshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a37d3798258184b93b3c817633bce58ad">1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__ae381d0ab56c14b50ae8b6b01c64e2791">Threshold limit on number of UPDATE delta files within a segment. In case the number of delta files goes beyond the threshold, the UPDATE delta files within the segment becomes eligible for horizontal compaction and are compacted into single UPDATE delta file. By default, this parameter is set to <strong id="mrs_01_1404__b1137246854104735">1</strong>. The value ranges from <strong id="mrs_01_1404__b456562901104735">1</strong> to <strong id="mrs_01_1404__b951413717104735">10000</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rd634d026e3f0462184a5ad98f529f27d"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a5828052483e74164ad82192b40d33521">carbon.horizontal.delete.compaction.threshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__ad95a0cfb9edc469994c1f8f2cc4ca650">1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a53fefd6e1c824c6f94164d355334bcf4">Threshold limit on number of DELETE incremental files within a block of a segment. In case the number of incremental files goes beyond the threshold, the DELETE incremental files for the particular block of the segment becomes eligible for horizontal compaction and are compacted into single DELETE incremental file. By default, this parameter is set to <strong id="mrs_01_1404__b809384660104735">1</strong>. The value ranges from <strong id="mrs_01_1404__b92599859104735">1</strong> to <strong id="mrs_01_1404__b1976317084104735">10000</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r1b7fbf0b721b4a339517b7b5232b0ef1"><td class="cellrowborder" colspan="3" valign="top" headers="mcps1.3.3.2.4.1.1 mcps1.3.3.2.4.1.2 mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__aca0296ebda164c84ad9a1b83d91dcc4c"><strong id="mrs_01_1404__abcad9792b5b440cab37c73a94894acf4">Query Configuration</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__ref9665aea9d6421680716f0eefb98b31"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__afbc61dc97100432991884526a15d0eb1">carbon.number.of.cores</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__aca0b2320e54443e19523f65fa7f3e091">4</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__abdc3d8de250d4b7faaacd3ec72552f4e">Number of cores to be used during query</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r5ce234c593964608ae917f1de5f5928a"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__ab7858681fa764a1595107850e5f43a84">carbon.limit.block.distribution.enable</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__ab318814cbcb8486891d9d2a05b717b48">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__aa77c3faf8c7741c09418919eda0e3abe">Whether to enable the CarbonData distribution for <span class="keyword" id="mrs_01_1404__ke263c58cbac249569011113b05776df8">limit</span> query. The default value is <span class="parmvalue" id="mrs_01_1404__p70876c03ba6444ebb93d5bc5c01adda1"><b>false</b></span>, indicating that block distribution is disabled for query statements that contain the keyword <span class="keyword" id="mrs_01_1404__k60c3f30936a449409c30cfba22a3bcf8">limit</span>. For details about how to optimize this parameter, see <a href="mrs_01_1421.html">Configurations for Performance Tuning</a>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r1cef48fa9fda4dfca12fa371e0213b6b"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a78e1b3ab559941b7a2b1bcbe8a1966ca">carbon.custom.block.distribution</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__abaf6d3db00ce40a7ba499a2fd60cef8b">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a5ec82edb7fc64613a91539c30cc7d098">Whether to enable Spark or CarbonData block distribution. By default, the value is <span class="parmvalue" id="mrs_01_1404__p194fc313f5394d928b8e582b8fb5dda3"><b>false</b></span>, indicating that Spark block distribution is enabled. To enable CarbonData block distribution, change the value to <span class="parmvalue" id="mrs_01_1404__p2113bee121bd404d8285d434b8b2f84a"><b>true</b></span>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r260dffcea5784b3ea96921bc7de6396a"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__afeee67c1979d4eb78a3e979230ff89d6">carbon.infilter.subquery.pushdown.enable</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__afffe5c7cf91b45d1b2af474abd021c28">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a74491d617573459c8388ad527310eb04">If this is set to <strong id="mrs_01_1404__b482396136104735">true</strong> and a Select query is triggered in the filter with subquery, the subquery is executed and the output is broadcast as IN filter to the left table. Otherwise, SortMergeSemiJoin is executed. You are advised to set this to <strong id="mrs_01_1404__b1382709338104735">true</strong> when IN filter subquery does not return too many records. For example, when the IN sub-sentence query returns 10,000 or fewer records, enabling this parameter will give the query results faster.</p>
|
|
<p id="mrs_01_1404__abc01c95b8c244009bdbed521028e1be0">Example: <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_1404__cmdname17617131316127">select * from flow_carbon_256b where cus_no in (select cus_no from flow_carbon_256b where dt>='20260101' and dt<='20260701' and txn_bk='tk_1' and txn_br='tr_1') limit 1000;</span></b></i></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r967ba54d8fbe4fb589b47b156138072c"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a398b00caadcc48629c2597c9959423f7">carbon.scheduler.minRegisteredResourcesRatio</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a747d1716c930410caf84f6deb613f317">0.8</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__ab0737f76503c42a5996edd07a0df8428">Minimum resource (executor) ratio needed for starting the block distribution. The default value is <span class="parmvalue" id="mrs_01_1404__p51fcb26938dd42ef856b0b95fe0df490"><b>0.8</b></span>, indicating that 80% of the requested resources are allocated for starting block distribution.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rdf21c1b1ea8a4949af365e0678eb7047"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a9e4ab824fe324d84b2700252ed2fef9e">carbon.dynamicAllocation.schedulerTimeout</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a1377fb6546fa497d93068a06bb77b366">5</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a7707a416fc924feb9ed582776e870beb">Maximum time that the scheduler waits for executors to be active. The default value is <span class="parmvalue" id="mrs_01_1404__p11405ab9f65c492788e82d2853bc87e1"><b>5</b></span> seconds, and the maximum value is <span class="parmvalue" id="mrs_01_1404__pcc9db4df176046f4941d69c9f731ec1d"><b>15</b></span> seconds.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__ra62fec1e492d40b9b8a7acc01d88dd23"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__abb2195dc4a124630abb271bc6cd54f03">enable.unsafe.in.query.processing</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__ac962598d1c7e4be9b7c01a73e183507f">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__a3fb8d493dfae4a1baab63b019d76fcf4">Whether to use unsafe sort during query. Unsafe sort reduces the garbage collection during query, resulting in better performance. The default value is <span class="parmvalue" id="mrs_01_1404__p2331c2c509cc41c399979bd13706d643"><b>true</b></span>, indicating that unsafe sort is enabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rc8a28c8c95c64c97957f1101fe636e21"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__ae76fd0e57f854b908bef35cbf72dc084">carbon.enable.vector.reader</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__a8977efd1a48e48698aeaf1e6c44ca49c">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__aad5569cc61874738a34130c3ad958b9d">Whether to enable vector processing for result collection to improve query performance</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row99971347131416"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__p199724731414">carbon.query.show.datamaps</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__p399784711420">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__p209971347181413"><b><span class="cmdname" id="mrs_01_1404__cmdname188711934141514">SHOW TABLES</span></b> lists all tables including the primary table and datamaps. To filter out the datamaps, set this parameter to <strong id="mrs_01_1404__b597724903104735">false</strong>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r90e1e16857a8405896b5e89f714df297"><td class="cellrowborder" colspan="3" valign="top" headers="mcps1.3.3.2.4.1.1 mcps1.3.3.2.4.1.2 mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__adba75c5a8b6141979b5510e230be4d0b"><strong id="mrs_01_1404__a283941c067014db9a7c7d2976c13d80c">Secondary Index Configuration</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rf9fc6a5178e94d95a1cfbfb792d513d6"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a1cce4685644142a89989570eacd61fcd">carbon.secondary.index.creation.threads</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__aa39e30a6ed0241b78bb105fad4bc88a0">1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__aa34ff35119924975b65c8afa3698c71f">Number of threads to concurrently process segments during secondary index creation. This property helps fine-tuning the system when there are a lot of segments in a table. The value ranges from 1 to 50.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r1242d385c7f547dd8c3ad9dceaae884e"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__a2a8bb7563cb540b2aa8361914663981c">carbon.si.lookup.partialstring</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__ae77b417ac8d445e297a9ca088425a9b1">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><ul id="mrs_01_1404__uf5c6890d93514e18afdfcdce795b49ae"><li id="mrs_01_1404__l43be1a896c9742a1aab435775c51bf4e">When the parameter value is <strong id="mrs_01_1404__b2074278829104735">true</strong>, it includes indexes started with, ended with, and contained.</li><li id="mrs_01_1404__l0213bb02d639447f9c487bba3749a7ad">When the parameter value is <strong id="mrs_01_1404__b700567824104735">false</strong>, it includes only secondary indexes started with.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row05464498562"><td class="cellrowborder" valign="top" width="18.67%" headers="mcps1.3.3.2.4.1.1 "><p id="mrs_01_1404__p154644955610">carbon.si.segment.merge</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="14.979999999999999%" headers="mcps1.3.3.2.4.1.2 "><p id="mrs_01_1404__p10546134919568">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="66.35%" headers="mcps1.3.3.2.4.1.3 "><p id="mrs_01_1404__p10546549105616">Enabling this property merges <strong id="mrs_01_1404__b92291881915">.carbondata</strong> files inside the secondary index segment. The merging will happen after the load operation. That is, at the end of the secondary index table load, small files are checked and merged.</p>
|
|
<div class="note" id="mrs_01_1404__note119768411255"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1404__p997654117514">Table Block Size is used as the size threshold for merging small files.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1404__td75b0415aa2043ab98f02b438267e3c8" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Other configurations in <strong id="mrs_01_1404__b1354832342316">carbon.properties</strong></caption><thead align="left"><tr id="mrs_01_1404__r41be7fffe13e49f081e73bf4b3e2b1ca"><th align="left" class="cellrowborder" valign="top" width="18.48%" id="mcps1.3.4.2.4.1.1"><p id="mrs_01_1404__a5388678567ad41c3b9025ece5a7def9b">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="21.310000000000002%" id="mcps1.3.4.2.4.1.2"><p id="mrs_01_1404__ae5959a3013774dda9f0e25d835db13c7">Default Value</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="60.209999999999994%" id="mcps1.3.4.2.4.1.3"><p id="mrs_01_1404__a067675b728fb415d86c4138bcf8daf88">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1404__r64cfcebe35fd4be8a7aed62b5b525a86"><td class="cellrowborder" colspan="3" valign="top" headers="mcps1.3.4.2.4.1.1 mcps1.3.4.2.4.1.2 mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a9d267d82d99843e48430aac51efd4a3e"><strong id="mrs_01_1404__a40a295908f3e41d082905c4f06436c82">Data Loading Configuration</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rd9f30bc907f94b29afa25b19f9bc73f6"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__ae3fc222cae0c41b2b33e3103733a98b6">carbon.lock.type</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__aff0665c509074d3ca03ac644d9cb0fac">HDFSLOCK</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a7b2313db51454d36b5122c062fe1fd80">Type of lock to be acquired during concurrent operations on a table. </p>
|
|
<p id="mrs_01_1404__af31940b35e0d4697a89ec3da2661e09f">There are following types of lock implementation: </p>
|
|
<ul id="mrs_01_1404__u88c438c3a7f44513b767a5c034d4b35c"><li id="mrs_01_1404__l35a1d89806754fe493bc9d3723b524b4"><strong id="mrs_01_1404__b413582695104735">LOCALLOCK</strong>: Lock is created on local file system as a file. This lock is useful when only one Spark driver (or JDBCServer) runs on a machine.</li><li id="mrs_01_1404__lb54376886fba40d2bfb95cb4571ac30b"><strong id="mrs_01_1404__b627998577104735">HDFSLOCK</strong>: Lock is created on HDFS file system as a file. This lock is useful when multiple Spark applications are running and no ZooKeeper is running on a cluster.</li></ul>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rdbff704dba234070a7feef5649710a56"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__a1d128edc571745258c7edbe9ba3ecc11">carbon.sort.intermediate.files.limit</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__a27c4aa70d8564e0a99d97566dfe51421">20</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a768c79c9e8f8479cb0661e93bde4cbca">Minimum number of intermediate files. After intermediate files are generated, sort and merge the files. For details about how to optimize this parameter, see <a href="mrs_01_1421.html">Configurations for Performance Tuning</a>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r16721a38e4924c89b84327c6ce4638b8"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__afc85cdb80c4a4ac7b89b22823acd0c16">carbon.csv.read.buffersize.byte</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__a7b50d1a42565452b885c03a2e3444cb9">1048576</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__afff5bb874e1b4d32869a49a192e8ca71">Size of CSV reading buffer</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r935fe6f8f5a7482aabf049e01225aeae"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__ac4085a81bf6c45d6b7165ba21cbed74a">carbon.merge.sort.reader.thread</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__a53e88f693ac7492cb54007a2f8364bcf">3</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a7ae32fe074154293895c77fead118985">Maximum number of threads used for reading intermediate files for final merging.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row3479443513"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__p149090014615">carbon.concurrent.lock.retries</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__p4480545511">100</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__p144807411514">Maximum number of retries used to obtain the concurrent operation lock. This parameter is used for concurrent loading.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row1727324518912"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__p108455539916">carbon.concurrent.lock.retry.timeout.sec</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__p1327313451914">1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__p1062511148114">Interval between the retries to obtain the lock for concurrent operations.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rf22aee29390544cd9001b01ba524aab5"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__p8418154221210">carbon.lock.retries</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__ac64a9d520c7244abb5e1f4abcecbf592">3</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__abc1f57fdaf534df29e81198c9ba4af3c">Maximum number of retries to obtain the lock for any operations other than import.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r711f9cf2a1d5440db39e2d34df911aa9"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__p6585161601513">carbon.lock.retry.timeout.sec</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__a37fa7a0c1e4d452f991a634f9661f3c6">5</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__abc9ae9bd65df4c81a17f6bd981c85ad8">Interval between the retries to obtain the lock for any operation other than import.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rcf9ce76d3e6143b082a5107ec25d0955"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__a4d5cee76082843fa9a68a426aaac8166">carbon.tempstore.location</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__a342856d891d84fa7b960552e37581d83">/opt/Carbon/TempStoreLoc</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a0de169994ea44c0ab898a19994dc504c">Temporary storage location. By default, the <span class="parmvalue" id="mrs_01_1404__p493b171fe47441719874ed3d5d19a952"><b>System.getProperty("java.io.tmpdir")</b></span> method is used to obtain the value. For details about how to optimize this parameter, see the description of <span class="parmname" id="mrs_01_1404__pb2a5a0867b0b42d480111a62ba6d9fc1"><b>carbon.use.local.dir</b></span> in <a href="mrs_01_1421.html">Configurations for Performance Tuning</a>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rea5bb4dd0bd042509ebb251372130ec6"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__aba9d411604224654a29a5cb8eb78c827">carbon.load.log.counter</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__acfb19ca530394caaaddc290a20e1b876">500000</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a32b2fc16036649bb98366a0b4fffbb6f">Data loading records count in logs</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r0df04c8a7a804ed7b635f7fc7adbcfc1"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__a30452fd1f755406fab4797f51f10a7ce">SERIALIZATION_NULL_FORMAT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__ad2a6c510394d44b09d82f3a2300b3a3e">\N</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a1b46ad0196824b51a8973b161395caa0">Value to be replaced with NULL</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row1281734961615"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__p27207599161">carbon.skip.empty.line</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__p4818124931616">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__p17818144912169">Setting this property will ignore the empty lines in the CSV file during data loading.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row12777219112"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__p177771819811">carbon.load.datamaps.parallel</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__p677791911116">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__p377715199118">Whether to enable parallel datamap loading for all tables in all sessions. This property will improve the time to load datamaps into memory by distributing the job among executors, thus improving query performance.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r16dbcf2d54f6466f83f664497527ea32"><td class="cellrowborder" colspan="3" valign="top" headers="mcps1.3.4.2.4.1.1 mcps1.3.4.2.4.1.2 mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__aca904c76b1af42c69c27c3829b89607b"><strong id="mrs_01_1404__a1b87206bbf1c42d789bdb6a05e20c736">Merging Configuration</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r23878f85dcfe4233b9e3ed9b93134209"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__af1e2caa57f1f43158564119f4db880fb">carbon.numberof.preserve.segments</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__a14f2b5a93f76496db58c2dddf3817589">0</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__aca2286f88a5e4458b82113f966bceddd">If you want to preserve some number of segments from being compacted, then you can set this configuration.</p>
|
|
<p id="mrs_01_1404__aa9a4a379fabd4d64b1cb5cbbf83d112f">For example, if <span class="parmname" id="mrs_01_1404__p4500763c9e6946f890a03cedb4847cf7"><b>carbon.numberof.preserve.segments</b></span> is set to <span class="parmvalue" id="mrs_01_1404__p624fa5c81cff4fbe82f72d6b80886464"><b>2</b></span>, the latest two segments will always be excluded from the compaction.</p>
|
|
<p id="mrs_01_1404__a26e01dc58f1942b8a79ce7112d80bb5f">No segments will be preserved by default.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r25b71bb90828495296a87476df4a9f06"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__ae4c723708c914ed9a64c35fbae0255fb">carbon.allowed.compaction.days</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__ae0ab6e707bee44b2bd801a10b01a2cd4">0</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__aca46d4449d854e63bada3a4d335e4922">This configuration is used to control on the number of recent segments that needs to be merged.</p>
|
|
<p id="mrs_01_1404__a9eaac55fd0454e859035871b9c7d56c4">For example, if this parameter is set to <span class="parmvalue" id="mrs_01_1404__p8d8ab73912f4444081c4f7e43679b5eb"><b>2</b></span>, the segments which are loaded in the time frame of past 2 days only will get merged. Segments which are loaded earlier than 2 days will not be merged.</p>
|
|
<p id="mrs_01_1404__a5e9640a064fa46cfbbf1a849b1eed2e9">This configuration is disabled by default.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rf2c613aee3034203b0df2c66b77e44ac"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__a9d2d6055f42a4ffeaa36d782a8434ae3">carbon.enable.auto.load.merge</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__a18b881ed101241b0bdcce33f8147ec4d">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__af1c0da27f77c4ccaa2d3f5d14efb1209">Whether to enable compaction along with data loading.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row1357105441713"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__p1095985515173">carbon.merge.index.in.segment</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__p057110548172">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__p1157195413178">This configuration enables to merge all the CarbonIndex files (<strong id="mrs_01_1404__b1682117181796">.carbonindex</strong>) into a single MergeIndex file (<strong id="mrs_01_1404__b119915311920">.carbonindexmerge</strong>) upon data loading completion. This significantly reduces the delay in serving the first query.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r2ee3cd61ff66422a81260fa9c8d8a730"><td class="cellrowborder" colspan="3" valign="top" headers="mcps1.3.4.2.4.1.1 mcps1.3.4.2.4.1.2 mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__acff0bdc9b7cf4871a2fc0e88a353db4f"><strong id="mrs_01_1404__aa401e2d4bf2a460fb438ffdfb672764e">Query Configuration</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r416858c870f0421ab2ca272f5ad67280"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__a324393cd66d245aa9f6dd13dc774f43a">max.query.execution.time</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__a9be1f2c10b67414baa251b83bd3939a2">60</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a05dd68257d1148d7b1d1f8fd5288566e">Maximum time allowed for one query to be executed. </p>
|
|
<p id="mrs_01_1404__ace0af524b98a4669862dbe17c6915ada">The unit is minute.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rcede48bcf6e346e7b137ecac4781b9a4"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__ae155bc03b3b84d69b3f19d187864307f">carbon.enableMinMax</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__ae1871640794f465c8e6093e1f3670bdc">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a0b1fc2577660433c8d7ae8525edf8c23">MinMax is used to improve query performance. You can set this to <strong id="mrs_01_1404__b1908479534104735">false</strong> to disable this function.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r269eb69ee3a64a3ea543e408b06d0818"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__ae91a58a683ca4229ac3f191854aa251d">carbon.lease.recovery.retry.count</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__ae287018d2edb40f293b5ba392b6f002e">5</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a827259faae4b459482d8e1ca004ab83d">Maximum number of attempts that need to be made for recovering a lease on a file.</p>
|
|
<p id="mrs_01_1404__a443f10b8d07f4bedb6305bde1faefd3e">Minimum value: <strong id="mrs_01_1404__b768500793104735">1</strong></p>
|
|
<p id="mrs_01_1404__ac103be1ebd3949c8ae3a3f531e1a094d">Maximum value: <strong id="mrs_01_1404__b1831522372104735">50</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r130f962a2baa43948354a3dfc0cea040"><td class="cellrowborder" valign="top" width="18.48%" headers="mcps1.3.4.2.4.1.1 "><p id="mrs_01_1404__a0c0904c58cf3457aafc1c72849023561">carbon.lease.recovery.retry.interval</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="21.310000000000002%" headers="mcps1.3.4.2.4.1.2 "><p id="mrs_01_1404__af75109fdaffe4119b64b67dce9bd9837">1000 (ms)</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="60.209999999999994%" headers="mcps1.3.4.2.4.1.3 "><p id="mrs_01_1404__a3b0656c5c868466a8d58c186d7e8b041">Interval or pause time after a lease recovery attempt is made on a file.</p>
|
|
<p id="mrs_01_1404__a5835134c2f3b47afb161d79b5d097f96">Minimum value: <strong id="mrs_01_1404__b568765828104735">1000</strong> (ms)</p>
|
|
<p id="mrs_01_1404__a1b83d40df5644f72abcc509b27eeec11">Maximum value: <strong id="mrs_01_1404__b2144752913104735">10000</strong> (ms)</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1404__t1b6130ca31344a93b827fe435e7f26ec" frame="border" border="1" rules="all"><caption><b>Table 4 </b>Spark configuration reference in <strong id="mrs_01_1404__b23726170104735">spark-defaults.conf</strong></caption><thead align="left"><tr id="mrs_01_1404__r1f9c9656e0ec4b6da4116e922b2f9b89"><th align="left" class="cellrowborder" valign="top" width="18.3%" id="mcps1.3.5.2.4.1.1"><p id="mrs_01_1404__ad57d7837ac35491faaa60d39758b4c89"><strong id="mrs_01_1404__a670b0244bb2c4d19a270a1c0f7875e37">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="24.01%" id="mcps1.3.5.2.4.1.2"><p id="mrs_01_1404__a42cc67f1ff0244998138e3a316c91e0a"><strong id="mrs_01_1404__a7e381524fd814fc39b6ee6416d026c1c">Default Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="57.69%" id="mcps1.3.5.2.4.1.3"><p id="mrs_01_1404__a8ad46e6e574e44288f2478dd3f2483e3"><strong id="mrs_01_1404__acb14e092ac914a9994bb3337ca652bd4">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1404__r15dcdd59aca7442a8d0bf374f73a93fb"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1404__ac68a2ec1d0c94a8baa90c84b9c498477">spark.driver.memory</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="24.01%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1404__a9d4e4d10cd714d5bab92ed0c03f3d831">4G</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="57.69%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1404__ae784ee55601a4345829ab0009307a075">Memory to be used for the driver process. SparkContext has been initialized.</p>
|
|
<div class="note" id="mrs_01_1404__nb130e06f240a430e9815b742b995b738"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1404__ae640baffad4848d8965fe7837a1b3e3f">In client mode, do not use SparkConf to set this parameter in the application because the driver JVM has been started. To configure this parameter, configure it in the <strong id="mrs_01_1404__b1273114114122">--driver-memory</strong> command-line option or in the default property file.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__rc2d7e4fc8b6d458ebbca32f7dab7e8ec"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1404__a1e997d2e951342c8b344cb88f862f364">spark.executor.memory</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="24.01%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1404__ad39216f3fa7f473e9a98b0f5b8b9bc70">4 GB</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="57.69%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1404__a448fe9a4a4af402faebd63866e4d3535">Memory to be used for each executor process.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__r54201747c1cb46cb9a8d5abb9548eb01"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.5.2.4.1.1 "><p id="mrs_01_1404__a473d0d09c01947aeae9ee638ef0a1f9d">spark.sql.crossJoin.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="24.01%" headers="mcps1.3.5.2.4.1.2 "><p id="mrs_01_1404__ad385292057bc4ceb9687df9d75bb8d1b">true</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="57.69%" headers="mcps1.3.5.2.4.1.3 "><p id="mrs_01_1404__a6bd4150a41334e369086ec4c5902b2b6">If the query contains a cross join, enable this property so that no error is thrown. In this case, you can use a cross join instead of a join for better performance.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_1404__a56f3123938ef4fc1a776930b0a904f08">Configure the following parameters in the <span class="filepath" id="mrs_01_1404__f4131a2ceb7c84670b3845617fc947310"><b>spark-defaults.conf</b></span> file on the Spark driver.</p>
|
|
<ul id="mrs_01_1404__ud71f2fa2572642dabafb1856e0ceb375"><li id="mrs_01_1404__l2b042083ba6c4288a5f4eba4af29c1e4">In spark-sql mode:
|
|
<div class="tablenoborder"><a name="mrs_01_1404__ta902cd071dfb426097416a5c7034ee6c"></a><a name="ta902cd071dfb426097416a5c7034ee6c"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1404__ta902cd071dfb426097416a5c7034ee6c" frame="border" border="1" rules="all"><caption><b>Table 5 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1404__r246afbb59dcc4f34b0cb6762faa93246"><th align="left" class="cellrowborder" valign="top" width="18.3%" id="mcps1.3.7.1.1.2.4.1.1"><p id="mrs_01_1404__af494fc47cf734fd2822a335bbd8bede5"><strong id="mrs_01_1404__aca81b2c783304f3089ec3387e90ced8c">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="42.699999999999996%" id="mcps1.3.7.1.1.2.4.1.2"><p id="mrs_01_1404__a563839d5260e471ca783333c57fa830e"><strong id="mrs_01_1404__acd7d04342de549ceb543ff6d689fd450">Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="39%" id="mcps1.3.7.1.1.2.4.1.3"><p id="mrs_01_1404__ac77cf6d5162642c59c42b2847166b8fe"><strong id="mrs_01_1404__a202ce110bf734d1abd93cb1fa6a9bd99">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1404__rc1d6f0a06cdf4ea8926761f6819fee89"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.7.1.1.2.4.1.1 "><p id="mrs_01_1404__a7602fc44a7c24240a72fc064894a3bfb"><span id="mrs_01_1404__pb6b3f54765194b809a255cc13c1fdc24">spark.driver.extraJavaOptions</span></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="42.699999999999996%" headers="mcps1.3.7.1.1.2.4.1.2 "><p id="mrs_01_1404__p103624445411">-Dlog4j.configuration=file:/opt/client/Spark2x/spark/conf/log4j.properties -Djetty.version=x.y.z -Dzookeeper.server.principal=zookeeper/hadoop.<em id="mrs_01_1404__i1954248201617"><System domain name></em> -Djava.security.krb5.conf=/opt/client/KrbClient/kerberos/var/krb5kdc/krb5.conf -Djava.security.auth.login.config=/opt/client/Spark2x/spark/conf/jaas.conf -Dorg.xerial.snappy.tempdir=/opt/client/Spark2x/tmp -Dcarbon.properties.filepath=/opt/client/Spark2x/spark/conf/carbon.properties -Djava.io.tmpdir=/opt/client/Spark2x/tmp</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39%" headers="mcps1.3.7.1.1.2.4.1.3 "><p id="mrs_01_1404__adeb9d4c1db37460bb3169c2062a428e2"><span id="mrs_01_1404__pa89e68e0505a4f6cbb45a0a526c90205">The default value</span> <span class="filepath" id="mrs_01_1404__f01d126358c7f4f6589cca06bcfae6705"><b>/opt/client/Spark2x/spark</b></span> <span id="mrs_01_1404__p031cf640c99746bbad6d7672a966b26b">indicates <strong id="mrs_01_1404__b5594173010426">CLIENT_HOME</strong> of the client and is added to the end of the value of</span> <span class="parmname" id="mrs_01_1404__pceb90c71406945b4b24d4f9036384d64"><b>spark.driver.extraJavaOptions</b></span>. <span id="mrs_01_1404__pf263028fb00b4aabae3b46f2f5b15cb3">This parameter is used to specify the path of the</span> <span class="filepath" id="mrs_01_1404__f2c615f22157b466aa9631cedc5016f33"><b>carbon.properties</b></span><span id="mrs_01_1404__pae5ef8547eea461d84b734d362a783df">file in Driver.</span></p>
|
|
<div class="note" id="mrs_01_1404__nf1fbea1931394260b966951f5518d513"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1404__a9b5259756fed40e783960f08475abb91">Spaces next to equal marks (=) are not allowed.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row7480638145518"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.7.1.1.2.4.1.1 "><p id="mrs_01_1404__p173244559580">spark.sql.session.state.builder</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="42.699999999999996%" headers="mcps1.3.7.1.1.2.4.1.2 "><p id="mrs_01_1404__p16299158195920">org.apache.spark.sql.hive.FIHiveACLSessionStateBuilder</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39%" headers="mcps1.3.7.1.1.2.4.1.3 "><p id="mrs_01_1404__p048014386559">Session state constructor.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row1625203585916"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.7.1.1.2.4.1.1 "><p id="mrs_01_1404__p134671253195915">spark.carbon.sqlastbuilder.classname</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="42.699999999999996%" headers="mcps1.3.7.1.1.2.4.1.2 "><p id="mrs_01_1404__p196035532115">org.apache.spark.sql.hive.CarbonInternalSqlAstBuilder</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39%" headers="mcps1.3.7.1.1.2.4.1.3 "><p id="mrs_01_1404__p15625173513591">AST constructor.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row1094294219594"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.7.1.1.2.4.1.1 "><p id="mrs_01_1404__p74379413117">spark.sql.catalog.class</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="42.699999999999996%" headers="mcps1.3.7.1.1.2.4.1.2 "><p id="mrs_01_1404__p145301609217">org.apache.spark.sql.hive.HiveACLExternalCatalog</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39%" headers="mcps1.3.7.1.1.2.4.1.3 "><p id="mrs_01_1404__p14943164255914">Hive External catalog to be used. This parameter is mandatory if Spark ACL is enabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row186071039175920"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.7.1.1.2.4.1.1 "><p id="mrs_01_1404__p1435918471510">spark.sql.hive.implementation</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="42.699999999999996%" headers="mcps1.3.7.1.1.2.4.1.2 "><p id="mrs_01_1404__p1285910715219">org.apache.spark.sql.hive.HiveACLClientImpl</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39%" headers="mcps1.3.7.1.1.2.4.1.3 "><p id="mrs_01_1404__p8607173935918">How to call the Hive client. This parameter is mandatory if Spark ACL is enabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row1934117136216"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.7.1.1.2.4.1.1 "><p id="mrs_01_1404__p19808220625">spark.sql.hiveClient.isolation.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="42.699999999999996%" headers="mcps1.3.7.1.1.2.4.1.2 "><p id="mrs_01_1404__p789915271424">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39%" headers="mcps1.3.7.1.1.2.4.1.3 "><p id="mrs_01_1404__p13341111318213">This parameter is mandatory if Spark ACL is enabled.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li></ul>
|
|
<ul id="mrs_01_1404__u8f413618fd1d4c929e2ff3103fa2e6d6"><li id="mrs_01_1404__l6c4c8098762d4052a5d77138d22e40eb">In JDBCServer mode:
|
|
<div class="tablenoborder"><a name="mrs_01_1404__t3897ae14f205433fb0f98b79411cfa0c"></a><a name="t3897ae14f205433fb0f98b79411cfa0c"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1404__t3897ae14f205433fb0f98b79411cfa0c" frame="border" border="1" rules="all"><caption><b>Table 6 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1404__rc34e389c7357412aaa19afabfbec948a"><th align="left" class="cellrowborder" valign="top" width="18.3%" id="mcps1.3.8.1.1.2.4.1.1"><p id="mrs_01_1404__ad723fda6bb7a4b23988f9f0ff860df8e"><strong id="mrs_01_1404__a498fd496521e40b598d4c91dc32182a8">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="25.06%" id="mcps1.3.8.1.1.2.4.1.2"><p id="mrs_01_1404__a92fe2262957d4790805fc7634a5b96cb"><strong id="mrs_01_1404__a0852f8cf24a64630ad4278654d9ac8b7">Value</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="56.64%" id="mcps1.3.8.1.1.2.4.1.3"><p id="mrs_01_1404__aeae17335d31d4200b7448cf17eb34fba"><strong id="mrs_01_1404__a9434b56edb424ec6a34e8e8d9497dccf">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1404__r914cc469dbab4f2f867aa8ed8cc141bb"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.8.1.1.2.4.1.1 "><p id="mrs_01_1404__a5394d1b4143e4ad39b95f07287cdb04f">spark.driver.extraJavaOptions</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="25.06%" headers="mcps1.3.8.1.1.2.4.1.2 "><p id="mrs_01_1404__p19142949843">-Xloggc:${SPARK_LOG_DIR}/indexserver-omm-%p-gc.log -XX:+PrintGCDetails -XX:-OmitStackTraceInFastThrow -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:MaxDirectMemorySize=512M -XX:MaxMetaspaceSize=512M -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=10M -XX:OnOutOfMemoryError='kill -9 %p' -Djetty.version=x.y.z -Dorg.xerial.snappy.tempdir=${BIGDATA_HOME}/tmp/spark2x/JDBCServer/snappy_tmp -Djava.io.tmpdir=${BIGDATA_HOME}/tmp/spark2x/JDBCServer/io_tmp -Dcarbon.properties.filepath=${SPARK_CONF_DIR}/carbon.properties -Djdk.tls.ephemeralDHKeySize=2048 -Dspark.ssl.keyStore=${SPARK_CONF_DIR}/child.keystore #{java_stack_prefer}</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="56.64%" headers="mcps1.3.8.1.1.2.4.1.3 "><p id="mrs_01_1404__a3a0aba11dc53498ebf2d3e108e2b260d">The default value <strong id="mrs_01_1404__b1061941785104735">${SPARK_CONF_DIR}</strong> depends on a specific cluster and is added to the end of the value of the <strong id="mrs_01_1404__b1899947449104735">spark.driver.extraJavaOptions</strong> parameter. This parameter is used to specify the path of the <strong id="mrs_01_1404__b1634983568104735">carbon.properties</strong> file in Driver.</p>
|
|
<div class="note" id="mrs_01_1404__n5840263993d34ae9a557c798602a345e"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1404__a81d83d9372e64acd98a4233ccfae8b59">Spaces next to equal marks (=) are not allowed.</p>
|
|
</div></div>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row15289145201615"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.8.1.1.2.4.1.1 "><p id="mrs_01_1404__p10203132541618">spark.sql.session.state.builder</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="25.06%" headers="mcps1.3.8.1.1.2.4.1.2 "><p id="mrs_01_1404__p720982511164">org.apache.spark.sql.hive.FIHiveACLSessionStateBuilder</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="56.64%" headers="mcps1.3.8.1.1.2.4.1.3 "><p id="mrs_01_1404__p12217125101614">Session state constructor.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row1280281919161"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.8.1.1.2.4.1.1 "><p id="mrs_01_1404__p42304254162">spark.carbon.sqlastbuilder.classname</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="25.06%" headers="mcps1.3.8.1.1.2.4.1.2 "><p id="mrs_01_1404__p202361325101610">org.apache.spark.sql.hive.CarbonInternalSqlAstBuilder</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="56.64%" headers="mcps1.3.8.1.1.2.4.1.3 "><p id="mrs_01_1404__p10241152561615">AST constructor.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row83600234169"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.8.1.1.2.4.1.1 "><p id="mrs_01_1404__p142581625161612">spark.sql.catalog.class</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="25.06%" headers="mcps1.3.8.1.1.2.4.1.2 "><p id="mrs_01_1404__p9265325141613">org.apache.spark.sql.hive.HiveACLExternalCatalog</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="56.64%" headers="mcps1.3.8.1.1.2.4.1.3 "><p id="mrs_01_1404__p16271025131613">Hive External catalog to be used. This parameter is mandatory if Spark ACL is enabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row225512128162"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.8.1.1.2.4.1.1 "><p id="mrs_01_1404__p122971825141617">spark.sql.hive.implementation</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="25.06%" headers="mcps1.3.8.1.1.2.4.1.2 "><p id="mrs_01_1404__p15306425121612">org.apache.spark.sql.hive.HiveACLClientImpl</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="56.64%" headers="mcps1.3.8.1.1.2.4.1.3 "><p id="mrs_01_1404__p19312525141611">How to call the Hive client. This parameter is mandatory if Spark ACL is enabled.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1404__row01747971611"><td class="cellrowborder" valign="top" width="18.3%" headers="mcps1.3.8.1.1.2.4.1.1 "><p id="mrs_01_1404__p1732932581619">spark.sql.hiveClient.isolation.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="25.06%" headers="mcps1.3.8.1.1.2.4.1.2 "><p id="mrs_01_1404__p3334202510169">false</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="56.64%" headers="mcps1.3.8.1.1.2.4.1.3 "><p id="mrs_01_1404__p134122517162">This parameter is mandatory if Spark ACL is enabled.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1400.html">Using CarbonData (for MRS 3.x or Later)</a></div>
|
|
</div>
|
|
</div>
|
|
|