vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide/mrs_01_1017.html
Yang, Tong 6182f91ba8 MRS component operation guide_normal 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2022-12-09 14:55:21 +00:00

188 lines
27 KiB
HTML
Raw Blame History

<a name="mrs_01_1017"></a><a name="mrs_01_1017"></a>
<h1 class="topictitle1">Improving Real-time Data Write Performance</h1>
<div id="body1590128863843"><div class="section" id="mrs_01_1017__s99ea6e0b7c7f4b52aad90e7ffbb3c111"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1017__a12d1d8ad6ea7472093e34542bc9d415b">Scenarios where data needs to be written to HBase in real time, or large-scale and consecutive put scenarios</p>
<div class="note" id="mrs_01_1017__note3659121463219"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1017__p136601141326">This section applies to MRS 3.<em id="mrs_01_1017__i17349122116713">x</em> and later versions.</p>
</div></div>
</div>
<div class="section" id="mrs_01_1017__s3b36ebe1e0fe43df8b7144021cae1b92"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_1017__ac79b93f5df404c5690cb222aad7c0a40">The HBase put or delete interface can be used to save data to HBase.</p>
</div>
<div class="section" id="mrs_01_1017__s483f1b71973641ce88cbf9fcc7e875ef"><h4 class="sectiontitle">Procedure</h4><ul id="mrs_01_1017__u548b0ec870ea405b8976b25b15ffceb9"><li id="mrs_01_1017__l3eaf9c74871a49139fdd84b216b13666"><strong id="mrs_01_1017__b842311381016">Data writing server tuning</strong><p id="mrs_01_1017__a257a506742254f5dae7d77ba7ab5fce2">Parameter portal:</p>
<p id="mrs_01_1017__p21172338305">Go to the <strong id="mrs_01_1017__b10599425478">All Configurations</strong> page of the HBase service. For details, see <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.</p>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1017__t570590893991433b88f7d3238a28dbb3" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Configuration items that affect real-time data writing</caption><thead align="left"><tr id="mrs_01_1017__r69a7f3de1b3b47f6833daa788c3ce626"><th align="left" class="cellrowborder" valign="top" width="27.26%" id="mcps1.3.3.2.1.4.2.4.1.1"><p id="mrs_01_1017__en-us_topic_0116526934_p285027721170"><strong id="mrs_01_1017__b1323162904211">Parameter</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="48.6%" id="mcps1.3.3.2.1.4.2.4.1.2"><p id="mrs_01_1017__en-us_topic_0116526934_p418822571170"><strong id="mrs_01_1017__en-us_topic_0116526934_b413959981170">Description</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="24.14%" id="mcps1.3.3.2.1.4.2.4.1.3"><p id="mrs_01_1017__en-us_topic_0116526934_p647415761170"><strong id="mrs_01_1017__en-us_topic_0116526934_b458032791170">Default Value</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1017__r6167235094894418a7ae13debac6135d"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p18236151170"></p>
<p id="mrs_01_1017__p297894315534">hbase.wal.hsync</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p134951091170">Controls the synchronization degree when HLogs are written to the HDFS. If the value is <strong id="mrs_01_1017__b132901850184217">true</strong>, HDFS returns only when data is written to the disk. If the value is <strong id="mrs_01_1017__b1829114020435">false</strong>, HDFS returns when data is written to the OS cache.</p>
<p id="mrs_01_1017__en-us_topic_0116526934_p543471251170">Set the parameter to <strong id="mrs_01_1017__b56091820134316">false</strong> to improve write performance.</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p400410011170">true</p>
</td>
</tr>
<tr id="mrs_01_1017__r8a31d5e8818840e1b659074a9ceabb28"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__p13825537536">hbase.hfile.hsync</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p16224611170">Controls the synchronization degree when HFiles are written to the HDFS. If the value is <strong id="mrs_01_1017__b11157124624317">true</strong>, HDFS returns only when data is written to the disk. If the value is <strong id="mrs_01_1017__b216234654317">false</strong>, HDFS returns when data is written to the OS cache.</p>
<p id="mrs_01_1017__en-us_topic_0116526934_p146021541170">Set the parameter to <strong id="mrs_01_1017__b1535755116434">false</strong> to improve write performance.</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p419237921170">true</p>
</td>
</tr>
<tr id="mrs_01_1017__r794aeae6299e42a5bdfa10a82976d4d4"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p279116021170">GC_OPTS</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p462472931170">You can increase HBase memory to improve HBase performance because read and write operations are performed in HBase memory. <strong id="mrs_01_1017__b16831617184516">HeapSize</strong> and <strong id="mrs_01_1017__b19400132111458">NewSize</strong> need to be adjusted. When you adjust <strong id="mrs_01_1017__b492982864520">HeapSize</strong>, set <strong id="mrs_01_1017__b1592013854519">Xms</strong> and <strong id="mrs_01_1017__b187524410459">Xmx</strong> to the same value to avoid performance problems when JVM dynamically adjusts <strong id="mrs_01_1017__b839373714149">HeapSize</strong>. Set <strong id="mrs_01_1017__b4107205319456">NewSize</strong> to 1/8 of <strong id="mrs_01_1017__b713384631420">HeapSize</strong>.</p>
<ul id="mrs_01_1017__u18fcc6203c354292a8fc147c3674ec77"><li id="mrs_01_1017__l96e5cc6576414a6db9f08936e56bf589"><strong id="mrs_01_1017__b13858171217467">HMaster</strong>: If HBase clusters enlarge and the number of Regions grows, properly increase the <strong id="mrs_01_1017__b261711213460">GC_OPTS</strong> parameter value of the HMaster. </li><li id="mrs_01_1017__l257202a7c64b46d89ce2fec0576ad5f7"><strong id="mrs_01_1017__b113152309461">RegionServer</strong>: A RegionServer needs more memory than an HMaster. If sufficient memory is available, increase the <strong id="mrs_01_1017__b965610401469">HeapSize</strong> value.</li></ul>
<div class="note" id="mrs_01_1017__nd9e74efa89d44cbeb0567c596eb3c17d"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1017__en-us_topic_0116526934_p301917211735">When the value of <strong id="mrs_01_1017__b913984964619">HeapSize</strong> for the active HMaster is 4 GB, the HBase cluster can support 100,000 regions. Empirically, each time 35,000 regions are added to the cluster, the value of <strong id="mrs_01_1017__b2665136134719">HeapSize</strong> must be increased by 2 GB. It is recommended that the value of <strong id="mrs_01_1017__b151391911154716">HeapSize</strong> for the active HMaster not exceed 32 GB.</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><ul id="mrs_01_1017__ul19121124913710"><li id="mrs_01_1017__li112154915375">HMaster<p id="mrs_01_1017__p3915133412383"><a name="mrs_01_1017__li112154915375"></a><a name="li112154915375"></a>-server -Xms4G -Xmx4G -XX:NewSize=512M -XX:MaxNewSize=512M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=512M -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=65 -XX:+PrintGCDetails -Dsun.rmi.dgc.client.gcInterval=0x7FFFFFFFFFFFFFE -Dsun.rmi.dgc.server.gcInterval=0x7FFFFFFFFFFFFFE -XX:-OmitStackTraceInFastThrow -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M</p>
</li><li id="mrs_01_1017__li697792153915">Region Server<p id="mrs_01_1017__p164037811395"><a name="mrs_01_1017__li697792153915"></a><a name="li697792153915"></a>-server -Xms6G -Xmx6G -XX:NewSize=1024M -XX:MaxNewSize=1024M -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=512M -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=65 -XX:+PrintGCDetails -Dsun.rmi.dgc.client.gcInterval=0x7FFFFFFFFFFFFFE -Dsun.rmi.dgc.server.gcInterval=0x7FFFFFFFFFFFFFE -XX:-OmitStackTraceInFastThrow -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M</p>
</li></ul>
</td>
</tr>
<tr id="mrs_01_1017__r76cb162c7e4d4813b1e0670482eed8e0"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p85737481170">hbase.regionserver.handler.count</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p233849511170">Indicates the number of RPC server instances started on RegionServer. If the parameter is set to an excessively large value, threads will compete fiercely. If the parameter is set to an excessively small value, requests will be waiting for a long time in RegionServer, reducing the processing capability. You can add threads based on resources.</p>
<p id="mrs_01_1017__en-us_topic_0116526934_p91379751170">It is recommended that the value be set to <strong id="mrs_01_1017__b69315483215">100</strong> to <strong id="mrs_01_1017__b1614170321">300</strong> based on the CPU usage.</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p19785191170">200</p>
</td>
</tr>
<tr id="mrs_01_1017__r15979a3ee2da4dd19a8b7047eadf4561"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p330543901170">hbase.hregion.max.filesize</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p601599481170">Indicates the maximum size of an HStoreFile, in bytes. If the size of any HStoreFile exceeds the value of this parameter, the managed Hregion is divided into two parts.</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p411176261170">10737418240</p>
</td>
</tr>
<tr id="mrs_01_1017__r4055b409a3ce4fafa00c5b126cc118d2"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p441967291170">hbase.hregion.memstore.flush.size</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p231652591170">On the RegionServer, when the size of memstore that exists in memory of write operations exceeds <strong id="mrs_01_1017__b917219522287">memstore.flush.size</strong>, MemStoreFlusher performs the Flush operation to write the memstore to the corresponding store in the format of HFile. </p>
<p id="mrs_01_1017__en-us_topic_0116526934_p71607451170">If RegionServer memory is sufficient and active Regions are few, increase the parameter value and reduce compaction times to improve system performance. </p>
<p id="mrs_01_1017__en-us_topic_0116526934_p644467061170">The Flush operation may be delayed after it takes place. Write operations continue and memstore keeps increasing during the delay. The maximum size of memstore is: <strong id="mrs_01_1017__b565044994917">memstore.flush.size</strong> x <strong id="mrs_01_1017__b150865414494">hbase.hregion.memstore.block.multiplier</strong>. When the memstore size exceeds the maximum value, write operations are blocked. Properly increasing the value of <strong id="mrs_01_1017__b13331725165110">hbase.hregion.memstore.block.multiplier</strong> can reduce the blocks and make performance become more stable. Unit: byte</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p528007261170">134217728</p>
</td>
</tr>
<tr id="mrs_01_1017__rb9bd675884c74c7ea1abbf20c887d488"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p383505471170">hbase.regionserver.global.memstore.size</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p193866371170">Updates the size of all MemStores supported by the RegionServer before locking or forcible flush. On the RegionServer, the MemStoreFlusher thread performs the flush. The thread regularly checks memory occupied by write operations. When the total memory volume occupied by write operations exceeds the threshold, MemStoreFlusher performs the flush. Larger memstore will be flushed first and then smaller ones until the occupied memory is less than the threshold. </p>
<p id="mrs_01_1017__en-us_topic_0116526934_p402620121170">Threshold = hbase.regionserver.global.memstore.size x hbase.regionserver.global.memstore.size.lower.limit x HBase_HEAPSIZE</p>
<div class="note" id="mrs_01_1017__n7da7e948667245a1b4118764dfef416d"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1017__a85dbb82988d14d2c9f7fb4dd42972cc7">The sum of the parameter value and the value of <strong id="mrs_01_1017__b746071019521">hfile.block.cache.size</strong> cannot exceed 0.8, that is, memory occupied by read and write operations cannot exceed 80% of <strong id="mrs_01_1017__b10416426105217">HeapSize</strong>, ensuring stable running of other operations. </p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p185751161170">0.4</p>
</td>
</tr>
<tr id="mrs_01_1017__r725afaae45a44065884d32d4789f9629"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p523781061170">hbase.hstore.blockingStoreFiles</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p147681901170">Check whether the number of files is larger than the value of <strong id="mrs_01_1017__b107531515175313">hbase.hstore.blockingStoreFiles</strong> before you flush regions. </p>
<p id="mrs_01_1017__en-us_topic_0116526934_p658048531170">If it is larger than the value of <strong id="mrs_01_1017__b7539143312538">hbase.hstore.blockingStoreFiles</strong>, perform a compaction and configure <strong id="mrs_01_1017__b12164174725317">hbase.hstore.blockingWaitTime</strong> to 90s to make the flush delay for 90s. During the delay, write operations continue and the memstore size keeps increasing and exceeds the threshold (<strong id="mrs_01_1017__b18458116115419">memstore.flush.size</strong> x <strong id="mrs_01_1017__b4148812195419">hbase.hregion.memstore.block.multiplier</strong>), blocking write operations. After compaction is complete, a large number of writes may be generated. As a result, the performance fluctuates sharply.</p>
<p id="mrs_01_1017__en-us_topic_0116526934_p553727711170">Increase the value of <strong id="mrs_01_1017__b189381954205519">hbase.hstore.blockingStoreFiles</strong> to reduce block possibilities. </p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p560094891170">15</p>
</td>
</tr>
<tr id="mrs_01_1017__rb06a266fa3884fc99203f2273481fde1"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p287282951170">hbase.regionserver.thread.compaction.throttle</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p452905771170">The compression whose size is greater than the value of this parameter is executed by the large thread pool. The unit is bytes. Indicates a threshold of a total file size for compaction during a Minor Compaction. The total file size affects execution duration of a compaction. If the total file size is large, other compactions or flushes may be blocked. </p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p446581111170">1610612736</p>
</td>
</tr>
<tr id="mrs_01_1017__rb369ef00b4764fd6972b9885dfc73678"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p79642671170">hbase.hstore.compaction.min</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p411259081170">Indicates the minimum number of HStoreFiles on which minor compaction is performed each time. When the size of a file in a Store exceeds the value of this parameter, the file is compacted. You can increase the value of this parameter to reduce the number of times that the file is compacted. If there are too many files in the Store, read performance will be affected.</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p428642211170">6</p>
</td>
</tr>
<tr id="mrs_01_1017__r830e934609bd4f949ca228cd8e662415"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p423953611170">hbase.hstore.compaction.max</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__p17981221164015">Indicates the maximum number of HStoreFiles on which minor compaction is performed each time. The functions of the parameter and <strong id="mrs_01_1017__b7475152710386">hbase.hstore.compaction.max.size</strong> are similar. Both are used to limit the execution duration of one compaction.</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p568364481170">10</p>
</td>
</tr>
<tr id="mrs_01_1017__rb6543377f49a4a02b14fe1c432b8f6e0"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p276019611170">hbase.hstore.compaction.max.size</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p211663321170">If the size of an HFile is larger than the parameter value, the HFile will not be compacted in a Minor Compaction but can be compacted in a Major Compaction. </p>
<p id="mrs_01_1017__en-us_topic_0116526934_p562792681170">The parameter is used to prevent HFiles of large sizes from being compacted. After a Major Compaction is forbidden, multiple HFiles can exist in a Store and will not be merged into one HFile, without affecting data access performance. The unit is byte.</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p623268681170">9223372036854775807</p>
</td>
</tr>
<tr id="mrs_01_1017__rf0f6ffed1d7648c79f82226210a4ce23"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p35860271170">hbase.hregion.majorcompaction</p>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><p id="mrs_01_1017__en-us_topic_0116526934_p220327851170">Main compression interval of all HStoreFile files in a region. The unit is milliseconds. Execution of Major Compactions consumes much system resources and will affect system performance during peak hours. </p>
<p id="mrs_01_1017__en-us_topic_0116526934_p640773421170">If service updates, deletion, and reclamation of expired data space are infrequent, set the parameter to <strong id="mrs_01_1017__b4674518205919">0</strong> to disable Major Compactions.</p>
<p id="mrs_01_1017__en-us_topic_0116526934_p398251671170">If you must perform a Major Compaction to reclaim more space, increase the parameter value and configure the <strong id="mrs_01_1017__b2190154675912">hbase.offpeak.end.hour</strong> and <strong id="mrs_01_1017__b584215012597">hbase.offpeak.start.hour</strong> parameters to make the Major Compaction be triggered in off-peak hours.</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><p id="mrs_01_1017__en-us_topic_0116526934_p46131341170">604800000</p>
</td>
</tr>
<tr id="mrs_01_1017__r81c32fb8b7d249b7953e88ba6ad0ddaf"><td class="cellrowborder" valign="top" width="27.26%" headers="mcps1.3.3.2.1.4.2.4.1.1 "><ul id="mrs_01_1017__en-us_topic_0116526934_ul75321361170"><li id="mrs_01_1017__en-us_topic_0116526934_li6803661170">hbase.regionserver.maxlogs</li><li id="mrs_01_1017__en-us_topic_0116526934_li61232951170">hbase.regionserver.hlog.blocksize</li></ul>
</td>
<td class="cellrowborder" valign="top" width="48.6%" headers="mcps1.3.3.2.1.4.2.4.1.2 "><ul id="mrs_01_1017__u3111b965dc454636bef6d90a191d7d4d"><li id="mrs_01_1017__lb5b631b732244783bbde158a49bca88c">Indicates the threshold for the number of HLog files that are not flushed on a RegionServer. If the number of HLog files is greater than the threshold, the RegionServer forcibly performs flush operations.</li><li id="mrs_01_1017__l5816da75c50244e1abbe674827dd50f0">Indicates the maximum size of an HLog file. If the size of an HLog file is greater than the value of this parameter, a new HLog file is generated. The old HLog file is disabled and archived.</li></ul>
<p id="mrs_01_1017__en-us_topic_0116526934_p590460241170">The two parameters determine the number of HLogs that are not flushed in a RegionServer. When the data volume is less than the total size of memstore, the flush operation is forcibly triggered due to excessive HLog files. In this case, you can adjust the values of the two parameters to avoid forcible flush. Unit: byte</p>
</td>
<td class="cellrowborder" valign="top" width="24.14%" headers="mcps1.3.3.2.1.4.2.4.1.3 "><ul id="mrs_01_1017__u24b9bf8760f040fea8363c2d5a6b78e3"><li id="mrs_01_1017__la033ff159bbe423f96c264876b5175ac">32</li><li id="mrs_01_1017__le003183edcd640a49f5bd2d8b55a581c">134217728</li></ul>
</td>
</tr>
</tbody>
</table>
</div>
</li><li id="mrs_01_1017__l0bb96f6dd43d449ea38aff1a50fc727d"><strong id="mrs_01_1017__b719878201310">Data writing client tuning</strong><p id="mrs_01_1017__en-us_topic_0116526934_p47867571197">It is recommended that data is written in Put List mode if necessary, which greatly improves write performance. The length of each put list needs to be set based on the single put size and parameters of the actual environment. You are advised to do some basic tests before configuring parameters.</p>
</li><li id="mrs_01_1017__la948fb9ff9a640739dc4a45c6f92810d"><strong id="mrs_01_1017__b1915065119459">Data table writing design optimization</strong>
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1017__tc128a794bd0a4d54bfcebed794d1a3be" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameters affecting real-time data writing</caption><thead align="left"><tr id="mrs_01_1017__r110e574441ac44e0b01f563c686dc3de"><th align="left" class="cellrowborder" valign="top" width="18%" id="mcps1.3.3.2.3.2.2.4.1.1"><p id="mrs_01_1017__aa3d41ca1b8804aa5b400d39b806c5093"><strong id="mrs_01_1017__b42676817319">Parameter</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="56.00000000000001%" id="mcps1.3.3.2.3.2.2.4.1.2"><p id="mrs_01_1017__a89dde2e9823e4baca10d6466a6f82adc"><strong id="mrs_01_1017__a0b8604a993094a69b2159c5e4ef4746a">Description</strong></p>
</th>
<th align="left" class="cellrowborder" valign="top" width="26%" id="mcps1.3.3.2.3.2.2.4.1.3"><p id="mrs_01_1017__a362acc8315c348a787d513ca7284c1be"><strong id="mrs_01_1017__a349e413b871942c08220642d0b22973c">Default Value</strong></p>
</th>
</tr>
</thead>
<tbody><tr id="mrs_01_1017__r903c570752d5441da396d67f5d9fe4ad"><td class="cellrowborder" valign="top" width="18%" headers="mcps1.3.3.2.3.2.2.4.1.1 "><p id="mrs_01_1017__en-us_topic_0116526934_p13816711943">COMPRESSION</p>
</td>
<td class="cellrowborder" valign="top" width="56.00000000000001%" headers="mcps1.3.3.2.3.2.2.4.1.2 "><p id="mrs_01_1017__a52403acd6601416d9ed6061a50117e0f">The compression algorithm compresses blocks in HFiles. For compressible data, configure the compression algorithm to efficiently reduce disk I/Os and improve performance. </p>
<div class="note" id="mrs_01_1017__n888095f9a80b49fdbc2e59f4c703c98a"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_1017__a807bae449ff54902802b576a3ec9c7ba">Some data cannot be efficiently compressed. For example, a compressed figure can hardly be compressed again. The common compression algorithm is SNAPPY, because it has a high encoding/decoding speed and acceptable compression rate.</p>
</div></div>
</td>
<td class="cellrowborder" valign="top" width="26%" headers="mcps1.3.3.2.3.2.2.4.1.3 "><p id="mrs_01_1017__af21aff04f52b44b7b8add3e815f8af65">NONE</p>
</td>
</tr>
<tr id="mrs_01_1017__rea81cd68e0bc4637aab1106fab9b9288"><td class="cellrowborder" valign="top" width="18%" headers="mcps1.3.3.2.3.2.2.4.1.1 "><p id="mrs_01_1017__a0494f385b2324d39869d23b0270ef7e6">BLOCKSIZE</p>
</td>
<td class="cellrowborder" valign="top" width="56.00000000000001%" headers="mcps1.3.3.2.3.2.2.4.1.2 "><p id="mrs_01_1017__a9a4307f671be473bb62026cc6e25f789">Different block sizes affect HBase data read and write performance. You can configure sizes for blocks in an HFile. Larger blocks have a higher compression rate. However, they have poor performance in random data read, because HBase reads data in a unit of blocks. </p>
<p id="mrs_01_1017__a0a0980c7c393480aa2f279df7fc81ef4">Set the parameter to 128 KB or 256 KB to improve data write efficiency without greatly affecting random read performance. The unit is byte.</p>
</td>
<td class="cellrowborder" valign="top" width="26%" headers="mcps1.3.3.2.3.2.2.4.1.3 "><p id="mrs_01_1017__acdf3355587d94c15b5510ff9e63dd454">65536</p>
</td>
</tr>
<tr id="mrs_01_1017__rffbbd24f86cb41d6b818deb0e5a44f64"><td class="cellrowborder" valign="top" width="18%" headers="mcps1.3.3.2.3.2.2.4.1.1 "><p id="mrs_01_1017__a5c296050d5c74c29bb121f174671595b">IN_MEMORY</p>
</td>
<td class="cellrowborder" valign="top" width="56.00000000000001%" headers="mcps1.3.3.2.3.2.2.4.1.2 "><p id="mrs_01_1017__a75433de8ccf348bda8d3f7a30c9fd2c4">Whether to cache table data in the memory first, which improves data read performance. If you will frequently access some small tables, set the parameter. </p>
</td>
<td class="cellrowborder" valign="top" width="26%" headers="mcps1.3.3.2.3.2.2.4.1.3 "><p id="mrs_01_1017__a5efa96647c854006b1ae3b1b2869c19b">false</p>
</td>
</tr>
</tbody>
</table>
</div>
</li></ul>
</div>
</div>
<div>
<div class="familylinks">
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1013.html">HBase Performance Tuning</a></div>
</div>
</div>
View Git Blame Copy Permalink