forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
139 lines
26 KiB
HTML
139 lines
26 KiB
HTML
<a name="mrs_01_1667"></a><a name="mrs_01_1667"></a>
|
|
|
|
<h1 class="topictitle1">Balancing DataNode Capacity</h1>
|
|
<div id="body1595904094072"><div class="section" id="mrs_01_1667__sa2f81e4552564f5f83b6c20f3dff9870"><h4 class="sectiontitle">Scenario</h4><div class="note" id="mrs_01_1667__note17408121619136"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1667__p788202018132">This section applies to MRS 3.<em id="mrs_01_1667__i8151458105719">x</em> or later clusters.</p>
|
|
</div></div>
|
|
<p id="mrs_01_1667__a5d4c7cbff57444eba8f32f9d08bc1fdc">In the HDFS cluster, unbalanced disk usage among DataNodes may occur, for example, when new DataNodes are added to the cluster. Unbalanced disk usage may result in multiple problems. For example, MapReduce applications cannot make full use of local computing advantages, network bandwidth usage between data nodes cannot be optimal, or node disks cannot be used. Therefore, the system administrator needs to periodically check and maintain DataNode data balance.</p>
|
|
<p id="mrs_01_1667__a167b321d04804a81b544580394a1a55f">HDFS provides a capacity balancing program Balancer. By running Balancer, you can balance the HDFS cluster and ensure that the difference between the disk usage of each DataNode and that of the HDFS cluster does not exceed the threshold. DataNode disk usage before and after balancing is shown in <a href="#mrs_01_1667__ff269b77c9222460985503fffcebf980e">Figure 1</a> and <a href="#mrs_01_1667__fee19cefb9d104f238448abdcf62f1e49">Figure 2</a>, respectively.</p>
|
|
<div class="fignone" id="mrs_01_1667__ff269b77c9222460985503fffcebf980e"><a name="mrs_01_1667__ff269b77c9222460985503fffcebf980e"></a><a name="ff269b77c9222460985503fffcebf980e"></a><span class="figcap"><b>Figure 1 </b>DataNode disk usage before balancing</span><br><span><img id="mrs_01_1667__image51424194305" src="en-us_image_0000001349090497.png"></span></div>
|
|
<div class="fignone" id="mrs_01_1667__fee19cefb9d104f238448abdcf62f1e49"><a name="mrs_01_1667__fee19cefb9d104f238448abdcf62f1e49"></a><a name="fee19cefb9d104f238448abdcf62f1e49"></a><span class="figcap"><b>Figure 2 </b>DataNode disk usage after balancing</span><br><span><img id="mrs_01_1667__i783989f5ce50419f86ae9ce67b5ae857_1" src="en-us_image_0000001296250300.png"></span></div>
|
|
<p id="mrs_01_1667__ac7d0fe6d2a4544aaa3d34a2e1deb965a">The time of the balancing operation is affected by the following two factors:</p>
|
|
<ol id="mrs_01_1667__od82c203b162f4ba381f6ae01e2964db9"><li id="mrs_01_1667__l39e65497ea0f41cfbe80cb2198311b3b">Total amount of data to be migrated:<p id="mrs_01_1667__a26ac18b3e1334a85a0887c25224f5990"><a name="mrs_01_1667__l39e65497ea0f41cfbe80cb2198311b3b"></a><a name="l39e65497ea0f41cfbe80cb2198311b3b"></a>The data volume of each DataNode must be greater than (Average usage - Threshold) x Average data volume and less than (Average usage + Threshold) x Average data volume. If the actual data volume is less than the minimum value or greater than the maximum value, imbalance occurs. The system sets the largest deviation volume on all DataNodes as the total data volume to be migrated.</p>
|
|
</li><li id="mrs_01_1667__l1b23438ca89d48ce9739ba768f173a0d">Balancer migration is performed in sequence in iteration mode. The amount of data to be migrated in each iteration does not exceed 10 GB, and the usage of each iteration is recalculated.</li></ol>
|
|
<p id="mrs_01_1667__a884e4bb810aa44eb99e51fdf5f485ea2">Therefore, for a cluster, you can estimate the time consumed by each iteration (by observing the time consumed by each iteration recorded in balancer logs) and divide the total data volume by 10 GB to estimate the task execution time.</p>
|
|
<p id="mrs_01_1667__a37c21bd95b054b49968b0f56071698cd">The balancer can be started or stopped at any time.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1667__s59281af9a01e4bad95b47a2f23bb3c40"><h4 class="sectiontitle">Impact on the System</h4><ul id="mrs_01_1667__uf4b67eaebf574e6e92c0bf41d60ec628"><li id="mrs_01_1667__l514a10139c53408f975a58dea47055be">The balance operation occupies network bandwidth resources of DataNodes. Perform the operation during maintenance based on service requirements.</li><li id="mrs_01_1667__l8672f039479a4c6a9710eff7bb9eaacc">The balance operation may affect the running services if the bandwidth traffic (the default bandwidth control is 20 MB/s) is reset or the data volume is increased.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1667__scfe492c93ccd4716af19b200010bdde7"><h4 class="sectiontitle">Prerequisites</h4><p id="mrs_01_1667__aec3b42bc8bcc447db1c0c08fe61bb391">The client has been installed.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1667__s6b698d4d3cff4af0b91fa4b7e302e6b4"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_1667__oebc904276e2c4997b77b84c1242d879d"><li id="mrs_01_1667__l604fff1b2aee4177856f9b95bafc7f5f"><span>Log in to the node where the client is installed as a client installation user. Run the following command to switch to the client installation directory, for example, <span class="filepath" id="mrs_01_1667__fb7a42ecbecd846c580e24049fce29301"><b>/opt/client</b></span>:</span><p><p id="mrs_01_1667__aa75644f4f7364a87ab1b876bd741bc4e"><strong id="mrs_01_1667__ae4c163187f9d41cca68cf36343d60c19">cd /opt/client</strong></p>
|
|
<div class="note" id="mrs_01_1667__note68354112220"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1667__p108135442211">If the cluster is in normal mode, run the <strong id="mrs_01_1667__b14371588124132">su - omm</strong> command to switch to user <strong id="mrs_01_1667__b1372395494132">omm</strong>.</p>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_1667__l103797cf114d4a42acf3338f121febf7"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_1667__a63db1593d01e4848beaa6662e7bdafa6"><strong id="mrs_01_1667__ae9c81e31ba0247a2a11c74524665e7a1">source bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1667__l715e94e109b64cddb1454720cf5b355a"><span>If the cluster is in security mode, run the following command to authenticate the HDFS identity:</span><p><p id="mrs_01_1667__a8395c1de62e4448e91258bb04b634f08"><strong id="mrs_01_1667__a89d1854b858e470a8f038a9341fbfda5">kinit hdfs</strong></p>
|
|
</p></li><li id="mrs_01_1667__l7357e93e5ba24051bb5783f062d39801"><span>Determine whether to adjust the bandwidth control.</span><p><ul id="mrs_01_1667__u077e3eb4d0454a88928f0a02e1973bcc"><li id="mrs_01_1667__ld5403143115b445a8360c1eb3904bb5c">If yes, go to <a href="#mrs_01_1667__l91d088e58d8d4bbbb720317d843ca5d3">5</a>.</li><li id="mrs_01_1667__l86ba906ec32444d7a574ecc41f0a39b0">If no, go to <a href="#mrs_01_1667__ld8ed77b8b7c745308eea6a68de2f4233">6</a>.</li></ul>
|
|
</p></li><li id="mrs_01_1667__l91d088e58d8d4bbbb720317d843ca5d3"><a name="mrs_01_1667__l91d088e58d8d4bbbb720317d843ca5d3"></a><a name="l91d088e58d8d4bbbb720317d843ca5d3"></a><span>Run the following command to change the maximum bandwidth of Balancer, and then go to <a href="#mrs_01_1667__ld8ed77b8b7c745308eea6a68de2f4233">6</a>.</span><p><p id="mrs_01_1667__ada9f0a98b9d84968841b1b5e87b97057"><strong id="mrs_01_1667__ae589e9e7d3264e2aa401abcac631ae3e">hdfs dfsadmin -setBalancerBandwidth <em id="mrs_01_1667__a677f5a5140904011ab97019517195b7c"><bandwidth in bytes per second></em></strong></p>
|
|
<p id="mrs_01_1667__a9e8a211545044fa7a9f6b09ac8d2379e"><em id="mrs_01_1667__a1eb3ce5ce71c4aa19ff87af4083dc890"><bandwidth in bytes per second></em> indicates the bandwidth control value, in bytes. For example, to set the bandwidth control to 20 MB/s (the corresponding value is 20971520), run the following command:</p>
|
|
<p id="mrs_01_1667__afc1c0782fffa48d081ea6edae366cc07"><strong id="mrs_01_1667__ae41bb26104f949d08775f9e8f69de65a">hdfs dfsadmin -setBalancerBandwidth </strong><strong id="mrs_01_1667__a512d87b181a94dcaa4af1e79a4c81d51">20971520</strong></p>
|
|
<div class="note" id="mrs_01_1667__n2a3ce2ad75ee41b985fadc9d6b89960c"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1667__u0f8c05bf980447d59299525f925b4812"><li id="mrs_01_1667__l432355c553cf4a5b9eaa087e455b8174">The default bandwidth control is 20 MB/s. This value is applicable to the scenario where the current cluster uses the 10GE network and services are being executed. If the service idle time window is insufficient for balance maintenance, you can increase the value of this parameter to shorten the balance time, for example, to 209715200 (200 MB/s).</li><li id="mrs_01_1667__li171516116227">The value of this parameter depends on the networking. If the cluster load is high, you can change the value to 209715200 (200 MB/s). If the cluster is idle, you can change the value to 1073741824 (1 GB/s).</li><li id="mrs_01_1667__la711733d37f64b829efccd2fbc843884">If the bandwidth of the DataNodes cannot reach the specified maximum bandwidth, modify the HDFS parameter <span class="parmname" id="mrs_01_1667__peda6dcfca5034f2aa71cf98eee506f52"><b>dfs.datanode.balance.max.concurrent.moves</b></span> on FusionInsight Manager, and change the number of threads for balancing on each DataNode to <span class="parmvalue" id="mrs_01_1667__pbd9e4b16b59b4068aca42a9cc3af8db4"><b>32</b></span> and restart the HDFS service.</li></ul>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_1667__ld8ed77b8b7c745308eea6a68de2f4233"><a name="mrs_01_1667__ld8ed77b8b7c745308eea6a68de2f4233"></a><a name="ld8ed77b8b7c745308eea6a68de2f4233"></a><span>Run the following command to start the balance task:</span><p><p id="mrs_01_1667__abb5f90bee18244cfaa5e481d2cfecfe9"><strong id="mrs_01_1667__b878483519241">bash /opt/client/HDFS/hadoop/sbin/start-balancer.sh -threshold <em id="mrs_01_1667__a66574f833d744a888c77942328f81a82"><threshold of </em>balancer<em id="mrs_01_1667__abe09dc4d6c0043b3816c1701f8d9a199">></em></strong></p>
|
|
<p id="mrs_01_1667__a358685c69eea4da8a76b863f986147fc"><strong id="mrs_01_1667__a07616b00ccfd4783bd61516cc1d24a8b">-threshold</strong> specifies the deviation value of the DataNode disk usage, which is used for determining whether the HDFS data is balanced. When the difference between the disk usage of each DataNode and the average disk usage of the entire HDFS cluster is less than this threshold, the system considers that the HDFS cluster has been balanced and ends the balance task.</p>
|
|
<p id="mrs_01_1667__a0b12c4134988451e8ac15a0432dcc96f">For example, to set deviation rate to 5%, run the following command:</p>
|
|
<p id="mrs_01_1667__a1ce6fb4c0ffc45cd82d33d9342d3d227"><strong id="mrs_01_1667__b1051607819258">bash /opt/client/HDFS/hadoop/sbin/start-balancer.sh -threshold 5</strong></p>
|
|
<div class="note" id="mrs_01_1667__n281b88c507ed4956ab5555acca484ce9"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_1667__ua0f3aaedb2974df283f3b90fce566c3a"><li id="mrs_01_1667__li6583806485925">The preceding command executes the task in the background. You can query related logs in the <strong id="mrs_01_1667__b12772048224132">hadoop-root-balancer-</strong><em id="mrs_01_1667__i14907698484132">host name</em><strong id="mrs_01_1667__b12490388944132">.out log</strong> file in the <span class="filepath" id="mrs_01_1667__filepath5606458285934"><b>/opt/client/HDFS/hadoop/logs</b></span> directory of the host.</li><li id="mrs_01_1667__l43dd96b83334431baa9ad73057eb893a">To stop the balance task, run the following command:<p id="mrs_01_1667__a27deb7f90fb94fc78f81d33078075879"><a name="mrs_01_1667__l43dd96b83334431baa9ad73057eb893a"></a><a name="l43dd96b83334431baa9ad73057eb893a"></a><strong id="mrs_01_1667__abe220b02f4c24c958f0239138c7be899">bash /opt/client/HDFS/hadoop/sbin/stop-balancer.sh</strong></p>
|
|
</li><li id="mrs_01_1667__li11447990163820">If only data on some nodes needs to be balanced, you can add the <strong id="mrs_01_1667__b2669897354132">-include</strong> parameter in the script to specify the nodes to be migrated. You can run commands to view the usage of different parameters.</li><li id="mrs_01_1667__li15985114161125"><span class="filepath" id="mrs_01_1667__filepath317296312617"><b>/opt/client</b></span> is the client installation directory. If the directory is inconsistent, replace it.</li><li id="mrs_01_1667__li280619398507">If the command fails to be executed and the error information <strong id="mrs_01_1667__b5141407434132">Failed to APPEND_FILE /system/balancer.id</strong> is displayed in the log, run the following command to forcibly delete <span class="filepath" id="mrs_01_1667__filepath132821528135419"><b>/system/balancer.id</b></span> and run the <strong id="mrs_01_1667__b15280356264132">start-balancer.sh</strong> script again:<p id="mrs_01_1667__p2490112534"><strong id="mrs_01_1667__b1988133805315">hdfs dfs -rm -f /system/balancer.id</strong></p>
|
|
</li></ul>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_1667__ldd4809ba655343e28e13bc9a4618ddf1"><span>After you run the script in <a href="#mrs_01_1667__ld8ed77b8b7c745308eea6a68de2f4233">6</a>, the <strong id="mrs_01_1667__b268442914377">hadoop-root-balancer-</strong><em id="mrs_01_1667__i1068472953716">Host name</em><strong id="mrs_01_1667__b168413291370">.out log</strong> file is generated in <span class="filepath" id="mrs_01_1667__filepath13686129183714"><b>/opt/client/HDFS/hadoop/logs</b></span>, the client installation directory. You can view the following information in the log:</span><p><ul id="mrs_01_1667__ul189163972711"><li id="mrs_01_1667__li988133942712">Time Stamp</li><li id="mrs_01_1667__li1288113972715">Bytes Already Moved</li><li id="mrs_01_1667__li389939142718">Bytes Left To Move</li><li id="mrs_01_1667__li12894398272">Bytes Being Moved</li></ul>
|
|
<p id="mrs_01_1667__p48491839192717">If message "Balance took <em id="mrs_01_1667__i1562502943911">xxx</em> seconds" is displayed in the log, the balancing operation is complete.</p>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_1667__s2d29d02d51814b39b580eee7c3fe78b3"><h4 class="sectiontitle">Related Tasks</h4><p id="mrs_01_1667__ac15633109dfa4b61b714d2eb9ea8c341"><strong id="mrs_01_1667__acdec23e53bb240a6ab3898b4577be867">Enable automatic execution of the balance task</strong></p>
|
|
<ol id="mrs_01_1667__o168842524bfb4c4fb48529bde29d3f84"><li id="mrs_01_1667__lcdb3f57f26834caea616d82d55f30e31"><span>Log in to FusionInsight Manager.</span></li><li id="mrs_01_1667__lb158aec85d9146d2870ff64ff50c4592"><span>Choose <strong id="mrs_01_1667__b10264010504132">Cluster</strong> > <em id="mrs_01_1667__i19364104654132">Name of the desired cluster</em> > <strong id="mrs_01_1667__b1295757864132">Services</strong> > <strong id="mrs_01_1667__b13004857874132">HDFS</strong> > <strong id="mrs_01_1667__b4932590744132">Configurations</strong>, select <span class="parmvalue" id="mrs_01_1667__p20d768b3e7714469ab7a4e7cb273bdcd"><b>All Configurations</b></span>, search for the following parameters, and change the parameter values.</span><p><ul id="mrs_01_1667__ul107742414323"><li id="mrs_01_1667__li197753418322"><span class="parmname" id="mrs_01_1667__p24df41ed81a043d18cc5a35827c0b404"><b>dfs.balancer.auto.enable</b></span> indicates whether to enable automatic balance task execution. The default value <span class="parmvalue" id="mrs_01_1667__pf1890637947046168862e1041fb7fa0f"><b>false</b></span> indicates that automatic balance task execution is disabled. The value <span class="parmvalue" id="mrs_01_1667__p30463741733442ecaa7068faa10fea00"><b>true</b></span> indicates that automatic execution is enabled.</li><li id="mrs_01_1667__li577544112322"><span class="parmname" id="mrs_01_1667__p9fe20e27091a4bb299d19fc6e745899c"><b>dfs.balancer.auto.cron.expression</b></span> indicates the task execution time. The default value <span class="parmvalue" id="mrs_01_1667__p27a8881744924372826e4d8e4ed60920"><b>0 1 * * 6</b></span> indicates that the task is executed at 01:00 every Saturday. This parameter is valid only when the automatic execution is enabled.<p id="mrs_01_1667__ad7e288273d2b49f0b7be0434535bc590"><a href="#mrs_01_1667__t3d64fdb3254a42beaed3c5e4c7087501">Table 1</a> describes the expression for modifying this parameter. <span class="parmname" id="mrs_01_1667__pf465a6f5207c4ad89ef486a1ce649542"><b>*</b></span> indicates consecutive time segments.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_1667__t3d64fdb3254a42beaed3c5e4c7087501"></a><a name="t3d64fdb3254a42beaed3c5e4c7087501"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1667__t3d64fdb3254a42beaed3c5e4c7087501" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameters in the execution expression</caption><thead align="left"><tr id="mrs_01_1667__r133881eafc1f46aea5f6371929c4e22f"><th align="left" class="cellrowborder" valign="top" width="34.589999999999996%" id="mcps1.3.5.3.2.2.1.2.4.2.3.1.1"><p id="mrs_01_1667__a80d6d38da36e4d6e9a37b77b56f4592f">Column</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="65.41%" id="mcps1.3.5.3.2.2.1.2.4.2.3.1.2"><p id="mrs_01_1667__a595c7e88e6b84c1ab4864c43da325d33">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1667__rbfe9d24c7ab04e05a4a58327b71d9a87"><td class="cellrowborder" valign="top" width="34.589999999999996%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.1 "><p id="mrs_01_1667__a11ff9d30f1094432803a6ddbc085270c">1</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.41%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.2 "><p id="mrs_01_1667__af72fbcb4cb3b4bcd9f366322734d23d2">Minute. The value ranges from 0 to 59.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1667__r9eec332c36544d9688d4008d3424636e"><td class="cellrowborder" valign="top" width="34.589999999999996%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.1 "><p id="mrs_01_1667__a5872e9d867924bed8427b703ffd6bd3f">2</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.41%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.2 "><p id="mrs_01_1667__a4063e331ed894557b44340dd903244be">Hour. The value ranges from 0 to 23.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1667__r809f072959fd4f65a658c13c0bf68309"><td class="cellrowborder" valign="top" width="34.589999999999996%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.1 "><p id="mrs_01_1667__a0684b368e7cc485db316bcc28a6b845e">3</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.41%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.2 "><p id="mrs_01_1667__a46f0dcb32b604285800b08f1abd8990a">Date. The value ranges from 1 to 31.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1667__r5e092720220e4b149f8b4b536a95abf0"><td class="cellrowborder" valign="top" width="34.589999999999996%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.1 "><p id="mrs_01_1667__a84f503646dd24e75be438e93da940eb7">4</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.41%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.2 "><p id="mrs_01_1667__a6dc2ac377b1a4d60b8c24dc35b389038">Month. The value ranges from 1 to 12.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1667__rba6b8f515b6842ada34444938bd95b66"><td class="cellrowborder" valign="top" width="34.589999999999996%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.1 "><p id="mrs_01_1667__ad81d7ea9f70b453d94bcf78920508388">5</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="65.41%" headers="mcps1.3.5.3.2.2.1.2.4.2.3.1.2 "><p id="mrs_01_1667__aa416e2cd6ac54b6abea73c05a63c0768">Week. The value ranges from 0 to 6. <strong id="mrs_01_1667__b10116414404132">0</strong> indicates Sunday.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</li><li id="mrs_01_1667__li18775144116322"><span class="parmname" id="mrs_01_1667__parmname59219439145128"><b>dfs.balancer.auto.stop.cron.expression</b></span> indicates the task ending time. The default value is empty, indicating that the running balance task is not automatically stopped. For example, <strong id="mrs_01_1667__b4104727534132">0 5 * * 6</strong> indicates that the balance task is stopped at 05:00 every Saturday. This parameter is valid only when the automatic execution is enabled.<p id="mrs_01_1667__p32045251145128"><a href="#mrs_01_1667__t3d64fdb3254a42beaed3c5e4c7087501">Table 1</a> describes the expression for modifying this parameter. <span class="parmname" id="mrs_01_1667__parmname45528568145128"><b>*</b></span> indicates consecutive time segments.</p>
|
|
</li></ul>
|
|
</p></li><li id="mrs_01_1667__l4d69ef5d8cd641a7b5ef72d3e52706bd"><span>Running parameters of the balance task that is automatically executed are shown in <a href="#mrs_01_1667__tc3bff391b0c14479916d9097f5e28238">Table 2</a>.</span><p>
|
|
<div class="tablenoborder"><a name="mrs_01_1667__tc3bff391b0c14479916d9097f5e28238"></a><a name="tc3bff391b0c14479916d9097f5e28238"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1667__tc3bff391b0c14479916d9097f5e28238" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Running parameters of the automatic balancer</caption><thead align="left"><tr id="mrs_01_1667__r1f85acce9189499d8194b497737f48dd"><th align="left" class="cellrowborder" valign="top" width="20.74%" id="mcps1.3.5.3.3.2.1.2.4.1.1"><p id="mrs_01_1667__ae1aeb98710e6446284dedef509b0eca3">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="45.93%" id="mcps1.3.5.3.3.2.1.2.4.1.2"><p id="mrs_01_1667__a7d18848d35b1439cae087f08bc488195">Parameter description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33%" id="mcps1.3.5.3.3.2.1.2.4.1.3"><p id="mrs_01_1667__a1a3d556b3d8447a291ac8a7d12b55123">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1667__r67feba2786d547f998bb0749c88dbf0a"><td class="cellrowborder" valign="top" width="20.74%" headers="mcps1.3.5.3.3.2.1.2.4.1.1 "><p id="mrs_01_1667__a1f8497a087ba41ae9308a676e20a88b9">dfs.balancer.auto.threshold</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="45.93%" headers="mcps1.3.5.3.3.2.1.2.4.1.2 "><p id="mrs_01_1667__a41425b1aa4624a4cbbbe635152245a65">Specifies the balancing threshold of the disk capacity percentage. This parameter is valid only when <strong id="mrs_01_1667__b186416111214">dfs.balancer.auto.enable</strong> is set to <strong id="mrs_01_1667__b14642013128">true</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.5.3.3.2.1.2.4.1.3 "><p id="mrs_01_1667__afdf425cc0c574a7a8c39e27515266fc8">10</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1667__rb278d63130bf42b0ab45e2d015ec98d7"><td class="cellrowborder" valign="top" width="20.74%" headers="mcps1.3.5.3.3.2.1.2.4.1.1 "><p id="mrs_01_1667__a3771c3a90b39427fbc2f501246f6393c">dfs.balancer.auto.exclude.datanodes</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="45.93%" headers="mcps1.3.5.3.3.2.1.2.4.1.2 "><p id="mrs_01_1667__aacfe871677644f46884b1d677fbb7890">Specifies the list of DataNodes on which automatic disk balancing is not required. This parameter is valid only when <strong id="mrs_01_1667__b4661362164132">dfs.balancer.auto.enable</strong> is set to <strong id="mrs_01_1667__b1046017064132">true</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.5.3.3.2.1.2.4.1.3 "><p id="mrs_01_1667__a541f5d0d9926458c907dfa71d4ef5ed2">The value is left blank by default.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1667__row16881153672413"><td class="cellrowborder" valign="top" width="20.74%" headers="mcps1.3.5.3.3.2.1.2.4.1.1 "><p id="mrs_01_1667__p16882113632412">dfs.balancer.auto.bandwidthPerSec</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="45.93%" headers="mcps1.3.5.3.3.2.1.2.4.1.2 "><p id="mrs_01_1667__p6882113642410">Specifies the maximum bandwidth (MB/s) of each DataNode for load balancing.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.5.3.3.2.1.2.4.1.3 "><p id="mrs_01_1667__p5882236142420">20</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1667__row14427114052417"><td class="cellrowborder" valign="top" width="20.74%" headers="mcps1.3.5.3.3.2.1.2.4.1.1 "><p id="mrs_01_1667__p24271240192412">dfs.balancer.auto.maxIdleIterations</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="45.93%" headers="mcps1.3.5.3.3.2.1.2.4.1.2 "><p id="mrs_01_1667__p8427194018243">Specifies the maximum number of consecutive idle iterations of Balancer. An idle iteration is an iteration without moving blocks. When the number of consecutive idle iterations reaches the maximum number, the balance task ends. The value <strong id="mrs_01_1667__b19494195874132">-1</strong> indicates infinity.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.5.3.3.2.1.2.4.1.3 "><p id="mrs_01_1667__p184271040132416">5</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1667__row1523733418470"><td class="cellrowborder" valign="top" width="20.74%" headers="mcps1.3.5.3.3.2.1.2.4.1.1 "><p id="mrs_01_1667__p18237134144715">dfs.balancer.auto.maxDataNodesNum</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="45.93%" headers="mcps1.3.5.3.3.2.1.2.4.1.2 "><p id="mrs_01_1667__p1057795312479">Controls the number of DataNodes that perform automatic balance tasks. Assume that the value of this parameter is <em id="mrs_01_1667__i725827053">N</em>. If <em id="mrs_01_1667__i1543711311657">N</em> is greater than 0, data is balanced between <em id="mrs_01_1667__i1831836557">N</em> DataNodes with the highest percentage of remaining space and <em id="mrs_01_1667__i201711401854">N</em> DataNodes with the lowest percentage of remaining space. If <em id="mrs_01_1667__i8650643952">N</em> is 0, data is balanced among all DataNodes in the cluster.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33%" headers="mcps1.3.5.3.3.2.1.2.4.1.3 "><p id="mrs_01_1667__p1823753474719">5</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</p></li><li id="mrs_01_1667__l4378a2e69b784fc1a5504d74d34a6dfd"><span>Click <span class="uicontrol" id="mrs_01_1667__uef233ed4273041cb89e87a11000681ad"><b>Save</b></span> to make configurations take effect. You do not need to restart the HDFS service.</span><p><p id="mrs_01_1667__a1b1ebe1a6dbf4b1aadc6db75a2a6f46a">Go to the <strong id="mrs_01_1667__b12145285374132">/var/log/Bigdata/hdfs/nn/hadoop-omm-balancer-</strong><em id="mrs_01_1667__i776770124132">Host name</em><strong id="mrs_01_1667__b18048468064132">.log</strong> file to view the task execution logs saved in the active NameNode.</p>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0790.html">Using HDFS</a></div>
|
|
</div>
|
|
</div>
|
|
|