forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
49 lines
6.3 KiB
HTML
49 lines
6.3 KiB
HTML
<a name="mrs_01_1670"></a><a name="mrs_01_1670"></a>
|
|
|
|
<h1 class="topictitle1">Configuring the NameNode Blacklist</h1>
|
|
<div id="body1595904094489"><div class="section" id="mrs_01_1670__s6dc176aeffe0429586033a843a87b4ff"><h4 class="sectiontitle">Scenario</h4><div class="note" id="mrs_01_1670__note17408121619136"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1670__p788202018132">This section applies to MRS 3.<em id="mrs_01_1670__i31521491551914">x</em> or later.</p>
|
|
</div></div>
|
|
<p id="mrs_01_1670__a44ceaaaed74c436e9ab789f38d520c9c">In the existing default DFSclient failover proxy provider, if a NameNode in a process is faulty, all HDFS client instances in the same process attempt to connect to the NameNode again. As a result, the application waits for a long time and timeout occurs.</p>
|
|
<p id="mrs_01_1670__ae0c903a4def34427ad2d1cdd88d41c36">When clients in the same JVM process connect to the NameNode that cannot be accessed, the system is overloaded. The NameNode blacklist is equipped with the MRS cluster to avoid this problem.</p>
|
|
<p id="mrs_01_1670__a3b63d79c3fe7499a9a5609b2735a9442">In the new Blacklisting DFSClient failover provider, the faulty NameNode is recorded in a list. The DFSClient then uses the information to prevent the client from connecting to such NameNodes again. This function is called NameNode blacklisting.</p>
|
|
<p id="mrs_01_1670__a28b60aeee9d3457bb79ae11bd1f178f7">For example, there is a cluster with the following configurations:</p>
|
|
<p id="mrs_01_1670__a6d5bb80d1175430e9c8f40e91ab2fad4">namenode: nn1, nn2</p>
|
|
<p id="mrs_01_1670__a2bc4728ae4ec4c04aff431b3ab2a3ae2">dfs.client.failover.connection.retries: 20</p>
|
|
<p id="mrs_01_1670__a239978a84eda486ea4032a0910fb3eb9">Processes in a single JVM: 10 clients</p>
|
|
<p id="mrs_01_1670__a19103610dc734cfbaed96a2f637836de">In the preceding cluster, if the active <strong id="mrs_01_1670__b133082847851914">nn1</strong> cannot be accessed, client1 will retry the connection for 20 times. Then, a failover occurs, and client1 will connect to <strong id="mrs_01_1670__b208930692551914">nn2</strong>. In the same way, other clients also connect to <strong id="mrs_01_1670__b207885016751914">nn2</strong> when the failover occurs after retrying the connection to <strong id="mrs_01_1670__b108677770251914">nn1</strong> for 20 times. Such process prolongs the fault recovery of NameNode.</p>
|
|
<p id="mrs_01_1670__aa622366b671c42ab9bf74cdfea2b3f13">In this case, the NameNode blacklisting adds <strong id="mrs_01_1670__b145100313451914">nn1</strong> to the blacklist when client1 attempts to connect to the active <strong id="mrs_01_1670__b82414218651914">nn1</strong> which is already faulty. Therefore, other clients will avoid trying to connect to <strong id="mrs_01_1670__b154084953151914">nn1</strong> but choose <strong id="mrs_01_1670__b72612214451914">nn2</strong> directly.</p>
|
|
<div class="note" id="mrs_01_1670__n1bb072514465476a89070ac3ac3967ac"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1670__aa1cfe6398f7347738444ff624c722789">If, at any time, all NameNodes are added to the blacklist, the content in the blacklist will be cleared, and the client attempts to connect to the NameNodes based on the initial NameNode list. If any fault occurs again, the NameNode is still added to the blacklist.</p>
|
|
</div></div>
|
|
<div class="fignone" id="mrs_01_1670__fe500771db76c4050878ae8c76d10f14f"><span class="figcap"><b>Figure 1 </b>NameNode blacklisting working principle</span><br><span><img id="mrs_01_1670__i7fca86409b144596b1c00bc142459d35" src="en-us_image_0000001296090668.jpg"></span></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1670__s9b3b566830ab41f38bda05ce68bbf9ac"><h4 class="sectiontitle">Configuration Description</h4><p id="mrs_01_1670__p39519895015">Go to the <strong id="mrs_01_1670__b40000339251914">All Configurations</strong> page of HDFS and enter a parameter name in the search box by referring to <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1670__t612869dbc5884d99b906970db3d622b1" frame="border" border="1" rules="all"><caption><b>Table 1 </b>NameNode blacklisting parameters</caption><thead align="left"><tr id="mrs_01_1670__reee29779c23d45c69312af6d5b76acc3"><th align="left" class="cellrowborder" valign="top" width="22.24%" id="mcps1.3.2.3.2.4.1.1"><p id="mrs_01_1670__a66cc961b1e1f47e8be948aab0ef9c2cb"><strong id="mrs_01_1670__a71da6668b2aa4b2b8a3f8f7dbbaf915c">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="59.85%" id="mcps1.3.2.3.2.4.1.2"><p id="mrs_01_1670__a4ddf7fc4f39d4a3e95d3125b14becb62"><strong id="mrs_01_1670__acdbd01395e3e47f2bde4d1a2ac242a92">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="17.91%" id="mcps1.3.2.3.2.4.1.3"><p id="mrs_01_1670__a71d128b827034ba882392ff46d2fd411"><strong id="mrs_01_1670__a25e267a6b6cd486ab7469a63cb2e0081">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1670__r3b6957ea2597406a85aab12626a25b59"><td class="cellrowborder" valign="top" width="22.24%" headers="mcps1.3.2.3.2.4.1.1 "><p id="mrs_01_1670__abe79098f12fe42a7982c1305dbe65e3d">dfs.client.failover.proxy.provider.<em id="mrs_01_1670__i88511626712">[nameservice ID]</em></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="59.85%" headers="mcps1.3.2.3.2.4.1.2 "><p id="mrs_01_1670__af5e96f274fd44eb2ba641fe1479d6539">Client Failover proxy provider class which creates the NameNode proxy using the authenticated protocol.</p>
|
|
<p id="mrs_01_1670__p17921734930">Set this parameter to <span class="parmvalue" id="mrs_01_1670__parmvalue115126475221"><b>org.apache.hadoop.hdfs.server.namenode.ha.BlackListingFailoverProxyProvider</b></span>.</p>
|
|
<p id="mrs_01_1670__p16896342319">You can configure the observer NameNode to process read requests.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="17.91%" headers="mcps1.3.2.3.2.4.1.3 "><p id="mrs_01_1670__p14762216515">org.apache.hadoop.hdfs.server.namenode.ha.AdaptiveFailoverProxyProvider</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0790.html">Using HDFS</a></div>
|
|
</div>
|
|
</div>
|
|
|