forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
37 lines
3.7 KiB
HTML
37 lines
3.7 KiB
HTML
<a name="mrs_01_1673"></a><a name="mrs_01_1673"></a>
|
|
|
|
<h1 class="topictitle1">Optimizing HDFS DataNode RPC QoS</h1>
|
|
<div id="body1595904095018"><div class="section" id="mrs_01_1673__scf3d57eb042a476eb4b45bd76e9109a5"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1673__a33bf5e65d2b2451f8844c9f9a1778ae6">When the speed at which the client writes data to the HDFS is greater than the disk bandwidth of the DataNode, the disk bandwidth is fully occupied. As a result, the DataNode does not respond. The client can back off only by canceling or restoring the channel, which results in write failures and unnecessary channel recovery operations.</p>
|
|
<div class="note" id="mrs_01_1673__note17408121619136"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1673__p788202018132">This section applies to MRS 3.<em id="mrs_01_1673__i204366736752853">x</em> or later.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1673__sdc5790c107d44805be290d9a4313d9f0"><h4 class="sectiontitle">Configuration</h4><p id="mrs_01_1673__ab7e001413e2e464a8de900c02bb39bd2">The new configuration parameter <span class="parmname" id="mrs_01_1673__p9568055d849b4b418ff34c7083988f7c"><b>dfs.pipeline.ecn</b></span> is introduced. When this configuration is enabled, the DataNode sends a signal from the write channel when the write channel is overloaded. The client may perform backoff based on the blocking signal to prevent the system from being overloaded. This configuration parameter is introduced to make the channel more stable and reduce unnecessary cancellation or recovery operations. After receiving the signal, the client backs off for a period of time (5,000 ms), and then adjusts the backoff time based on the related filter (the maximum backoff time is 50,000 ms).</p>
|
|
<p id="mrs_01_1673__p39519895015">Go to the <strong id="mrs_01_1673__b139895610267">All Configurations</strong> page of HDFS and enter a parameter name in the search box by referring to <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1673__t2f07c6e2b2004abbb948859ebcdc4bb6" frame="border" border="1" rules="all"><caption><b>Table 1 </b>DN ECN configuration</caption><thead align="left"><tr id="mrs_01_1673__r16449c2b14b7455e9f175612a5911c75"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.4.2.4.1.1"><p id="mrs_01_1673__a1616045700694ff0a305406a4c9d6b74">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.4.2.4.1.2"><p id="mrs_01_1673__a4d940e072fe14bca816a21764e5a9a4c">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.4.2.4.1.3"><p id="mrs_01_1673__a593aec9220554639a3ae8c1778a97123">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1673__r81a11e5a6ed544c88a4d549e34647315"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_1673__ae27775d476fc4cdfaa48585f46b08460">dfs.pipeline.ecn</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_1673__a2b5044a842394b02b599a5aec54bded6">After configuration, the DataNode can send blocking notifications to the client.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_1673__a6bac9e0be3e54ff8b962654012d8228d">false</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0790.html">Using HDFS</a></div>
|
|
</div>
|
|
</div>
|
|
|