forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
43 lines
4.2 KiB
HTML
43 lines
4.2 KiB
HTML
<a name="mrs_01_0837"></a><a name="mrs_01_0837"></a>
|
|
|
|
<h1 class="topictitle1">Reducing Client Application Failure Rate</h1>
|
|
<div id="body1590130628529"><div class="section" id="mrs_01_0837__s66b2f50a4632433bbe418b7865a29965"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0837__a18549684f01a4a5ebed52b0eb9241db7">When the network is unstable or the cluster I/O and CPU are overloaded, client applications might encounter running failures.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0837__sabeb2217f5a7471fbacf31755e6995a0"><h4 class="sectiontitle">Configuration</h4><p id="mrs_01_0837__a2ec8d96d88f44bab9a2ade2b1ba42841">Adjust the following parameters in the <span class="filepath" id="mrs_01_0837__f18b17f50d15c4b46a3a9b86ad48401fd"><b>mapred-site.xml</b></span> configuration file on the client to reduce the client application failure rate:</p>
|
|
<div class="note" id="mrs_01_0837__note32052054818"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0837__p1201820164811">The <span class="filepath" id="mrs_01_0837__filepath13951112824814"><b>mapred-site.xml</b></span> configuration file is in the <strong id="mrs_01_0837__b25651120361">conf</strong> directory of the client installation path, for example, <strong id="mrs_01_0837__b1430791233817">/opt/</strong><strong id="mrs_01_0837__b16307191253818"></strong><strong id="mrs_01_0837__b1330715122381">client/Yarn/config</strong>.</p>
|
|
</div></div>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0837__t645b88266ef2437faae23a9418fa7459" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0837__r25a644642ad64b9793d6c9b4ba72b9b0"><th align="left" class="cellrowborder" valign="top" width="22.43%" id="mcps1.3.2.4.2.4.1.1"><p id="mrs_01_0837__a5c59a3e433374847af5a6fa1009017d1"><strong id="mrs_01_0837__b3755725203612">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="63.6%" id="mcps1.3.2.4.2.4.1.2"><p id="mrs_01_0837__aae24dbde20a745d0b38647ab2d9f2782"><strong id="mrs_01_0837__b8617192618360">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="13.969999999999999%" id="mcps1.3.2.4.2.4.1.3"><p id="mrs_01_0837__ae26a0c67f9c94db48a14640077a6a4ee"><strong id="mrs_01_0837__b74001127183616">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0837__r353f398acd12469286f66ea5be3335a2"><td class="cellrowborder" valign="top" width="22.43%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0837__a0fb83d975a3e482f9c16ee93f831757f">mapreduce.reduce.shuffle.max-host-failures</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.6%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0837__a39d0c42277344eea90a1770f388885ac">Indicates the number of allowed failures of an MR task to read remote shuffle data in the Reduce process. When the number is set to be over 5, the client application failure rate can be reduced. This parameter applies to MRS 3.<em id="mrs_01_0837__i441712841916">x</em>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="13.969999999999999%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0837__ac23e0fcf36f249cca6c2d2f7bd852f6a">5</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0837__r44242cbd987649e28efcdc156c06cbd5"><td class="cellrowborder" valign="top" width="22.43%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0837__aae795b0604ea472f9e3b5a4d09b1eb62">mapreduce.client.submit.file.replication</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.6%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0837__a6d6383fdf6da47529e4f4e04a1565e96">Indicates the backup of job files on HDFS. MR tasks are dependent on the job files during running. When the number of backups is set to be over 10, the client application failure rate can be reduced.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="13.969999999999999%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0837__ac4ecd8c8d4984c4b8d3485cb50975ff1">10</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0834.html">Using MapReduce</a></div>
|
|
</div>
|
|
</div>
|
|
|