forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
38 lines
3.9 KiB
HTML
38 lines
3.9 KiB
HTML
<a name="mrs_01_0850"></a><a name="mrs_01_0850"></a>
|
|
|
|
<h1 class="topictitle1">Optimizing Performance for Committing MR Jobs</h1>
|
|
<div id="body1590130630218"><div class="section" id="mrs_01_0850__s88178fadea7f4f75a47f1ce875f1f9d6"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0850__a484f19f724474d0c8e45574b3eb4c92d">By default, if an MR job generates a large number of output files, it takes a long time for the job to commit the temporary outputs of a task to the final output directory in the commit phase. In large clusters, the time-consuming commit process of jobs greatly affects the performance.</p>
|
|
<p id="mrs_01_0850__a4cd84e668b42474cb692496a51ca5255">In this case, you can set the <span class="parmname" id="mrs_01_0850__pcebce47d233549028b06295db7968603"><b>mapreduce.fileoutputcommitter.algorithm.version</b></span> to <span class="parmvalue" id="mrs_01_0850__p7308dfc9cc3e4b74822ead4a120f97bf"><b>2</b></span> to improve the performance in the commit phase of MR jobs.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0850__s6ce45ca588b9483e9becf3cdba72b88c"><h4 class="sectiontitle">Procedure</h4><p id="mrs_01_0850__a7da0596c2f0b4a85a8ab64412f78cfa9">Navigation path for setting parameters:</p>
|
|
<p id="mrs_01_0850__p139709455225">On the <strong id="mrs_01_0850__b1437019254416">All Configurations</strong> page of the Yarn service, enter a parameter name in the search box. For details, see <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0850__td3c8ab11f2a44ebe83537e6632f53bb2" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0850__r3795b61fe8f548769634e0eaa89c194f"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.4.2.4.1.1"><p id="mrs_01_0850__a6190b38f360d45bf8edeb8023e834de2"><strong id="mrs_01_0850__b466519214263">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.4.2.4.1.2"><p id="mrs_01_0850__a8f547a2d56064272a7380b04d0423402"><strong id="mrs_01_0850__b14482103102620">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.2.4.2.4.1.3"><p id="mrs_01_0850__ac1bfb178b67e46adafdf130b8db0e601"><strong id="mrs_01_0850__b10309341267">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0850__r1f66bce2593b4e439c898d3dbfed7513"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_0850__a12efb75ca5a34b4fa18dbd293b316d85">mapreduce.fileoutputcommitter.algorithm.version</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_0850__a32910e360359497e966b4650ceb2de4a">Indicates the algorithm version submitted by a job. The value is <span class="parmvalue" id="mrs_01_0850__p3e14b92f9101481da91ba7e645fd767f"><b>1</b></span> or <span class="parmvalue" id="mrs_01_0850__pf16ea86adeb44bd7b95e4b8eca8f0d27"><b>2</b></span>.</p>
|
|
<div class="note" id="mrs_01_0850__n0f648c9ea0314af4b7bb16779050f436"><span class="notetitle"> NOTE: </span><div class="notebody"><p id="mrs_01_0850__af21c323fb3804f1b8eeecfc3a92f1644"><strong id="mrs_01_0850__b8877165042617">2</strong> is the recommended algorithm version. This algorithm enables tasks to directly commit the output results of each task to the final result output directory, reducing the time for the results of large jobs are committed.</p>
|
|
</div></div>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_0850__aa577c2d39016412880cc49f05582e1ba">2</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0843.html">MapReduce Performance Tuning</a></div>
|
|
</div>
|
|
</div>
|
|
|