forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
74 lines
10 KiB
HTML
74 lines
10 KiB
HTML
<a name="mrs_01_0859"></a><a name="mrs_01_0859"></a>
|
|
|
|
<h1 class="topictitle1">Using CGroups with YARN</h1>
|
|
<div id="body1590130746696"><p id="mrs_01_0859__p19473955182218">This section applies to MRS 3.<em id="mrs_01_0859__i86019105114">x</em> or later clusters.</p>
|
|
<div class="section" id="mrs_01_0859__saadb2b27ba104d8bb9dbb29bec4911d0"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_0859__a124c5fd80ace4ff28ed303c6c1739274">CGroups is a Linux kernel feature. In YARN this feature allows containers to be limited in their resource usage (example, CPU usage). Without CGroups, it is hard to limit the container CPU usage. Without CGroups, it is hard to limit the container CPU usage.</p>
|
|
<div class="note" id="mrs_01_0859__nd7c2254c94c244d5b565cc0503c7afb2"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_0859__a8adc9ee8a1c44a2db1338b7c1510a6a9">Currently, CGroups is only used for limiting the CPU usage.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_0859__sdb8fa94b74284e0e9ece0036c04e742a"><h4 class="sectiontitle">Configuration Description</h4><p id="mrs_01_0859__a86a69e00bced4eea8a499d09f61e3168">CGroups is a Linux kernel feature and is enabled by using LinuxContainerExecutor. For details about how to configure the LinuxContainerExecutor for security, see the official website. You can learn the file system permissions assigned for users and user groups from documentation published on the official website.</p>
|
|
<div class="note" id="mrs_01_0859__n2ff301597a5e4cea9f00d7cfacedb576"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0859__u5b3229c5ca2d4188b6dc73ac8a2e9d02"><li id="mrs_01_0859__l89f90106720b4e9b9450260ac314ce79">Do not modify users, user groups, and related permissions of various paths in the corresponding file system. Otherwise, functions of CGroups may become abnormal.</li><li id="mrs_01_0859__l0fb4a1d17cea44d68429d5e1a3e61e51">If the parameter value of <span class="parmname" id="mrs_01_0859__p5730396054bf4f3096286b2ab5b07ad3"><b>yarn.nodemanager.resource.percentage-physical-cpu-limit</b></span> is too small, the number of available cores may be less than one. For example, if the parameter of a four-core node is set to 20%, the number available core is less than one. As a result, all cores will be used. The Quota mode can be used in Linux versions, for example, Cent OS, that do not support Quota mode.</li></ul>
|
|
</div></div>
|
|
<p id="mrs_01_0859__a0ed4442ddf79469bb093942d75e2e401">The table below describes the parameter for configuring cpuset mode, that is, only configured CPUs can be used by YARN.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0859__t97b2bd7c74a543f8a43145ddc28e1af0"></a><a name="t97b2bd7c74a543f8a43145ddc28e1af0"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0859__t97b2bd7c74a543f8a43145ddc28e1af0" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0859__rf9bd600f0e4a4af3aba6abfd46933970"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.5.2.4.1.1"><p id="mrs_01_0859__a6b309f1b8b0a46f09abf4e9febbf4fb3">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.5.2.4.1.2"><p id="mrs_01_0859__a1416c4ad539048abb779b2d43549f0cf">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.5.2.4.1.3"><p id="mrs_01_0859__a03ac77592a1e429db41a257a749b8078">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0859__rf07def146587424ab9d553386a5a6ec7"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.4.1.1 "><p id="mrs_01_0859__a272e1601406f4e8a9636b687915982e1">yarn.nodemanager.linux-container-executor.cgroups.cpu-set-usage</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.4.1.2 "><p id="mrs_01_0859__ab53d0df6b0074b28bcfc99b559c1bdcc">Whether to enable the cpuset mode. If this parameter is set to <span class="parmvalue" id="mrs_01_0859__pa1f98c9888da44f18b680f6ac8b3fd79"><b>true</b></span>, the cpuset mode is enabled.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.5.2.4.1.3 "><p id="mrs_01_0859__a954b1038e7e94c2996571bfb492d7bd2">false</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_0859__a16b781a7ebb546d0aac26bccf26993e8">The table below describes the parameters for configuring the strictcpuset mode, that is, only configured CPUs can be used by containers.</p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0859__td100d24aaa184229963c9bf6323c0ea5"></a><a name="td100d24aaa184229963c9bf6323c0ea5"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0859__td100d24aaa184229963c9bf6323c0ea5" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_0859__rf5c86e20d4e54af7af94feb5167c054a"><th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.7.2.4.1.1"><p id="mrs_01_0859__ab081d4c0351f402884f31ed4f0cd8cfb">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.7.2.4.1.2"><p id="mrs_01_0859__abc86ab54e2e94d6f8d67e0a174868e7b">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.33333333333333%" id="mcps1.3.3.7.2.4.1.3"><p id="mrs_01_0859__ad35b8737b9d54d8ebfb2fa5bb5b70034">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0859__r6bdaedbf0e784f7a9eae02813385a453"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.7.2.4.1.1 "><p id="mrs_01_0859__a80b165b9225c4b75b1fa34373fa3e190">yarn.nodemanager.linux-container-executor.cgroups.cpu-set-usage</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.7.2.4.1.2 "><p id="mrs_01_0859__a1e26c87dff5c4676ba7c473f8363bc7c">Whether to enable the cpuset mode. If this parameter is set to <span class="parmvalue" id="mrs_01_0859__p42b023a1a2954458bff5c99747ceb2d0"><b>true</b></span>, the cpuset mode is enabled.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.7.2.4.1.3 "><p id="mrs_01_0859__a0e107dd438c0477fb36fe454f0351120">false</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0859__r76cc2057f5dc49a7b0ac348f46b0f445"><td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.7.2.4.1.1 "><p id="mrs_01_0859__a64f70fc9fdac4649adb3648b9b0bdfb5">yarn.nodemanager.linux-container-executor.cgroups.cpuset.strict.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.7.2.4.1.2 "><p id="mrs_01_0859__a7ae28064f5f0400ab3ff617d4e36973b">Whether containers use allocated CPUs. If this parameter is set to <strong id="mrs_01_0859__b189117218541422">true</strong>, the container can use the allocated CPUs.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.33333333333333%" headers="mcps1.3.3.7.2.4.1.3 "><p id="mrs_01_0859__aa08b8481dc4e4a81ad86b400b042fc76">false</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_0859__aecc05901a09b41468cab3c72ff232299">To switch from cpuset mode to quota mode, the following conditions must be met:</p>
|
|
<ul id="mrs_01_0859__ubea0f177db1c4b28b931de1892dd476f"><li id="mrs_01_0859__l2e75a342904d4fbeacf9fa393a9ff6b9">Set the <span class="parmname" id="mrs_01_0859__p55eb0a32a70142bfb65b16d882f33c1a"><b>yarn.nodemanager.linux-container-executor.cgroups.cpu-set-usage</b></span> parameter to <span class="parmvalue" id="mrs_01_0859__p51224d5c89cd4ae4b6686c91b84949e9"><b>false</b></span>.</li><li id="mrs_01_0859__lf1e118f1b18041aaa7c7b133397e3802">Delete container folders if exists.</li><li id="mrs_01_0859__en-us_topic_0039590419_li44155318457">Delete all the CUPs configured in the <strong id="mrs_01_0859__b144117212241422">cpuset.cpus</strong> file.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_0859__section14379950172815"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_0859__ol151739463396"><li id="mrs_01_0859__li617134614398"><span>Log in to Manager. Choose <strong id="mrs_01_0859__b5835195812249">Cluster</strong> > <em id="mrs_01_0859__i883610587245">Name of the desired cluster</em> > <strong id="mrs_01_0859__b383610584248">Services</strong> > <strong id="mrs_01_0859__b1837458182411">Yarn</strong> > <strong id="mrs_01_0859__b88371458192411">Configurations</strong> and select <span class="parmvalue" id="mrs_01_0859__parmvalue583713587243"><b>All Configurations</b></span>.</span></li><li id="mrs_01_0859__li20963215879"><span>In the navigation pane on the left, choose <strong id="mrs_01_0859__b138891132918">NodeManager </strong>> <strong id="mrs_01_0859__b1967643182917">Customization </strong>and find the <strong id="mrs_01_0859__b4918253291">yarn-site.xml</strong> file.</span></li><li id="mrs_01_0859__li5359711175611"><span>Add the parameters in <a href="#mrs_01_0859__t97b2bd7c74a543f8a43145ddc28e1af0">Table 1</a> and <a href="#mrs_01_0859__td100d24aaa184229963c9bf6323c0ea5">Table 2</a> as user-defined parameters.</span><p><p id="mrs_01_0859__p387981613566">Based on the configuration files and parameter functions, locate the row where parameter <strong id="mrs_01_0859__b1384711718344">yarn-site.xml</strong> resides. Enter the parameter name in the <strong id="mrs_01_0859__b1299441753316">Name</strong> column and enter the parameter value in the <strong id="mrs_01_0859__b1799412177335">Value</strong> column.</p>
|
|
<p id="mrs_01_0859__en-us_topic_0046736995_p44619987">Click <span class="uicontrol" id="mrs_01_0859__uicontrol2415133373010"><b>+</b></span> to add a customized parameter.</p>
|
|
</p></li><li id="mrs_01_0859__en-us_topic_0046736995_li47280701"><span>Click <strong id="mrs_01_0859__b70509969441422">Save</strong>. In the displayed <strong id="mrs_01_0859__b193643191441422">Save Configuration</strong> dialog box, confirm the modification and click <strong id="mrs_01_0859__b36745150641422">OK</strong>. Click <strong id="mrs_01_0859__b15664175814219">Finish</strong> when the system displays "Operation succeeded". The configuration is successfully saved.</span><p><p id="mrs_01_0859__p63911249124619">After the configuration is saved, restart the Yarn service whose configuration has expired for the configuration to take effect.</p>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0851.html">Using Yarn</a></div>
|
|
</div>
|
|
</div>
|
|
|