forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
28 lines
4.9 KiB
HTML
28 lines
4.9 KiB
HTML
<a name="mrs_01_0801"></a><a name="mrs_01_0801"></a>
|
|
|
|
<h1 class="topictitle1">Configuring ulimit for HBase and HDFS</h1>
|
|
<div id="body1590130533104"><div class="section" id="mrs_01_0801__s74264ff283cd4c788a81b59703b879b5"><h4 class="sectiontitle">Symptom</h4><p id="mrs_01_0801__af0cebea866e741ae97ae3c03723d1dee">When you open an HDFS file, an error occurs due to the limit on the number of file handles. Information similar to the following is displayed.</p>
|
|
<pre class="screen" id="mrs_01_0801__s6521cae9a035404f9ebf2d40fa0f73e2">IOException (Too many open files)</pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_0801__sf6092be34e7c4fd0a0dda0ef1fafa6b1"><h4 class="sectiontitle">Procedure</h4><p id="mrs_01_0801__ae221cbba40ee4d7eafc49cc7c35c0c64">You can contact the <span id="mrs_01_0801__ph179218753813">system </span>administrator to add file handles for each user. This is a configuration on the OS instead of HBase or HDFS. It is recommended that the <span id="mrs_01_0801__ph134281414143816">system </span>administrator configure the number of file handles based on the service traffic of HBase and HDFS and the rights of each user. If a user performs a large number of operations frequently on the HDFS that has large service traffic, set the number of file handles of this user to a large value.</p>
|
|
<ol id="mrs_01_0801__ofc08ed481bb14eb4be168a2609016d48"><li id="mrs_01_0801__lbfffda03c4e24d0e9c0884a82aaaeb78"><span>Log in to the OSs of all nodes or clients in the cluster as user <strong id="mrs_01_0801__a34d3ac2ba558477cbaed4b04af821b00">root</strong>, and go to the <strong id="mrs_01_0801__ac74931459ed54dbba053f3d2b29c4a88">/etc/security</strong> directory.</span></li><li id="mrs_01_0801__l3b265649cc0e49a8bbef63a057638b77"><span>Run the following command to edit the <strong id="mrs_01_0801__a9cae7c32b08840828d62dfcda9efb2cf">limits.conf</strong> file:</span><p><p id="mrs_01_0801__a8d3fda1b39af4e978cbb462b22757a73"><strong id="mrs_01_0801__a771c3e7cb5f74b568846520993171d10">vi limits.conf</strong></p>
|
|
<p id="mrs_01_0801__a720e186eae184c7c8bfe20ad171b60ab">Add the following information to the file.</p>
|
|
<pre class="screen" id="mrs_01_0801__s4cd9eede626f433cbc435838aa7d1e83">hdfs - nofile 32768
|
|
hbase - nofile 32768</pre>
|
|
<p id="mrs_01_0801__a75766ba52ec0458e99f1f857f4b1a73c"><strong id="mrs_01_0801__a37fd47986e80429082a42ff859dd8dc6">hdfs</strong> and <strong id="mrs_01_0801__ae4f407a2a8a1426ca37ace6f40167acd">hbase</strong> indicate the usernames of the OSs that are used during the services.</p>
|
|
<div class="note" id="mrs_01_0801__nb93dfaab21c74987a54c534891d1676c"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><ul id="mrs_01_0801__u530224c047bc489ab8f3df2a4edddb89"><li id="mrs_01_0801__l611b2b19d5c24bda9f47923ac1191b60">Only user <strong id="mrs_01_0801__ae116a351ef6745afbe51be49bb3128ba">root</strong> has the rights to edit the <strong id="mrs_01_0801__a6d2e37a5ccc04b06945960cce2c34481">limits.conf</strong> file.</li><li id="mrs_01_0801__l47a04b6c4a1e4ecdb816b99027095246">If this modification does not take effect, check whether other nofile values exist in the <strong id="mrs_01_0801__a8e8d9f9da403479d9d1743ce1cfd3f8f">/etc/security/limits.d</strong> directory. Such values may overwrite the values set in the <strong id="mrs_01_0801__a01450443528b43ae9cb44fac41830ec6">/etc/security/limits.conf</strong> file.</li><li id="mrs_01_0801__l3eabe1cd7d2140988e1d752c08705efd">If a user needs to perform operations on HBase, set the number of file handles of this user to a value greater than <strong id="mrs_01_0801__adf8ff80f2878414a8e2107532bf6eabd">10000</strong>. If a user needs to perform operations on HDFS, set the number of file handles of this user based on the service traffic. It is recommended that the value not be too small. If a user needs to perform operations on both HBase and HDFS, set the number of file handles of this user to a large value, such as <strong id="mrs_01_0801__adc4461d526e345278aaac8f8aa13ed6e">32768</strong>.</li></ul>
|
|
</div></div>
|
|
</p></li><li id="mrs_01_0801__l1346d24b541a4b1daa8b6131ac5cd8b4"><span>Run the following command to check the limit on the number of file handles of a user:</span><p><p id="mrs_01_0801__a247757c8e3a1418ca2ebd2cc96ebf8de"><strong id="mrs_01_0801__a0db22cf388fc497694979476589b17b0">su - </strong><em id="mrs_01_0801__ac87b6fc1600946f48bf04a47fc751723">user_name</em></p>
|
|
<p id="mrs_01_0801__a44f6a5dd6a7744418b75abbd4c2db6a4"><strong id="mrs_01_0801__ad78aef72446f4c25a86b655c929ef84f">ulimit -n</strong></p>
|
|
<p id="mrs_01_0801__aebefb8fa9d53422c99b4000ff12b9b80">The limit on the number of file handles of this user is displayed as follows.</p>
|
|
<pre class="screen" id="mrs_01_0801__s2f03183435a54512a7de48eeb5313854">8194</pre>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0790.html">Using HDFS</a></div>
|
|
</div>
|
|
</div>
|
|
|