forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
93 lines
12 KiB
HTML
93 lines
12 KiB
HTML
<a name="mrs_01_1663"></a><a name="mrs_01_1663"></a>
|
|
|
|
<h1 class="topictitle1">Using the HDFS Client</h1>
|
|
<div id="body1595904092462"><div class="section" id="mrs_01_1663__se9608011680e423ca403d5207c374daa"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_1663__a38fb70b283514ab78798b4fe0c90e0b6">This section describes how to use the HDFS client in an O&M scenario or service scenario. </p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1663__sa8a135a114cf4cbc8674242bd0cfebd7"><h4 class="sectiontitle">Prerequisites</h4><ul id="mrs_01_1663__u20fcfd465a494c5598386a4ce5d296d7"><li id="mrs_01_1663__la7415f93eb2e4da28a053cbf667f0824">The client has been installed.<p id="mrs_01_1663__a57a39be1ee37483eafbd78350f970751"><a name="mrs_01_1663__la7415f93eb2e4da28a053cbf667f0824"></a><a name="la7415f93eb2e4da28a053cbf667f0824"></a>For example, the installation directory is <strong id="mrs_01_1663__b1257120543614">/opt/hadoopclient</strong>. The client directory in the following operations is only an example. Change it to the actual installation directory.</p>
|
|
</li><li id="mrs_01_1663__la8a352e20c414015a0e60f8023ecc0a5">Service component users are created by the administrator as required. In security mode, machine-machine users need to download the keytab file. A human-machine user needs to change the password upon the first login. (This operation is not required in normal mode.)</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1663__s729d2c48e7354c7bb15884e152e37570"><h4 class="sectiontitle">Using the HDFS Client</h4><ol id="mrs_01_1663__o65a723c3f0644a5ca5bc95b50051ef8f"><li id="mrs_01_1663__l3a2e1b7a8d004152865881b7d9bda58b"><span>Log in to the node where the client is installed as the client installation user.</span></li><li id="mrs_01_1663__l918962d747ac488fae8d369a02cf0b01"><span>Run the following command to go to the client installation directory:</span><p><p id="mrs_01_1663__ae6fabba0d23e4ae4bc35c6ef79cff605"><strong id="mrs_01_1663__a5a32ca4aacc5401ca0e8e8a12040e570">cd /opt/hadoopclient</strong></p>
|
|
</p></li><li id="mrs_01_1663__lf110d935709c4ac48ee0ad1420665c2e"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_1663__acfff651122a74259a5e01741838c29e9"><strong id="mrs_01_1663__a468a1a99aae541fbab02686dddf0e45a">source bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_1663__lc24dad0c1fb34eec930f24087726a7fd"><span>If the cluster is in security mode, run the following command to authenticate the user. In normal mode, user authentication is not required.</span><p><p id="mrs_01_1663__a2a7d2557b3594004a61ea1dc86fb7d75"><strong id="mrs_01_1663__aa81cf28932fa4b73b9fa86f517864b8d">kinit</strong> <em id="mrs_01_1663__a3b95ea3e27ab410b9ddb246864099722">Component service user</em></p>
|
|
</p></li><li id="mrs_01_1663__li6626247810502"><span>Run the HDFS Shell command. Example:</span><p><p id="mrs_01_1663__p627806210507"><strong id="mrs_01_1663__b49237178104947">hdfs dfs -ls /</strong></p>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_1663__section1462718419"><h4 class="sectiontitle">Common HDFS Client Commands</h4><p id="mrs_01_1663__p10844101962010">The following table lists common HDFS client commands.</p>
|
|
<p id="mrs_01_1663__p13576181674114">For more commands, see <a href="https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CommandsManual.html#User_Commands" target="_blank" rel="noopener noreferrer">https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CommandsManual.html#User_Commands</a>.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1663__table122538012428" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Common HDFS client commands</caption><thead align="left"><tr id="mrs_01_1663__row142532012426"><th align="left" class="cellrowborder" valign="top" width="32.769999999999996%" id="mcps1.3.4.4.2.4.1.1"><p id="mrs_01_1663__p122532016422">Command</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="22.53%" id="mcps1.3.4.4.2.4.1.2"><p id="mrs_01_1663__p162532064218">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="44.7%" id="mcps1.3.4.4.2.4.1.3"><p id="mrs_01_1663__p19381142414424">Example</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1663__row825370154217"><td class="cellrowborder" valign="top" width="32.769999999999996%" headers="mcps1.3.4.4.2.4.1.1 "><p id="mrs_01_1663__p73441050124515"><strong id="mrs_01_1663__b084642014454">hdfs dfs -mkdir</strong> <em id="mrs_01_1663__i1717302413451">Folder name</em></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.53%" headers="mcps1.3.4.4.2.4.1.2 "><p id="mrs_01_1663__p14343750114520">Used to create a folder.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="44.7%" headers="mcps1.3.4.4.2.4.1.3 "><p id="mrs_01_1663__p10507828134610"><strong id="mrs_01_1663__b1550710285462">hdfs dfs -mkdir /tmp/mydir</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1663__row20253701428"><td class="cellrowborder" valign="top" width="32.769999999999996%" headers="mcps1.3.4.4.2.4.1.1 "><p id="mrs_01_1663__p19253180204218"><strong id="mrs_01_1663__b10296201617453">hdfs dfs -ls</strong> <em id="mrs_01_1663__i18303121395017">Folder name</em></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.53%" headers="mcps1.3.4.4.2.4.1.2 "><p id="mrs_01_1663__p182538014216">Used to view a folder.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="44.7%" headers="mcps1.3.4.4.2.4.1.3 "><p id="mrs_01_1663__p118764654513"><strong id="mrs_01_1663__b198744624515">hdfs dfs -ls /tmp</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1663__row3253608426"><td class="cellrowborder" valign="top" width="32.769999999999996%" headers="mcps1.3.4.4.2.4.1.1 "><p id="mrs_01_1663__p17651195517480"><strong id="mrs_01_1663__b135271910154910">hdfs dfs -put</strong> <em id="mrs_01_1663__i1449832285017">Local file on the client node</em> <em id="mrs_01_1663__i316313419370">Specified HDFS path</em></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.53%" headers="mcps1.3.4.4.2.4.1.2 "><p id="mrs_01_1663__p166501855104816">Used to upload a local file to a specified HDFS path.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="44.7%" headers="mcps1.3.4.4.2.4.1.3 "><p id="mrs_01_1663__p4381224154220"><strong id="mrs_01_1663__b1235793518513">hdfs dfs -put /opt/test.txt /tmp</strong></p>
|
|
<p id="mrs_01_1663__p86791037195116">Upload the <span class="filepath" id="mrs_01_1663__filepath0651255175220"><b>/opt/test.txt</b></span> file on the client node to the <span class="filepath" id="mrs_01_1663__filepath883416112533"><b>/tmp</b></span> directory of HDFS.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1663__row2025313010422"><td class="cellrowborder" valign="top" width="32.769999999999996%" headers="mcps1.3.4.4.2.4.1.1 "><p id="mrs_01_1663__p112541001425"><strong id="mrs_01_1663__b12856934135312">hdfs dfs -get</strong> <em id="mrs_01_1663__i1234994885018">Specified file on HDFS </em><em id="mrs_01_1663__i3152356175316">Specified path on the client node</em></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.53%" headers="mcps1.3.4.4.2.4.1.2 "><p id="mrs_01_1663__p22541020427">Used to download the HDFS file to the specified local path.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="44.7%" headers="mcps1.3.4.4.2.4.1.3 "><p id="mrs_01_1663__p2381424154211"><strong id="mrs_01_1663__b5974131017541">hdfs dfs -get /tmp/test.txt /opt/</strong></p>
|
|
<p id="mrs_01_1663__p74164123545">Download the <span class="filepath" id="mrs_01_1663__filepath1888121165511"><b>/tmp/test.txt</b></span> file on HDFS to the <span class="filepath" id="mrs_01_1663__filepath6997192318555"><b>/opt</b></span> path on the client node.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1663__row225414054213"><td class="cellrowborder" valign="top" width="32.769999999999996%" headers="mcps1.3.4.4.2.4.1.1 "><p id="mrs_01_1663__p7206115664812"><strong id="mrs_01_1663__b157571485477">hdfs dfs -rm -r -f </strong> <em id="mrs_01_1663__i189891456361">Specified folder on HDFS</em></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.53%" headers="mcps1.3.4.4.2.4.1.2 "><p id="mrs_01_1663__p120625694814">Used to delete a folder.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="44.7%" headers="mcps1.3.4.4.2.4.1.3 "><p id="mrs_01_1663__p123811524114215"><strong id="mrs_01_1663__b29910183718">hdfs dfs -rm -r -f /tmp/mydir</strong></p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1663__row16712134594813"><td class="cellrowborder" valign="top" width="32.769999999999996%" headers="mcps1.3.4.4.2.4.1.1 "><p id="mrs_01_1663__p67131645154812"><strong id="mrs_01_1663__b78181552164918">hdfs dfs -chmod</strong> <em id="mrs_01_1663__i2549307495">Permission parameter File directory</em></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="22.53%" headers="mcps1.3.4.4.2.4.1.2 "><p id="mrs_01_1663__p147135459488">Used to configure the HDFS directory permission for a user.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="44.7%" headers="mcps1.3.4.4.2.4.1.3 "><p id="mrs_01_1663__p1071313452483"><strong id="mrs_01_1663__b18890164974914">hdfs dfs -chmod 700 /tmp/test</strong></p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1663__section29499362105048"><h4 class="sectiontitle">Client-related FAQs</h4><ol id="mrs_01_1663__ol744197105431"><li id="mrs_01_1663__li15025511105431">What do I do when the HDFS client exits abnormally and error message "java.lang.OutOfMemoryError" is displayed after the HDFS client command is running?<p id="mrs_01_1663__p17611479105954"><a name="mrs_01_1663__li15025511105431"></a><a name="li15025511105431"></a>This problem occurs because the memory required for running the HDFS client exceeds the preset upper limit (128 MB by default). You can change the memory upper limit of the client by modifying <span class="parmname" id="mrs_01_1663__parmname343303011723"><b>CLIENT_GC_OPTS</b></span> in <em id="mrs_01_1663__i9633360343389"><Client installation path></em><strong id="mrs_01_1663__b1366028603389">/HDFS/component_env</strong>. For example, if you want to set the upper limit to 1 GB, run the following command:</p>
|
|
<pre class="screen" id="mrs_01_1663__screen14993084173648">CLIENT_GC_OPTS="-Xmx1G"</pre>
|
|
<p id="mrs_01_1663__p6462255411821">After the modification, run the following command to make the modification take effect:</p>
|
|
<p id="mrs_01_1663__p44359303111033"><strong id="mrs_01_1663__b17346645473389">source </strong><<em id="mrs_01_1663__i351568153389">Client installation path</em>>/<strong id="mrs_01_1663__b16655634523389">/bigdata_env</strong></p>
|
|
</li><li id="mrs_01_1663__li41575152105932">How do I set the log level when the HDFS client is running?<p id="mrs_01_1663__p63732354111346"><a name="mrs_01_1663__li41575152105932"></a><a name="li41575152105932"></a>By default, the logs generated during the running of the HDFS client are printed to the console. The default log level is INFO. To enable the DEBUG log level for fault locating, run the following command to export an environment variable:</p>
|
|
<p id="mrs_01_1663__p25300767111555"><strong id="mrs_01_1663__b38501362111627">export HADOOP_ROOT_LOGGER=DEBUG,console</strong></p>
|
|
<p id="mrs_01_1663__p54226982111623">Then run the HDFS Shell command to generate the DEBUG logs.</p>
|
|
<p id="mrs_01_1663__p26281297111741">If you want to print INFO logs again, run the following command:</p>
|
|
<p id="mrs_01_1663__p3770704811184"><strong id="mrs_01_1663__b13824849111810">export HADOOP_ROOT_LOGGER=INFO,console</strong></p>
|
|
</li><li id="mrs_01_1663__li3683143313555">How do I delete HDFS files permanently?<p id="mrs_01_1663__p927020207565"><a name="mrs_01_1663__li3683143313555"></a><a name="li3683143313555"></a>HDFS provides a recycle bin mechanism. Typically, after an HDFS file is deleted, the file is moved to the recycle bin of HDFS. If the file is no longer needed and the storage space needs to be released, clear the corresponding recycle bin directory, for example, <strong id="mrs_01_1663__b1420020551376">hdfs://hacluster/user/xxx/.Trash/Current/</strong><em id="mrs_01_1663__i0261058183714">xxx</em>.</p>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0790.html">Using HDFS</a></div>
|
|
</div>
|
|
</div>
|
|
|