forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
32 lines
5.7 KiB
HTML
32 lines
5.7 KiB
HTML
<a name="mrs_01_2360"></a><a name="mrs_01_2360"></a>
|
|
|
|
<h1 class="topictitle1">Using HDFS AZ Mover</h1>
|
|
<div id="body1606729563286"><div class="section" id="mrs_01_2360__section524983601415"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_2360__p6784661512">AZ Mover is a copy migration tool used to move copies to meet the new AZ policies set on the directory. It can be used to migrate copies from one AZ policy to another. AZ Mover instructs NameNode to move copies based on a new AZ policy. If the NameNode refuses to delete the old copies, the new policy may not be met. For example, the copies are marked as outdated.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_2360__section223117232228"><h4 class="sectiontitle">Restrictions</h4><ul id="mrs_01_2360__ul22311923132211"><li id="mrs_01_2360__li14977519121813">Changing the policy name to <strong id="mrs_01_2360__b89142028465">LOCAL_AZ</strong> is the same as that to <strong id="mrs_01_2360__b18143174818615">ONE_AZ</strong> because the client location cannot be determined when the uploaded file is written.</li><li id="mrs_01_2360__li16977101981814">Mover cannot determine the AZ status. As a result, the copy may be moved to the abnormal AZ and depends on NameNode for further processing.</li><li id="mrs_01_2360__li197731910182">Mover depends on whether the number of DataNodes in each AZ meets the minimum requirement. If the AZ Mover is executed in an AZ with a small number of DataNodes, the result may be different from the expected result.</li><li id="mrs_01_2360__li697710195185">Mover only meets the AZ-level policies and does not guarantee to meet the basic block placement policy (BPP).</li><li id="mrs_01_2360__li1797781981812">Mover does not support the change of replication factors. If the number of copies in the new AZ is different from that in the old AZ, an exception occurs.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_2360__section453268151617"><h4 class="sectiontitle">Procedure</h4><ol id="mrs_01_2360__ol68661933102018"><li id="mrs_01_2360__li18851186175920"><span>Run the following command to go to the client installation directory.</span><p><p id="mrs_01_2360__p56635217598"><strong id="mrs_01_2360__b327235215917">cd /opt/client</strong></p>
|
|
</p></li><li id="mrs_01_2360__li1485114614593"><span>Run the following command to configure environment variables:</span><p><p id="mrs_01_2360__p96161241595"><strong id="mrs_01_2360__b7679114810591">source bigdata_env</strong></p>
|
|
</p></li><li id="mrs_01_2360__li98513610593"><span>If the cluster is in security mode, the user must have the read permission on the source directory or file and the write permission on the destination directory, and run the following command to authenticate the user: In normal mode, skip user authentication.</span><p><p id="mrs_01_2360__p6272528115914"><strong id="mrs_01_2360__b946518553129">kinit</strong> <em id="mrs_01_2360__i147010558121">Component service user</em></p>
|
|
</p></li><li id="mrs_01_2360__li6866833112012"><span>Create a directory and set an AZ policy.</span><p><p id="mrs_01_2360__p312012913417">Run the following command to create a directory.</p>
|
|
<p id="mrs_01_2360__p171201092418"><strong id="mrs_01_2360__b46373500117">hdfs dfs -mkdir</strong> <<em id="mrs_01_2360__i1063895031117">path</em>></p>
|
|
<p id="mrs_01_2360__p0120799410">Run the following command to set the AZ policy (<strong id="mrs_01_2360__b8227714141310">azexpression</strong> indicates the AZ policy):</p>
|
|
<p id="mrs_01_2360__p18120149944"><strong id="mrs_01_2360__b187110594115">hdfs dfsadmin -setAZExpression </strong><em id="mrs_01_2360__i4590125921114"><path</em><strong id="mrs_01_2360__b1484915815126">> </strong><em id="mrs_01_2360__i722689171213"><azexpression</em>></p>
|
|
<p id="mrs_01_2360__p81201191341">Run the following command to view the AZ policy:</p>
|
|
<p id="mrs_01_2360__p18120298417"><strong id="mrs_01_2360__b2087695214136">hdfs dfsadmin -getAZExpression </strong><em id="mrs_01_2360__i17231135361315"><path</em>></p>
|
|
</p></li><li id="mrs_01_2360__li15866133318204"><span>Upload files to the directory.</span><p><p id="mrs_01_2360__p128211567416"><strong id="mrs_01_2360__b126911041416">hdfs dfs -put <</strong><em id="mrs_01_2360__i55101107140">localfile</em><strong id="mrs_01_2360__b11612266143">> <</strong><em id="mrs_01_2360__i12924196161413">hdfs-path</em><strong id="mrs_01_2360__b3613468143">></strong></p>
|
|
</p></li><li id="mrs_01_2360__li1986603310209"><span>Delete the old policy from the directory and set a new policy.</span><p><p id="mrs_01_2360__p322613312061">Run the following command to clear the old policy:</p>
|
|
<p id="mrs_01_2360__p3827129165"><strong id="mrs_01_2360__b113933336153">hdfs dfsadmin -clearAZExpression <</strong><em id="mrs_01_2360__i117110331152">path</em><strong id="mrs_01_2360__b83931733191519">></strong></p>
|
|
<p id="mrs_01_2360__p6774549263">Run the following command to configure a new policy:</p>
|
|
<p id="mrs_01_2360__p388691111714"><strong id="mrs_01_2360__b141921352141516">hdfs dfsadmin -setAZExpression <</strong><em id="mrs_01_2360__i11539115241516">path</em><strong id="mrs_01_2360__b1041217557155">> <</strong><em id="mrs_01_2360__i569614550157">azexpression</em><strong id="mrs_01_2360__b16412155591520">></strong></p>
|
|
</p></li><li id="mrs_01_2360__li5866103319207"><span>Run the <strong id="mrs_01_2360__b1276452212146">azmover</strong> command to make the copy distribution meet the new AZ policy.</span><p><p id="mrs_01_2360__p9620105210232"><strong id="mrs_01_2360__b12620165214232">hdfs azmover -p /targetDirecotry</strong></p>
|
|
</p></li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0790.html">Using HDFS</a></div>
|
|
</div>
|
|
</div>
|
|
|