forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
76 lines
11 KiB
HTML
76 lines
11 KiB
HTML
<a name="mrs_01_0757"></a><a name="mrs_01_0757"></a>
|
|
|
|
<h1 class="topictitle1">Common Operations of Alluxio</h1>
|
|
<div id="body1589421646586"><div class="section" id="mrs_01_0757__section1897373121617"><h4 class="sectiontitle">Preparations</h4><ol id="mrs_01_0757__ol12402192911716"><li id="mrs_01_0757__li540232921720">Create a cluster with Alluxio installed.</li><li id="mrs_01_0757__li1286930121811">Log in to the active Master node in a cluster as user <strong id="mrs_01_0757__b2081798379741">root</strong> using the password set during cluster creation.</li><li id="mrs_01_0757__li4119133341810">Run the following command to configure environment variables:<p id="mrs_01_0757__p114799417597"><a name="mrs_01_0757__li4119133341810"></a><a name="li4119133341810"></a><strong id="mrs_01_0757__b195093717188">source /opt/client/bigdata_env</strong></p>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0757__section173691030181412"><h4 class="sectiontitle">Using the Alluxio Shell</h4><p id="mrs_01_0757__p97661623111612">The <a href="https://docs.alluxio.io/os/user/2.0/en/basic/Command-Line-Interface.html" target="_blank" rel="noopener noreferrer">Alluxio shell</a> contains multiple command line operations that interact with Alluxio.</p>
|
|
<ul id="mrs_01_0757__ul48082262169"><li id="mrs_01_0757__li108081826101617">View a file system operation command list:<p id="mrs_01_0757__p45291425182813"><a name="mrs_01_0757__li108081826101617"></a><a name="li108081826101617"></a><b><span class="cmdname" id="mrs_01_0757__cmdname228514107292">alluxio fs</span></b></p>
|
|
</li><li id="mrs_01_0757__li7905133917165">Run the <strong id="mrs_01_0757__b14388193415117">ls</strong> command to list the files in Alluxio. For example, list all files in the root directory:<p id="mrs_01_0757__p13888123432810"><b><span class="cmdname" id="mrs_01_0757__cmdname817814902818">alluxio fs ls /</span></b></p>
|
|
</li><li id="mrs_01_0757__li5541163312185">Run the <strong id="mrs_01_0757__b455558087741">copyFromLocal</strong> command to copy local files to Alluxio:<p id="mrs_01_0757__p889372912298"><b><span class="cmdname" id="mrs_01_0757__cmdname1370154217293">alluxio fs copyFromLocal /home/test_input.txt /test_input.txt</span></b></p>
|
|
<div class="p" id="mrs_01_0757__p315035214293">Command output:<pre class="screen" id="mrs_01_0757__screen647342142912">Copied file:///home/test_input.txt to /test_input.txt</pre>
|
|
</div>
|
|
</li><li id="mrs_01_0757__li690045611194">Run the <strong id="mrs_01_0757__b281871618741">ls</strong> command again to list the files in Alluxio. The copied <strong id="mrs_01_0757__b1969681347741">test_input.txt</strong> file is listed:<p id="mrs_01_0757__p1686003983110"><b><span class="cmdname" id="mrs_01_0757__cmdname19649134613314">alluxio fs ls /</span></b></p>
|
|
<div class="p" id="mrs_01_0757__p434413583211">Command output:<pre class="screen" id="mrs_01_0757__screen1977863712312">12 PERSISTED 11-28-2019 17:10:17:449 100% /test_input.txt</pre>
|
|
</div>
|
|
<p id="mrs_01_0757__p67780379318">The <strong id="mrs_01_0757__b1641880488741">test_input.txt</strong> file is displayed in Alluxio. The parameters in the file indicate the file size, whether the file is persistent, creation date, cache ratio of the file in Alluxio, and file name.</p>
|
|
</li><li id="mrs_01_0757__li1948210508213">Run the <strong id="mrs_01_0757__b973328303741">cat</strong> command to print file content:<p id="mrs_01_0757__p2856155511352"><b><span class="cmdname" id="mrs_01_0757__cmdname13460137203615">alluxio fs cat /test_input.txt</span></b></p>
|
|
<div class="p" id="mrs_01_0757__p5540636367">Command output:<pre class="screen" id="mrs_01_0757__screen4481205303519">Test Alluxio</pre>
|
|
</div>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_0757__section0330205781418"><h4 class="sectiontitle">Mounting Function of Alluxio</h4><p id="mrs_01_0757__p2380201410260">Alluxio uses a unified namespace feature to unify the access to storage systems. For details, see <a href="https://docs.alluxio.io/os/user/2.0/en/advanced/Namespace-Management.html" target="_blank" rel="noopener noreferrer">https://docs.alluxio.io/os/user/2.0/en/advanced/Namespace-Management.html</a>.</p>
|
|
<p id="mrs_01_0757__p19553152262814">This feature allows users to mount different storage systems to an Alluxio namespace and seamlessly access files across storage systems through the Alluxio namespace.</p>
|
|
<ol id="mrs_01_0757__ol1695177172411"><li id="mrs_01_0757__li109514710247">Create a directory as a mount point in Alluxio.<div class="p" id="mrs_01_0757__p8704177182414"><a name="mrs_01_0757__li109514710247"></a><a name="li109514710247"></a><strong id="mrs_01_0757__b16753141322418">alluxio fs mkdir /mnt</strong><pre class="screen" id="mrs_01_0757__screen87798202410">Successfully created directory /mnt</pre>
|
|
</div>
|
|
</li><li id="mrs_01_0757__li1842114221242">Mount an existing OBS file system to Alluxio. (Prerequisite: An agency with the <strong id="mrs_01_0757__b1438114421865">OBS OperateAccess</strong> permission has been configured for the cluster. The <strong id="mrs_01_0757__b166462913368">obs-mrstest</strong> file system is used as an example. Replace the file system name with the actual one.<div class="p" id="mrs_01_0757__p26283412911"><strong id="mrs_01_0757__b11796111102917">alluxio fs mount /mnt/obs obs://obs-mrstest/data</strong><pre class="screen" id="mrs_01_0757__screen174511202918">Mounted obs://obs-mrstest/data at /mnt/obs</pre>
|
|
</div>
|
|
</li><li id="mrs_01_0757__li14969163614260">List files in the OBS file system using the Alluxio namespace. Run the <strong id="mrs_01_0757__b442122152719">ls</strong> command to list the files in the OBS mount directory.<div class="p" id="mrs_01_0757__p16466113313012"><strong id="mrs_01_0757__b13475339193012">alluxio fs ls /mnt/obs</strong><pre class="screen" id="mrs_01_0757__screen15859431163019">38 PERSISTED 11-28-2019 17:42:54:554 0% /mnt/obs/hive_load.txt
|
|
12 PERSISTED 11-28-2019 17:43:07:743 0% /mnt/obs/test_input.txt</pre>
|
|
</div>
|
|
<p id="mrs_01_0757__p11859203193015">You can also view the newly mounted files and directories on the Alluxio web UI.</p>
|
|
</li><li id="mrs_01_0757__li86686241398">After the mounting is complete, you can seamlessly exchange data between different storage systems through the unified namespace of Alluxio. For example, run the <strong id="mrs_01_0757__b11218109121510">ls -R</strong> command to list all files in a directory recursively:<div class="p" id="mrs_01_0757__p839411345404"><strong id="mrs_01_0757__b95891255184011">alluxio fs ls -R /</strong><pre class="screen" id="mrs_01_0757__screen102691632144013"> 0 PERSISTED 11-28-2019 11:15:19:719 DIR /app-logs
|
|
1 PERSISTED 11-28-2019 11:18:36:885 DIR /apps
|
|
1 PERSISTED 11-28-2019 11:18:40:209 DIR /apps/templeton
|
|
239440292 PERSISTED 11-28-2019 11:18:40:209 0% /apps/templeton/hive.tar.gz
|
|
.....
|
|
1 PERSISTED 11-28-2019 19:00:23:879 DIR /mnt
|
|
2 PERSISTED 11-28-2019 19:00:23:879 DIR /mnt/obs
|
|
38 PERSISTED 11-28-2019 17:42:54:554 0% /mnt/obs/hive_load.txt
|
|
12 PERSISTED 11-28-2019 17:43:07:743 0% /mnt/obs/test_input.txt
|
|
.....</pre>
|
|
</div>
|
|
<p id="mrs_01_0757__p1526917327401">The command output shows all files that are from the mounted storage system in the root directory of the Alluxio file system (the default directory is the HDFS root directory, that is, <strong id="mrs_01_0757__b949391114156">hdfs://hacluster/</strong>). The <strong id="mrs_01_0757__b3478140152">/app-logs</strong> and <strong id="mrs_01_0757__b1147614131519">/apps</strong> directories are in HDFS, and the <strong id="mrs_01_0757__b347131412157">/mnt/obs/</strong> directory is in OBS.</p>
|
|
</li></ol>
|
|
<p id="mrs_01_0757__p7440934174"></p>
|
|
</div>
|
|
<div class="section" id="mrs_01_0757__section784202117153"><h4 class="sectiontitle">Using Alluxio to Accelerate Data Access</h4><p id="mrs_01_0757__p145654573215">Alluxio can accelerate data access, because it uses memory to store data. Example commands are provided as follows:</p>
|
|
<ol id="mrs_01_0757__ol9263155015321"><li id="mrs_01_0757__li9263850183212">Upload the <strong id="mrs_01_0757__b404814853741">test_data.csv</strong> file (a sample that records recipes) to the <strong id="mrs_01_0757__b1287355128741">/data</strong> directory of the <strong id="mrs_01_0757__b542416378741">obs-mrstest</strong> file system. Run the <strong id="mrs_01_0757__b54801592331">ls</strong> command to display the file status.<div class="p" id="mrs_01_0757__p0418133054211"><strong id="mrs_01_0757__b8160173493013">alluxio fs ls /mnt/obs/test_data.csv</strong><pre class="screen" id="mrs_01_0757__screen686322854217">294520189 PERSISTED 11-28-2019 19:38:55:000 0% /mnt/obs/test_data.csv</pre>
|
|
</div>
|
|
<p id="mrs_01_0757__p1886392818422">The output indicates that the cache percentage of the file in Alluxio is 0%, that is, the file is not in Alluxio memory.</p>
|
|
</li><li id="mrs_01_0757__li20345125116338">Count the occurrence times of the word "milk" in the file, and calculate the time consumed.<div class="p" id="mrs_01_0757__p413014425422"><a name="mrs_01_0757__li20345125116338"></a><a name="li20345125116338"></a><strong id="mrs_01_0757__b2411740143015">time alluxio fs cat /mnt/obs/test_data.csv | grep -c milk</strong><pre class="screen" id="mrs_01_0757__screen1650414409422">52180
|
|
|
|
real 0m10.765s
|
|
user 0m5.540s
|
|
sys 0m0.696s</pre>
|
|
</div>
|
|
</li><li id="mrs_01_0757__li814431253715">Data is stored in memory after being read for the first time. When Alluxio reads data again, the data access speed is increased. For example, after running the <strong id="mrs_01_0757__b976217950741">cat</strong> command to obtain a file, run the <strong id="mrs_01_0757__b663137762741">ls</strong> command to check the file status.<div class="p" id="mrs_01_0757__p138018504446"><strong id="mrs_01_0757__b5584194633020">alluxio fs ls /mnt/obs/test_data.csv</strong><pre class="screen" id="mrs_01_0757__screen433184444410">294520189 PERSISTED 11-28-2019 19:38:55:000 100% /mnt/obs/test_data.csv</pre>
|
|
</div>
|
|
<p id="mrs_01_0757__p103311544204418">The output shows that the file has been fully loaded to Alluxio.</p>
|
|
</li><li id="mrs_01_0757__li73311835143812">Access the file again, count the occurrence times of the word "eggs", and calculate the time consumed.<div class="p" id="mrs_01_0757__p55488198451"><a name="mrs_01_0757__li73311835143812"></a><a name="li73311835143812"></a><strong id="mrs_01_0757__b91701544303">time alluxio fs cat /mnt/obs/test_data.csv | grep -c eggs</strong><pre class="screen" id="mrs_01_0757__screen1551491794519">59510
|
|
|
|
real 0m5.777s
|
|
user 0m5.992s
|
|
sys 0m0.592s</pre>
|
|
</div>
|
|
<p id="mrs_01_0757__p165145176459">According to the comparison of the two time consumption records, the time consumed for accessing data stored in Alluxio memory is significantly reduced.</p>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0756.html">Using Alluxio</a></div>
|
|
</div>
|
|
</div>
|
|
|