forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
41 lines
6.1 KiB
HTML
41 lines
6.1 KiB
HTML
<a name="mrs_01_1469"></a><a name="mrs_01_1469"></a>
|
|
|
|
<h1 class="topictitle1">How Do I Logically Split Data Across Different Namespaces?</h1>
|
|
<div id="body1595920216657"><div class="section" id="mrs_01_1469__s47c9cbcf573843febf16d62025f89d4b"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1469__a2787d081065d403a849032e9d6c61723">How do I logically split data across different namespaces?</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1469__s000282ebeb92488aa9a30ff46e247825"><h4 class="sectiontitle">Answer</h4><ul id="mrs_01_1469__u29496c038ff845be927a2d8e071bd73e"><li id="mrs_01_1469__l590c73753ccc4bb587348727d1de7179">Configuration:<p id="mrs_01_1469__af8ab41e3fc0d44d6b6e3a4e45561f98b"><a name="mrs_01_1469__l590c73753ccc4bb587348727d1de7179"></a><a name="l590c73753ccc4bb587348727d1de7179"></a>To logically split data across different namespaces, you must update the following configuration in the <strong id="mrs_01_1469__b111707119882018">core-site.xml</strong> file of HDFS, Hive, and Spark.</p>
|
|
<div class="note" id="mrs_01_1469__nff1f4dc98d38415f9576e849353428c9"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1469__a85c6b0224e944928914b9ef1ace41c2d">Changing the Hive component will change the locations of carbonstore and warehouse.</p>
|
|
</div></div>
|
|
<ul id="mrs_01_1469__u2dc951f5a8d24fd18f355b45d3dbdff6"><li id="mrs_01_1469__l55176a557cd9415aaa3c8eb1868f1085">Configuration in HDFS<ul id="mrs_01_1469__u6d18b5a0933e4c9abd46c195cb15903a"><li id="mrs_01_1469__lcf71602ce65b4f488ca3f91df0910c17"><strong id="mrs_01_1469__b202691399682018">fs.defaultFS</strong>: Name of the default file system. The URI mode must be set to <strong id="mrs_01_1469__b156793437182018">viewfs</strong>. When <strong id="mrs_01_1469__b174789463482018">viewfs</strong> is used, the permission part must be <strong id="mrs_01_1469__b104785144682018">ClusterX</strong>.</li><li id="mrs_01_1469__lc78f91e86f0c44b4b846f6f06a94d7a3"><strong id="mrs_01_1469__b175494545582018">fs.viewfs.mountable.ClusterX.homedir</strong>: Home directory base path. You can use the getHomeDirectory() method defined in <strong id="mrs_01_1469__b58721984182018">FileSystem/FileContext</strong> to access the home directory.</li><li id="mrs_01_1469__l2a2338decfd54b6d84c9895bf6557bee">fs.viewfs.mountable.default.link.<dir_name>: ViewFS mount table.</li></ul>
|
|
<p id="mrs_01_1469__a52df4ec196df4294a54455d263fcf9cd">Example:</p>
|
|
<pre class="screen" id="mrs_01_1469__s6c1844885f624fecb407e404e80eb58a"><property>
|
|
<name>fs.defaultFS</name>
|
|
<value>viewfs://ClusterX/</value>
|
|
</property>
|
|
<property>
|
|
<name>fs.viewfs.mounttable.ClusterX.link./folder1</name>
|
|
<value>hdfs://NS1/folder1</value>
|
|
</property>
|
|
<property>
|
|
<name>fs.viewfs.mounttable.ClusterX.link./folder2</name>
|
|
<value>hdfs://NS2/folder2</value>
|
|
</property></pre>
|
|
</li><li id="mrs_01_1469__lbb80b556e9a148bc892c53a287262a56">Configurations in Hive and Spark<p id="mrs_01_1469__af88d558da1b64714888f09ffe7dcaf89"><a name="mrs_01_1469__lbb80b556e9a148bc892c53a287262a56"></a><a name="lbb80b556e9a148bc892c53a287262a56"></a><strong id="mrs_01_1469__b95530621482018">fs.defaultFS</strong>: Name of the default file system. The URI mode must be set to <strong id="mrs_01_1469__b17004898182018">viewfs</strong>. When <strong id="mrs_01_1469__b33842450182018">viewfs</strong> is used, the permission part must be <strong id="mrs_01_1469__b74683985082018">ClusterX</strong>.</p>
|
|
</li></ul>
|
|
</li></ul>
|
|
<ul id="mrs_01_1469__u974cb7a65c6a4d3c94af9877757112b4"><li id="mrs_01_1469__lc4c2c526663b4ee781ec203871edfec3">Syntax:<p id="mrs_01_1469__aa864c4763c8c4cdd87a315a21e58d6da"><a name="mrs_01_1469__lc4c2c526663b4ee781ec203871edfec3"></a><a name="lc4c2c526663b4ee781ec203871edfec3"></a><b><span class="cmdname" id="mrs_01_1469__c6a851486a77a47828084f6954e179083">LOAD DATA INPATH</span></b> <i><span class="varname" id="mrs_01_1469__vc0be17634f454f72ba62d8980a0f91b7">'path to data' INTO TABLE table_name OPTIONS ('...');</span></i></p>
|
|
<div class="note" id="mrs_01_1469__n9414b39c9e924d8aa77edb0aea78ba3f"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1469__ae9320fe28fda4e8199ceb98dd6579635">When Spark is configured with the viewFS file system and attempts to load data from HDFS, users must specify a path such as <strong id="mrs_01_1469__b3910154782110">viewfs://</strong> or a relative path as the file path in the <strong id="mrs_01_1469__b21401752122116">LOAD</strong> statement.</p>
|
|
</div></div>
|
|
</li><li id="mrs_01_1469__l4ac7b5991f3543649b12c32eb505a5a6">Example:<ul id="mrs_01_1469__u5c20eb3cfdd54b38a408c3a134ce324c"><li id="mrs_01_1469__ldd374fbc057f4248b702f9b7dd09a464">Sample viewFS path:<p id="mrs_01_1469__aef95cececbe74d8e8372798034f1ce56"><a name="mrs_01_1469__ldd374fbc057f4248b702f9b7dd09a464"></a><a name="ldd374fbc057f4248b702f9b7dd09a464"></a><b><span class="cmdname" id="mrs_01_1469__c14572837c9044c6ca50d504d6c8b41ed">LOAD DATA INPATH</span></b> <i><span class="varname" id="mrs_01_1469__v1034dece72b34c749adc8e36fb5b553d">'viewfs://ClusterX/dir/data.csv' INTO TABLE table_name OPTIONS ('...');</span></i></p>
|
|
</li><li id="mrs_01_1469__lf1bc52c0809d426d8ed3080acc079678">Sample relative path:<p id="mrs_01_1469__a88428b0949ab493db286a00accaf544e"><a name="mrs_01_1469__lf1bc52c0809d426d8ed3080acc079678"></a><a name="lf1bc52c0809d426d8ed3080acc079678"></a><b><span class="cmdname" id="mrs_01_1469__ce1d2a0e90ade412eabc6ee98b7cf1d1f">LOAD DATA INPATH</span></b> <i><span class="varname" id="mrs_01_1469__v9aa9043f1084494999778b223d813f67">'/apps/input_data1.txt'</span></i> <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_1469__c4b26aa2244934a688fc7f08a11f9efe9">INTO TABLE</span></b></i> <i><span class="varname" id="mrs_01_1469__v0dcaadd37ead42babea64fcecca66f7c">table_name</span></i>;</p>
|
|
</li></ul>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1457.html">CarbonData FAQ</a></div>
|
|
</div>
|
|
</div>
|
|
|