forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
117 lines
16 KiB
HTML
117 lines
16 KiB
HTML
<a name="mrs_01_1406"></a><a name="mrs_01_1406"></a>
|
|
|
|
<h1 class="topictitle1">CarbonData Quick Start</h1>
|
|
<div id="body1595920208265"><p id="mrs_01_1406__ae3ba0b2375534d618d3040836c6f88a8">This section describes how to create CarbonData tables, load data, and query data. This quick start provides operations based on the Spark Beeline client. If you want to use Spark shell, wrap the queries with <strong id="mrs_01_1406__b289155745314">spark.sql()</strong>.</p>
|
|
<p id="mrs_01_1406__p1649165317476"><strong id="mrs_01_1406__b35036220215">The following describes how to load data from a CSV file to a CarbonData table</strong>.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1406__tc9e1ff32aac6410798846181cf7fb057" frame="border" border="1" rules="all"><caption><b>Table 1 </b>CarbonData Quick Start</caption><thead align="left"><tr id="mrs_01_1406__row9477582353"><th align="left" class="cellrowborder" valign="top" width="36.67%" id="mcps1.3.3.2.3.1.1"><p id="mrs_01_1406__p347816893518"><strong id="mrs_01_1406__b69916012493">Operation</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="63.33%" id="mcps1.3.3.2.3.1.2"><p id="mrs_01_1406__p12478984357"><strong id="mrs_01_1406__b16161912497">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1406__row6909629541"><td class="cellrowborder" valign="top" width="36.67%" headers="mcps1.3.3.2.3.1.1 "><p id="mrs_01_1406__p159101529543"><a href="#mrs_01_1406__section497653817420">Preparing a CSV File</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.33%" headers="mcps1.3.3.2.3.1.2 "><p id="mrs_01_1406__p139109215416">Prepare the CSV file to be loaded to the CarbonData Table.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1406__rca77e7c8a8ff4d369172d0072e9e7234"><td class="cellrowborder" valign="top" width="36.67%" headers="mcps1.3.3.2.3.1.1 "><p id="mrs_01_1406__a1f51fbebc1f14c49a0f43c0f67b9fd3b"><a href="#mrs_01_1406__s2af9b9318a0f44c48f3b0fa8217a12fe">Connecting to CarbonData</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.33%" headers="mcps1.3.3.2.3.1.2 "><p id="mrs_01_1406__a252b8024092d429c876204b00744711f">Connect to CarbonData before performing any operations on CarbonData.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1406__r8996ea6ada284683b0ce67ee3fb24d40"><td class="cellrowborder" valign="top" width="36.67%" headers="mcps1.3.3.2.3.1.1 "><p id="mrs_01_1406__a24a914c6ee09442598968b4a49d4a912"><a href="#mrs_01_1406__sffd808b54dab44fc8613a01cc8e39baf">Creating a CarbonData Table</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.33%" headers="mcps1.3.3.2.3.1.2 "><p id="mrs_01_1406__a43dd7a9f9a054d93a5f481ebcd58611b">Create a CarbonData table to load data and perform query operations.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1406__r4f97d98ae5bd49698b6d5d3dfa4e1de2"><td class="cellrowborder" valign="top" width="36.67%" headers="mcps1.3.3.2.3.1.1 "><p id="mrs_01_1406__acda32a2fac9b4aaaaa815adc739c4413"><a href="#mrs_01_1406__s79b594787a2a46819cf07478d4a0d81c">Loading Data to a CarbonData Table</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.33%" headers="mcps1.3.3.2.3.1.2 "><p id="mrs_01_1406__a61096dff64ef4425b6b371ee89c17436">Load data from CSV to the created table.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_1406__reff67320c9a84bac8534a458e514346f"><td class="cellrowborder" valign="top" width="36.67%" headers="mcps1.3.3.2.3.1.1 "><p id="mrs_01_1406__a28fb1826868041b88297ca2bb0694dab"><a href="#mrs_01_1406__s68e9413d1b234b2d91557a1739fc7828">Querying Data from a CarbonData Table</a></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="63.33%" headers="mcps1.3.3.2.3.1.2 "><p id="mrs_01_1406__aa9f561a86e874564b7648658a8355c3e">Perform query operations such as filters and groupby.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="section" id="mrs_01_1406__section497653817420"><a name="mrs_01_1406__section497653817420"></a><a name="section497653817420"></a><h4 class="sectiontitle">Preparing a CSV File</h4><ol id="mrs_01_1406__ol818835614218"><li id="mrs_01_1406__li218825619423">Prepare a CSV file named <strong id="mrs_01_1406__b8623182012356">test.csv</strong> on the local PC. An example is as follows:<pre class="screen" id="mrs_01_1406__screen101887566427">13418592122,1001,<em id="mrs_01_1406__i195961910122118"> MAC address</em>, 2017-10-23 15:32:30,2017-10-24 15:32:30,62.50,74.56
|
|
13418592123 1002, <em id="mrs_01_1406__i5329943135212"> MAC address</em>, 2017-10-23 16:32:30,2017-10-24 16:32:30,17.80,76.28
|
|
13418592124,1003,<em id="mrs_01_1406__i10560049165210"> MAC address</em>, 2017-10-23 17:32:30,2017-10-24 17:32:30,20.40,92.94
|
|
13418592125 1004, <em id="mrs_01_1406__i165578586521"> MAC address</em>, 2017-10-23 18:32:30,2017-10-24 18:32:30,73.84,8.58
|
|
13418592126,1005,<em id="mrs_01_1406__i1461010419532"> MAC address</em>, 2017-10-23 19:32:30,2017-10-24 19:32:30,80.50,88.02
|
|
13418592127 1006, <em id="mrs_01_1406__i8176990538"> MAC address</em>, 2017-10-23 20:32:30,2017-10-24 20:32:30,65.77,71.24
|
|
13418592128,1007,<em id="mrs_01_1406__i229301511533"> MAC address</em>, 2017-10-23 21:32:30,2017-10-24 21:32:30,75.21,76.04
|
|
13418592129,1008,<em id="mrs_01_1406__i128916200535"> MAC address</em>, 2017-10-23 22:32:30,2017-10-24 22:32:30,63.30,94.40
|
|
13418592130, 1009,<em id="mrs_01_1406__i659826145311"> MAC address</em>, 2017-10-23 23:32:30,2017-10-24 23:32:30,95.51,50.17
|
|
13418592131,1010,<em id="mrs_01_1406__i1230191222111"> MAC address</em>, 2017-10-24 00:32:30,2017-10-25 00:32:30,39.62,99.13</pre>
|
|
</li><li id="mrs_01_1406__li0188135634215">Use WinSCP to import the CSV file to the directory of the node where the client is installed, for example, <strong id="mrs_01_1406__b1458913416509">/opt</strong>.</li><li id="mrs_01_1406__li0451175113431">Log in to FusionInsight Manager and choose <strong id="mrs_01_1406__b14797742163810">System</strong>. In the navigation pane on the left, choose <strong id="mrs_01_1406__b186431345163815">Permission</strong> > <strong id="mrs_01_1406__b161711506387">User</strong>, click <strong id="mrs_01_1406__b19138298516">Create</strong> to create human-machine user <strong id="mrs_01_1406__b17191623133913">sparkuser</strong>, and add the user to user groups hadoop (primary group) and hive.</li><li id="mrs_01_1406__li144511151174317">Run the following commands to go to the client installation directory, load environment variables, and authenticate the user.<p id="mrs_01_1406__p745105119433"><a name="mrs_01_1406__li144511151174317"></a><a name="li144511151174317"></a><strong id="mrs_01_1406__b167712182212">cd /</strong><em id="mrs_01_1406__i15683101882110">Client installation directory</em></p>
|
|
<p id="mrs_01_1406__p204516513438"><strong id="mrs_01_1406__b1451195174311">source ./bigdata_env</strong></p>
|
|
<p id="mrs_01_1406__p945145112431"><strong id="mrs_01_1406__b4451135115430">source ./Spark2x/component_env</strong></p>
|
|
<p id="mrs_01_1406__p945155184318"><strong id="mrs_01_1406__b18451155144315">kinit sparkuser</strong></p>
|
|
</li><li id="mrs_01_1406__li122143593123"><a name="mrs_01_1406__li122143593123"></a><a name="li122143593123"></a>Run the following command to upload the CSV file to the <strong id="mrs_01_1406__b1452155544412">/data</strong> directory of the HDFS.<p id="mrs_01_1406__p862595921214"><strong id="mrs_01_1406__b8121318134518">hdfs dfs -put /opt/test.csv /</strong><strong id="mrs_01_1406__b172356291454">data</strong><strong id="mrs_01_1406__b612191812457">/</strong></p>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_1406__s2af9b9318a0f44c48f3b0fa8217a12fe"><a name="mrs_01_1406__s2af9b9318a0f44c48f3b0fa8217a12fe"></a><a name="s2af9b9318a0f44c48f3b0fa8217a12fe"></a><h4 class="sectiontitle">Connecting to CarbonData</h4><ul id="mrs_01_1406__u6ab77193f8f64f989853e2baf78058cb"><li id="mrs_01_1406__le269de8821ba405fa0c4bcb404486367">Use Spark SQL or Spark shell to connect to Spark and run Spark SQL commands.</li><li id="mrs_01_1406__l71ad3c644a0a44b09dc4cd2cc31acc19">Run the following commands to start the JDBCServer and use a JDBC client (for example, Spark Beeline) to connect to the JDBCServer.<p id="mrs_01_1406__p18120635194614"><a name="mrs_01_1406__l71ad3c644a0a44b09dc4cd2cc31acc19"></a><a name="l71ad3c644a0a44b09dc4cd2cc31acc19"></a><strong id="mrs_01_1406__b18120835194610">cd ./Spark2x/spark/bin</strong></p>
|
|
<p id="mrs_01_1406__p1012083524613"><strong id="mrs_01_1406__b1912093515460">./spark-beeline</strong></p>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1406__sffd808b54dab44fc8613a01cc8e39baf"><a name="mrs_01_1406__sffd808b54dab44fc8613a01cc8e39baf"></a><a name="sffd808b54dab44fc8613a01cc8e39baf"></a><h4 class="sectiontitle">Creating a CarbonData Table</h4><p id="mrs_01_1406__a132bc4ce9c8a4976a361e2a1ed30fbae">After connecting Spark Beeline with the JDBCServer, create a CarbonData table to load data and perform query operations. Run the following commands to create a simple table:</p>
|
|
<p id="mrs_01_1406__p7897153725816"><strong id="mrs_01_1406__b555244465815">create table x1 (imei string, deviceInformationId int, mac string, productdate timestamp, updatetime timestamp, gamePointId double, contractNumber double) STORED AS carbondata TBLPROPERTIES ('SORT_COLUMNS'='imei,mac');</strong></p>
|
|
<p id="mrs_01_1406__adf903c5382e1440cb85d6abd32e9af67">The command output is as follows:</p>
|
|
<pre class="screen" id="mrs_01_1406__s0bde6cee8a13468b815e9b05096aadf2">+---------+
|
|
| Result |
|
|
+---------+
|
|
+---------+
|
|
No rows selected (1.093 seconds)</pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_1406__s79b594787a2a46819cf07478d4a0d81c"><a name="mrs_01_1406__s79b594787a2a46819cf07478d4a0d81c"></a><a name="s79b594787a2a46819cf07478d4a0d81c"></a><h4 class="sectiontitle">Loading Data to a CarbonData Table</h4><p id="mrs_01_1406__a6fc2a8f1d9c94f27bb97de59ccc4d822">After you have created a CarbonData table, you can load the data from CSV to the created table.</p>
|
|
<p id="mrs_01_1406__a59f53a20e20b488e83ae5dd3926f704a">Run the following command with required parameters to load data from CSV. The column names of the CarbonData table must match the column names of the CSV file.</p>
|
|
<p id="mrs_01_1406__af01eb112638d4a25b3aca51b14b07995"><strong id="mrs_01_1406__b1485119505372">LOAD DATA inpath 'hdfs://hacluster/data/</strong><em id="mrs_01_1406__i75582513370">test.csv</em><strong id="mrs_01_1406__b1018144183814">' into table </strong><em id="mrs_01_1406__i5922443386">x1</em><strong id="mrs_01_1406__b51817423817"> options('DELIMITER'=',', 'QUOTECHAR'='"','FILEHEADER'='imei, deviceinformationid,mac, productdate,updatetime, gamepointid,contractnumber');</strong></p>
|
|
<p id="mrs_01_1406__ac6c375c07b6c4fab83a1a81e64a6d106"><span class="filepath" id="mrs_01_1406__f0f2197f7f5584322a4d7162988558437"><b>test.csv</b></span> is the CSV file prepared in <a href="#mrs_01_1406__section497653817420">Preparing a CSV File</a> and <span class="filepath" id="mrs_01_1406__f84feff9a8a904f00b039794afb492369"><b>x1</b></span> is the table name.</p>
|
|
<p id="mrs_01_1406__a574ab7d6706d46aca3d50ef661dee431">The CSV example file is as follows:</p>
|
|
<pre class="screen" id="mrs_01_1406__s3569e87c602c4d24bee8182682c18dd4">13418592122,1001,<em id="mrs_01_1406__i2238143711528"> MAC address</em>, 2017-10-23 15:32:30,2017-10-24 15:32:30,62.50,74.56
|
|
13418592123 1002, <em id="mrs_01_1406__i120961070"> MAC address</em>, 2017-10-23 16:32:30,2017-10-24 16:32:30,17.80,76.28
|
|
13418592124,1003,<em id="mrs_01_1406__i224651844"> MAC address</em>, 2017-10-23 17:32:30,2017-10-24 17:32:30,20.40,92.94
|
|
13418592125 1004, <em id="mrs_01_1406__i401281217"> MAC address</em>, 2017-10-23 18:32:30,2017-10-24 18:32:30,73.84,8.58
|
|
13418592126,1005,<em id="mrs_01_1406__i1509909288"> MAC address</em>, 2017-10-23 19:32:30,2017-10-24 19:32:30,80.50,88.02
|
|
13418592127 1006, <em id="mrs_01_1406__i1042901796"> MAC address</em>, 2017-10-23 20:32:30,2017-10-24 20:32:30,65.77,71.24
|
|
13418592128,1007,<em id="mrs_01_1406__i1705242573"> MAC address</em>, 2017-10-23 21:32:30,2017-10-24 21:32:30,75.21,76.04
|
|
13418592129,1008,<em id="mrs_01_1406__i1200047"> MAC address</em>, 2017-10-23 22:32:30,2017-10-24 22:32:30,63.30,94.40
|
|
13418592130, 1009,<em id="mrs_01_1406__i365030345"> MAC address</em>, 2017-10-23 23:32:30,2017-10-24 23:32:30,95.51,50.17
|
|
13418592131,1010,<em id="mrs_01_1406__i144547353535"> MAC address</em>, 2017-10-24 00:32:30,2017-10-25 00:32:30,39.62,99.13</pre>
|
|
<p id="mrs_01_1406__add5d2c8a3ea443da9dc289195d5a7bd1">The command output is as follows:</p>
|
|
<pre class="screen" id="mrs_01_1406__s238251f4d7e04e05be734d01d51d35f6">+------------+
|
|
|Segment ID |
|
|
+------------+
|
|
|0 |
|
|
+------------+
|
|
No rows selected (3.039 seconds)</pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_1406__s68e9413d1b234b2d91557a1739fc7828"><a name="mrs_01_1406__s68e9413d1b234b2d91557a1739fc7828"></a><a name="s68e9413d1b234b2d91557a1739fc7828"></a><h4 class="sectiontitle">Querying Data from a CarbonData Table</h4><p id="mrs_01_1406__a424f8ab3e4bb46098ab6092f4cc066f8">After a CarbonData table is created and the data is loaded, you can perform query operations as required. Some query operations are provided as examples.</p>
|
|
<ul id="mrs_01_1406__u35a27d6f28644b0fbe301eae9f99aa4e"><li id="mrs_01_1406__l7f4c4fcb65674ed9923fce7f1497b97e"><strong id="mrs_01_1406__b1216027195415">Obtaining the number of records</strong><p id="mrs_01_1406__abda24e0f5e634590a77b909201390de1">Run the following command to obtain the number of records in the CarbonData table:</p>
|
|
<p id="mrs_01_1406__a2bdaf1ec9afd4ff8bb80a2d132f1de11"><b><span class="cmdname" id="mrs_01_1406__cmdname17438204511388">select count(*) from x1;</span></b></p>
|
|
</li><li id="mrs_01_1406__l233e7fe5540a4acab1a0f4182a2d2aa5"><strong id="mrs_01_1406__b1965443114548">Querying with the groupby condition</strong><p id="mrs_01_1406__a1f4a3640520c4acfb44b489c0abffac5">Run the following command to obtain the <strong id="mrs_01_1406__b1573158175420">deviceinformationid</strong> records without repetition in the CarbonData table:</p>
|
|
<p id="mrs_01_1406__a88b1b251a7cc43a7a9907c3790bb0b7e"><b><span class="cmdname" id="mrs_01_1406__cmdname19302194219380">select deviceinformationid,count (distinct deviceinformationid) from x1 group by deviceinformationid;</span></b></p>
|
|
</li><li id="mrs_01_1406__l68f35b253090447f84e1ba2c7c8e7fe5"><strong id="mrs_01_1406__b0116352557">Querying with Filter</strong><p id="mrs_01_1406__a4a76d68008a247a1b3b99eafcc6cc989">Run the following command to obtain specific <strong id="mrs_01_1406__b023833145512">deviceinformationid</strong> records:</p>
|
|
<p id="mrs_01_1406__aa1bcbe4e46584a23b3f9161350cb4380"><b><span class="cmdname" id="mrs_01_1406__cmdname7302034165320">select * from x1 where deviceinformationid='1010';</span></b></p>
|
|
</li></ul>
|
|
<div class="note" id="mrs_01_1406__n17bebcb97003447ea9a95ecf1cf1fd1f"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1406__af1105a524b0b4a83a227063b91075966">If the query result has non-English characters, the columns in the query result may not be aligned. This is because characters of different languages occupy different widths.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1406__s1ab71eae69124272b091d392642421fc"><h4 class="sectiontitle">Using CarbonData on Spark-shell</h4><p id="mrs_01_1406__ae5cad4ad44ca4debb65181e12a57ba11">If you need to use CarbonData on a Spark-shell, you need to create a CarbonData table, load data to the CarbonData table, and query data in CarbonData as follows:</p>
|
|
<pre class="screen" id="mrs_01_1406__saaf0f2c736d54c8ab8c7b4376a24e5aa">spark.sql("CREATE TABLE x2(imei string, deviceInformationId int, mac string, productdate timestamp, updatetime timestamp, gamePointId double, contractNumber double) STORED AS carbondata")
|
|
spark.sql("LOAD DATA inpath 'hdfs://hacluster/data/x1_without_header.csv' into table x2 options('DELIMITER'=',', 'QUOTECHAR'='\"','FILEHEADER'='imei, deviceinformationid,mac, productdate,updatetime, gamepointid,contractnumber')")
|
|
spark.sql("SELECT * FROM x2").show()</pre>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1405.html">CarbonData Operation Guide</a></div>
|
|
</div>
|
|
</div>
|
|
|