forked from docs/doc-exports
Su, Xiaomeng
04d4597cf3
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com> Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com> Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
101 lines
17 KiB
HTML
101 lines
17 KiB
HTML
<a name="dli_08_0059"></a><a name="dli_08_0059"></a>
|
|
|
|
<h1 class="topictitle1">Complex Data Types</h1>
|
|
<div id="body8662426"><p id="dli_08_0059__en-us_topic_0093946794_aa6ab77a46cef4e5a90a7c8db3105f876">Spark SQL supports complex data types, as shown in <a href="#dli_08_0059__en-us_topic_0093946794_tc03311896fc248a4883a320378f84a7a">Table 1</a>.</p>
|
|
|
|
<div class="tablenoborder"><a name="dli_08_0059__en-us_topic_0093946794_tc03311896fc248a4883a320378f84a7a"></a><a name="en-us_topic_0093946794_tc03311896fc248a4883a320378f84a7a"></a><table cellpadding="4" cellspacing="0" summary="" id="dli_08_0059__en-us_topic_0093946794_tc03311896fc248a4883a320378f84a7a" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Complex data types</caption><thead align="left"><tr id="dli_08_0059__en-us_topic_0093946794_rfafa3c50eba24deaa609f9fe8594a1ff"><th align="left" class="cellrowborder" valign="top" width="11.78%" id="mcps1.3.2.2.4.1.1"><p id="dli_08_0059__en-us_topic_0093946794_a813700cd46844dcba5585d1e4b30ae48">Data Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="50.91%" id="mcps1.3.2.2.4.1.2"><p id="dli_08_0059__en-us_topic_0093946794_ad2f95d53e6e741baaaf744827c813403">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="37.31%" id="mcps1.3.2.2.4.1.3"><p id="dli_08_0059__p19489551132711">Syntax</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="dli_08_0059__en-us_topic_0093946794_r3094c19a573b4675ae7f78b9ef649e20"><td class="cellrowborder" valign="top" width="11.78%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0059__en-us_topic_0093946794_a02354fe2fc564b21bee49e133029b671">ARRAY</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.91%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0059__en-us_topic_0093946794_a1beda7386b894de4bda444252c2bd342">A set of ordered fields that construct an ARRAY with the specified values. The value can be of any type and the data type of all fields must be the same.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.31%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0059__p19489145162719">array(<value>,<value>[, ...])</p>
|
|
<p id="dli_08_0059__p35281142356">For details, see <a href="#dli_08_0059__en-us_topic_0093946794_sdba49492924046f2a1ffb33968e73073">Example of ARRAY</a>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0059__en-us_topic_0093946794_r179c223215ca4e68afad7bae42cd078a"><td class="cellrowborder" valign="top" width="11.78%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0059__en-us_topic_0093946794_a6e1fcccbccd343cc9a29e3b7bf3b5578">MAP</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.91%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0059__en-us_topic_0093946794_a7b85f5d1f72c4517a5ba8db213735d3f">A group of unordered key/value pairs used to generate a MAP. The key must be native data type, but the value can be either native data type or complex data type. The type of the same MAP key, as well as the MAP value, must be the same.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.31%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0059__p64895519272">map(K <key1>, V <value1>, K <key2>, V <value2>[, ...])</p>
|
|
<p id="dli_08_0059__p11655242153516">For details, see <a href="#dli_08_0059__en-us_topic_0093946794_s2e73f4a839c94069b7811eac162dd4f5">Example of Map</a>.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="dli_08_0059__en-us_topic_0093946794_r0170aa7d2e6645b8959a8924682a90f0"><td class="cellrowborder" valign="top" width="11.78%" headers="mcps1.3.2.2.4.1.1 "><p id="dli_08_0059__en-us_topic_0093946794_a65eb49b96203461a9a0e9f58a44e19bb">STRUCT</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="50.91%" headers="mcps1.3.2.2.4.1.2 "><p id="dli_08_0059__en-us_topic_0093946794_a90fcb70c4b6141808319d6db6a176593">Indicates a group of named fields. The data types of the fields can be different.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="37.31%" headers="mcps1.3.2.2.4.1.3 "><p id="dli_08_0059__p14490195182710">struct(<value1>,<value2>[, ...])</p>
|
|
<p id="dli_08_0059__p615674510356">For details, see <a href="#dli_08_0059__en-us_topic_0093946794_s955554408c884d2b85ddab9a3b86c6a4">Example of STRUCT</a>.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="section" id="dli_08_0059__section1856663211199"><h4 class="sectiontitle">Restrictions</h4><ul id="dli_08_0059__ul12778154518190"><li id="dli_08_0059__li510192916214">When a table containing fields of the complex data type is created, the storage format of this table cannot be CSV (txt).</li><li id="dli_08_0059__li3205175911190">If a table contains fields of the complex data type, data in CSV (txt) files cannot be imported to the table.</li><li id="dli_08_0059__li2099305013204">When creating a table of the MAP data type, you must specify the schema and do not support the <strong id="dli_08_0059__b4530113031712">date</strong>, <strong id="dli_08_0059__b5531183016174">short</strong>, and <strong id="dli_08_0059__b11531103018179">timestamp</strong> data types.</li><li id="dli_08_0059__li10433438104912">For the OBS table in JSON format, the key type of the MAP supports only the STRING type.</li><li id="dli_08_0059__li9778184519199">The key of the MAP type cannot be <strong id="dli_08_0059__b51156214174">NULL</strong>. Therefore, the MAP key does not support implicit conversion between inserted data formats where NULL values are allowed. For example, the STRING type cannot be converted to other native types, the FLOAT type cannot be converted to the TIMESTAMP type, and other native types cannot be converted to the DECIMAL type.</li><li id="dli_08_0059__li13695114010229">Values of the <strong id="dli_08_0059__b1166122401718">double</strong> or <strong id="dli_08_0059__b21663242179">boolean</strong> data type cannot be included in the <strong id="dli_08_0059__b141671024161712">STRUCT</strong> data type does not support the.</li></ul>
|
|
</div>
|
|
<div class="section" id="dli_08_0059__en-us_topic_0093946794_sdba49492924046f2a1ffb33968e73073"><a name="dli_08_0059__en-us_topic_0093946794_sdba49492924046f2a1ffb33968e73073"></a><a name="en-us_topic_0093946794_sdba49492924046f2a1ffb33968e73073"></a><h4 class="sectiontitle">Example of ARRAY</h4><p id="dli_08_0059__en-us_topic_0093946794_a9be9ddf77b48456aa936b945e6e03134">Create an <span class="parmname" id="dli_08_0059__parmname2421172292515"><b>array_test</b></span> table, set <span class="parmname" id="dli_08_0059__parmname194724382513"><b>id</b></span> to <span class="parmname" id="dli_08_0059__parmname33811454192518"><b>ARRAY<INT></b></span>, and <span class="parmname" id="dli_08_0059__parmname45133412268"><b>name</b></span> to <span class="parmname" id="dli_08_0059__parmname1742916103261"><b>STRING</b></span>. After the table is created, insert test data into <strong id="dli_08_0059__b998932713585">array_test</strong>. The procedure is as follows:</p>
|
|
<ol id="dli_08_0059__en-us_topic_0093946794_o4b293062ec574316bae836234d47c762"><li id="dli_08_0059__en-us_topic_0093946794_l6b26fc559e6149e8b5283b6c24ab3066">Create a table.<p id="dli_08_0059__en-us_topic_0093946794_a7bd67eb503a4482d8721dc8ded93ff24"><a name="dli_08_0059__en-us_topic_0093946794_l6b26fc559e6149e8b5283b6c24ab3066"></a><a name="en-us_topic_0093946794_l6b26fc559e6149e8b5283b6c24ab3066"></a><strong id="dli_08_0059__b649124734312">CREATE TABLE array_test(name STRING, id ARRAY < INT >) USING PARQUET;</strong></p>
|
|
</li><li id="dli_08_0059__en-us_topic_0093946794_lfb85cb1e1fda465abe9f591105866248">Run the following statements to insert test data: <p id="dli_08_0059__p1022015391461"><a name="dli_08_0059__en-us_topic_0093946794_lfb85cb1e1fda465abe9f591105866248"></a><a name="en-us_topic_0093946794_lfb85cb1e1fda465abe9f591105866248"></a><strong id="dli_08_0059__b9681144774610">INSERT INTO array_test VALUES ('test',array(1,2,3,4));</strong></p>
|
|
<p id="dli_08_0059__p10220113912463"><strong id="dli_08_0059__b8693114784615">INSERT INTO array_test VALUES ('test2',array(4,5,6,7))</strong></p>
|
|
<p id="dli_08_0059__p2220193912461"><strong id="dli_08_0059__b177071547134620">INSERT INTO array_test VALUES ('test3',array(7,8,9,0));</strong></p>
|
|
</li><li id="dli_08_0059__en-us_topic_0093946794_l9b5fe43fa54e449f9ed15a996daddc04">Query the result.<p id="dli_08_0059__en-us_topic_0093946794_a15990154569b402fb3d163937cf038ab"><a name="dli_08_0059__en-us_topic_0093946794_l9b5fe43fa54e449f9ed15a996daddc04"></a><a name="en-us_topic_0093946794_l9b5fe43fa54e449f9ed15a996daddc04"></a>To query all data in the <span class="parmname" id="dli_08_0059__parmname178991227174114"><b>array_test</b></span> table, run the following statement:</p>
|
|
<p id="dli_08_0059__p396054034715"><strong id="dli_08_0059__b12957151344810">SELECT * FROM</strong> <strong id="dli_08_0059__b1344213924818">array_test</strong>;</p>
|
|
<pre class="screen" id="dli_08_0059__en-us_topic_0093946794_s9d0c37c902974e00855ba891c9281b03">test3 [7,8,9,0]
|
|
test2 [4,5,6,7]
|
|
test [1,2,3,4]</pre>
|
|
<p id="dli_08_0059__en-us_topic_0093946794_ae9ba623c95814cedbed7ef8da71eebe4">To query the data of element <strong id="dli_08_0059__b4754418194214">0</strong> in the <strong id="dli_08_0059__b2357175514115">id</strong> array in the <span class="parmname" id="dli_08_0059__parmname11118784218"><b>array_test</b></span> table, run the following statement:</p>
|
|
<p id="dli_08_0059__p76661830204813"><strong id="dli_08_0059__b3666153014481">SELECT id[0] FROM</strong> <strong id="dli_08_0059__b1566793017484">array_test</strong>;</p>
|
|
<pre class="screen" id="dli_08_0059__en-us_topic_0093946794_sceab7d8a3adf486ba4a0990b7b9e4688">7
|
|
4
|
|
1</pre>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_08_0059__en-us_topic_0093946794_s2e73f4a839c94069b7811eac162dd4f5"><a name="dli_08_0059__en-us_topic_0093946794_s2e73f4a839c94069b7811eac162dd4f5"></a><a name="en-us_topic_0093946794_s2e73f4a839c94069b7811eac162dd4f5"></a><h4 class="sectiontitle">Example of Map</h4><p id="dli_08_0059__en-us_topic_0093946794_a6ce5c74e81594c31a2059232fe2a6840">Create the <strong id="dli_08_0059__b765211811012">map_test</strong> table and set <strong id="dli_08_0059__b1848642117012">score</strong> to <strong id="dli_08_0059__b938719241809">map<STRING,INT></strong>. The key is of the <strong id="dli_08_0059__b1117213452006">STRING</strong> type and the value is of the <strong id="dli_08_0059__b103633432008">INT</strong> type. After the table is created, insert test data to <strong id="dli_08_0059__b17965174219">map_test</strong>. The procedure is as follows:</p>
|
|
<ol id="dli_08_0059__en-us_topic_0093946794_o12cd23f45cde4d299d39c2ec39e7a972"><li id="dli_08_0059__en-us_topic_0093946794_l43bd87b71a084c33939fbb5460320444">Create a table.<p id="dli_08_0059__p250011481532"><a name="dli_08_0059__en-us_topic_0093946794_l43bd87b71a084c33939fbb5460320444"></a><a name="en-us_topic_0093946794_l43bd87b71a084c33939fbb5460320444"></a><strong id="dli_08_0059__b12627521735">CREATE TABLE map_test(id STRING, score map<STRING,INT>) USING PARQUET;</strong></p>
|
|
</li><li id="dli_08_0059__en-us_topic_0093946794_l47f128763f484054bd0cb86c58f8621d">Run the following statements to insert test data: <p id="dli_08_0059__p115131858495"><a name="dli_08_0059__en-us_topic_0093946794_l47f128763f484054bd0cb86c58f8621d"></a><a name="en-us_topic_0093946794_l47f128763f484054bd0cb86c58f8621d"></a><strong id="dli_08_0059__b6254915109">INSERT INTO map_test VALUES ('test4',map('math',70,'chemistry',84));</strong></p>
|
|
<p id="dli_08_0059__p75138581694"><strong id="dli_08_0059__b122651010106">INSERT INTO map_test VALUES ('test5',map('math',85,'chemistry',97));</strong></p>
|
|
<p id="dli_08_0059__p1651320581916"><strong id="dli_08_0059__b11279191171010">INSERT INTO map_test VALUES ('test6',map('math',88,'chemistry',80));</strong></p>
|
|
</li><li id="dli_08_0059__en-us_topic_0093946794_le9a16e5655084fe8ad045e456e466174">Query the result.<p id="dli_08_0059__en-us_topic_0093946794_a751d7e2fd8bd4b8a87855c34e2a6a222"><a name="dli_08_0059__en-us_topic_0093946794_le9a16e5655084fe8ad045e456e466174"></a><a name="en-us_topic_0093946794_le9a16e5655084fe8ad045e456e466174"></a>To query all data in the <span class="parmname" id="dli_08_0059__parmname1153015618241"><b>map_test</b></span> table, run the following statement:</p>
|
|
<p id="dli_08_0059__p168231501511"><strong id="dli_08_0059__b1945664158">SELECT * FROM map_test;</strong></p>
|
|
<pre class="screen" id="dli_08_0059__en-us_topic_0093946794_s0b675d86447e450bbe78d5d02078d5e1">test6 {"chemistry":80,"math":88}
|
|
test5 {"chemistry":97,"math":85}
|
|
test4 {"chemistry":84,"math":70}</pre>
|
|
<p id="dli_08_0059__en-us_topic_0093946794_a903a91e9d58e4de191d7cce603b2a39d">To query the math score in the <span class="parmname" id="dli_08_0059__parmname194311948114615"><b>map_test</b></span> table, run the following statement:</p>
|
|
<p id="dli_08_0059__en-us_topic_0093946794_a58a380f44fda4bacb922a469334c40a6"><strong id="dli_08_0059__b13480120131111">SELECT id, score['Math'] FROM map_test;</strong></p>
|
|
<pre class="screen" id="dli_08_0059__screen09061757111015">test6 88
|
|
test5 85
|
|
test4 70</pre>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="dli_08_0059__en-us_topic_0093946794_s955554408c884d2b85ddab9a3b86c6a4"><a name="dli_08_0059__en-us_topic_0093946794_s955554408c884d2b85ddab9a3b86c6a4"></a><a name="en-us_topic_0093946794_s955554408c884d2b85ddab9a3b86c6a4"></a><h4 class="sectiontitle">Example of STRUCT</h4><p id="dli_08_0059__en-us_topic_0093946794_a1eb61903facf4695b6907e8eb7d95a11">Create a <span class="parmname" id="dli_08_0059__parmname071162704712"><b>struct_test</b></span> table and set <strong id="dli_08_0059__b939723864710">info</strong> to the <span class="parmname" id="dli_08_0059__parmname65637535478"><b>STRUCT<name:STRING, age:INT></b></span> data type (the field consists of <strong id="dli_08_0059__b220664134815">name</strong> and <strong id="dli_08_0059__b103340110481">age</strong>, where the type of <strong id="dli_08_0059__b99001120104817">name</strong> is <strong id="dli_08_0059__b1473195714820">STRING</strong> and <strong id="dli_08_0059__b187379817491">age</strong> is <strong id="dli_08_0059__b20518286491">INT</strong>). After the table is created, insert test data into the <strong id="dli_08_0059__b1388311474119">struct_test</strong> table. The procedure is as follows:</p>
|
|
<ol id="dli_08_0059__en-us_topic_0093946794_o88331d1379c14cdb8756cfb488b00d1c"><li id="dli_08_0059__en-us_topic_0093946794_lf57fd650b18e4249914f2f607a16f590">Create a table.<p id="dli_08_0059__p1648822731918"><a name="dli_08_0059__en-us_topic_0093946794_lf57fd650b18e4249914f2f607a16f590"></a><a name="en-us_topic_0093946794_lf57fd650b18e4249914f2f607a16f590"></a><strong id="dli_08_0059__b32064361917">CREATE TABLE struct_test(id INT, info STRUCT<name:STRING,age:INT>) USING PARQUET;</strong></p>
|
|
</li><li id="dli_08_0059__en-us_topic_0093946794_ld7653ffc401f4844aeb463f48d77f970">Run the following statements to insert test data: <p id="dli_08_0059__p1431116479218"><a name="dli_08_0059__en-us_topic_0093946794_ld7653ffc401f4844aeb463f48d77f970"></a><a name="en-us_topic_0093946794_ld7653ffc401f4844aeb463f48d77f970"></a><strong id="dli_08_0059__b79851549182118">INSERT INTO struct_test VALUES (8, struct('zhang',23));</strong></p>
|
|
<p id="dli_08_0059__p931184742116"><strong id="dli_08_0059__b299612493213">INSERT INTO struct_test VALUES (9, struct('li',25));</strong></p>
|
|
<p id="dli_08_0059__p63111147152110"><strong id="dli_08_0059__b810155011214">INSERT INTO struct_test VALUES (10, struct('wang',26));</strong></p>
|
|
</li><li id="dli_08_0059__en-us_topic_0093946794_l6cf9d8ad6d3a4e66b8217f474bc47142">Query the result.<p id="dli_08_0059__en-us_topic_0093946794_a4712e5b5ccb4480ca70e08a59e7f6225"><a name="dli_08_0059__en-us_topic_0093946794_l6cf9d8ad6d3a4e66b8217f474bc47142"></a><a name="en-us_topic_0093946794_l6cf9d8ad6d3a4e66b8217f474bc47142"></a>To query all data in the <span class="parmname" id="dli_08_0059__parmname199711709512"><b>struct_test</b></span> table, run the following statement:</p>
|
|
<p id="dli_08_0059__p8824520112215"><strong id="dli_08_0059__b7828112342214">SELECT * FROM struct_test;</strong></p>
|
|
<pre class="screen" id="dli_08_0059__en-us_topic_0093946794_s9e738f73a92949f2a8b24cdbf63dcb8b">8 {"name":"zhang","age":23}
|
|
10 {"name":"wang","age":26}
|
|
9 {"name":"li","age":25}</pre>
|
|
<p id="dli_08_0059__en-us_topic_0093946794_a1391de198a6b472fa066524394beae1f">Query <strong id="dli_08_0059__b4559105620112">name</strong> and <strong id="dli_08_0059__b9262458201111">age</strong> in the <strong id="dli_08_0059__b250015126126">struct_test</strong> table.</p>
|
|
<p id="dli_08_0059__p8896134610236"><strong id="dli_08_0059__b1473617508234">SELECT id,info.name,info.age FROM struct_test;</strong></p>
|
|
<pre class="screen" id="dli_08_0059__en-us_topic_0093946794_s38f94f8bdb3249439f22a5d2f279cc1c">8 zhang 23
|
|
10 wang 26
|
|
9 li 25</pre>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="dli_08_0056.html">Data Types</a></div>
|
|
</div>
|
|
</div>
|
|
|