forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
33 lines
3.7 KiB
HTML
33 lines
3.7 KiB
HTML
<a name="mrs_01_2024"></a><a name="mrs_01_2024"></a>
|
|
|
|
<h1 class="topictitle1">Why Spark SQL Is Displayed as a Temporary Table in Different Databases?</h1>
|
|
<div id="body1595920220419"><div class="section" id="mrs_01_2024__s0293ed22141c43f199f85cfe77869ecf"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2024__ab467e779d12f459ab82cbc3baabf2c72">Why temporary tables of the previous database are displayed after the database is switched?</p>
|
|
<ol id="mrs_01_2024__o189940c970864ec6b3aff2c7e5bae540"><li id="mrs_01_2024__l6f7263a0dc944062a0e14045927ec172">Create a temporary DataSource table, for example:<pre class="screen" id="mrs_01_2024__sf5d465435971456ab7960e67129557c7">create temporary table ds_parquet
|
|
using org.apache.spark.sql.parquet
|
|
options(path '/tmp/users.parquet');</pre>
|
|
</li><li id="mrs_01_2024__l215e417dc6ca459484d4f470ed0d7b97">Switch to another database, and run <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_2024__c8d9dc9e8d39a4ae0b8521d2743f837fa">show tables</span></b></i>. The temporary table created in the previous table is displayed.<pre class="screen" id="mrs_01_2024__s3e62eff9d9e240309a8d383752b7cdf6">0: jdbc:hive2://192.168.169.84:22550/default> show tables;
|
|
+-----------------+--------------+--+
|
|
| tableName | isTemporary |
|
|
+-----------------+--------------+--+
|
|
| ds_parquet | true |
|
|
| cmb_tbl_carbon | false |
|
|
+-----------------+--------------+--+
|
|
2 rows selected (0.109 seconds)
|
|
0: jdbc:hive2://192.168.169.84:22550/default></pre>
|
|
</li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_2024__sabe13a18872e4dd8ae487e46c6973d04"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_2024__a9d6c58c99bca44a5bf1a272b17df6f8e">The table management hierarchy of Spark is shown in <a href="#mrs_01_2024__fa8229aa6660f4ac789ff2821c92b06a2">Figure 1</a>. The lowest layer stores all temporary DataSource tables. There is no such concept as database at this layer. DataSource tables are visible in various databases.</p>
|
|
<p id="mrs_01_2024__a896259328b7d4eb18502c76a3c167004">The MetaStore of Hive is located at the upper layer. This layer distinguishes among databases. In each database, there are two types of Hive table, permanent and temporary. Therefore, Spark supports data tables of the same name at three layers.</p>
|
|
<p id="mrs_01_2024__ab8967a33538c4286a7d9508492a535bf">During query, SparkSQL first checks for temporary Spark tables, then temporary Hive tables in the current database, and at last the permanent tables in the current database.</p>
|
|
<div class="fignone" id="mrs_01_2024__fa8229aa6660f4ac789ff2821c92b06a2"><a name="mrs_01_2024__fa8229aa6660f4ac789ff2821c92b06a2"></a><a name="fa8229aa6660f4ac789ff2821c92b06a2"></a><span class="figcap"><b>Figure 1 </b>Spark table management hierarchy</span><br><span><img id="mrs_01_2024__i3fe24b63615041baa1955fdfe3f0efbf" src="en-us_image_0000001348771181.png"></span></div>
|
|
<p id="mrs_01_2024__a1268844d77f746b385f5c39bad5da836">When a session quits, temporary tables related to the user operation are automatically deleted. Manual deletion of temporary files is not recommended.</p>
|
|
<p id="mrs_01_2024__ae93e02a2aebe4bbe9a004c6272b999f3">When deleting temporary files, use the same priority as that for query. The priorities are temporary Spark table, temporary Hive table, and permanent Hive table ranging from high to low. If you want to directly delete Hive tables but not temporary Spark tables, you can directly use the <i><b><span class="cmdname" style="font-family:Arial" id="mrs_01_2024__cbe652b063a5747e49191260f1e442ff4">drop table DbName.TableName</span></b></i> command.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2022.html">Spark SQL and DataFrame</a></div>
|
|
</div>
|
|
</div>
|
|
|