Save the partition information about the datasource table to the Metastore and process partition information in the Metastore.
You need only to process data corresponding to partCol=1 when performing the TableScan operation in the physical plan.
Parameter |
Description |
Default Value |
---|---|---|
spark.sql.hive.manageFilesourcePartitions |
Specifies whether to enable Metastore partition management (including datasource tables and converted Hive).
|
true |
spark.sql.hive.metastorePartitionPruning |
Specifies whether to support pushing down predicate to Hive Metastore.
|
true |
spark.sql.hive.filesourcePartitionFileCacheSize |
The cache size of the partition file metadata in the memory. All tables share a cache that can use up to specified num bytes for file metadata. This parameter is valid only when spark.sql.hive.manageFilesourcePartitions is set to true. |
250 * 1024 * 1024 |
spark.sql.hive.convertMetastoreOrc |
The processing approach of ORC tables.
|
true |