Scenarios
When using a cluster, if you want to switch between multi-active instance mode and multi-tenant mode, the following configurations are required.
- Switch from multi-tenant mode to multi-active instance mode.
Modify the following parameters of the Spark2x service:
- spark.thriftserver.proxy.enabled=false
- spark.scheduler.allocation.file=#{conf_dir}/fairscheduler.xml
- spark.proxyserver.hash.enabled=false
- Switch from multi-active instance mode to multi-tenant mode.
Modify the following parameters of the Spark2x service:
- spark.thriftserver.proxy.enabled=true
- spark.scheduler.allocation.file=./__spark_conf__/__hadoop_conf__/fairscheduler.xml
- spark.proxyserver.hash.enabled=true
Configuration Description
Log in to Manager, choose Cluster > Name of the desired cluster > Service > Spark2x > Configuration, click All Configurations, and search for and modify the following parameters.
Table 1 Parameter descriptionParameter
|
Description
|
Default Value
|
|---|
spark.thriftserver.proxy.enabled
|
Specifies whether to use the multi-tenant mode.
- false: The multi-instance mode is used.
- true: The multi-tenant mode is used.
|
true
|
spark.scheduler.allocation.file
|
Specifies the fair scheduling file path.
- If the multi-active instance mode is used, the path is changed to #{conf_dir}/fairscheduler.xml.
- If multi-tenant mode is used, the path is changed to ./__spark_conf__/__hadoop_conf__/fairscheduler.xml.
|
./__spark_conf__/__hadoop_conf__/fairscheduler.xml
|
spark.proxyserver.hash.enabled
|
Specifies whether to connect to ProxyServer using the Hash algorithm.
- true indicates using the Hash algorithm. In multi-tenant mode, this parameter must be configured to true.
- false indicates using random connection. In multi-active instance mode, this parameter must be configured to false.
|
true
NOTE: After this parameter is modified, you need to download the client again.
|