vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_2025.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

1.5 KiB
Raw Permalink Blame History

How to Assign a Parameter Value in a Spark Command?

Question

Is it possible to assign parameter values through Spark commands, in addition to through a user interface or a configuration file?

Answer

Spark configuration options can be defined either in a configuration file or in Spark commands.

To assign a parameter value, run the --conf command on a Spark client. The parameter value takes effect immediately after the command is run.

The command format is --conf + parameter name + parameter value. Example command:

--conf spark.eventQueue.size=50000

Parent topic: Spark SQL and DataFrame