mikacur/MRS-operation-guide-lts
1
0
Fork 0
forked from docs/mapreduce-service
Code Pull Requests Activity
MRS-operation-guide-lts/doc/component-operation-guide-lts/source/using_loader/common_loader_parameters.rst
proposalbot d31d5b10fe Changes to mrs_operation-guide-lts from docs/doc-exports#515 (MRS comp-lts 2.0.3 
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com>
Co-authored-by: proposalbot <proposalbot@otc-service.com>
Co-committed-by: proposalbot <proposalbot@otc-service.com>
2023-01-19 16:53:12 +00:00

8.2 KiB
Raw Blame History

original_name

mrs_01_1784.html

Common Loader Parameters

Navigation Path

For details about the how to set parameters, see Modifying Cluster Service Configuration Parameters <mrs_01_1293>.

Parameter Description

Table 1 Common Loader parameters
Parameter Description Default Value Value Range
mapreduce.client.submit.file.replication Number of copies of the job files that the MapReduce task depends on in HDFS. If the number of DataNodes in the cluster is less than the value of this parameter, the number of copies is equal to the number of DataNodes. If the number of DataNodes is greater than or equal to the value of this parameter, the number of copies is the value of this parameter. 10 3 to 256
loader.fault.tolerance.rate

Error tolerance.

If the value is greater than 0, the error tolerance mechanism is enabled. When enabling the fault tolerance mechanism, you are advised to set the number of Map jobs to be greater than or equal to 3. It is recommended that this function be used when the job data volume is large.

 0 0 to 1.0
loader.input.field.separator Default input field separator. The parameter value takes effect only when input and output conversion steps are configured. The conversion steps can be left blank. If no separators are configured in job conversion steps, the default separator is used. , -
loader.input.line.separator Default input line separator. The parameter value takes effect only when input and output conversion steps are configured. The conversion steps can be left blank. If no separators are configured in job conversion steps, the default separator is used. - -
loader.output.field.separator Default output field separator. The parameter value takes effect only when input and output conversion steps are configured. The conversion steps can be left blank. If no separators are configured in job conversion steps, the default separator is used. , -
loader.output.line.separator Line separator of data that Loader outputs - -

Note

  • Because it needs time to calculate the fault tolerance rate, you are recommended to use the loader.fault.tolerance.rate parameter when the job runtime is longer than 2 minutes to ensure user experience.
  • Default separators are configured for the parameters in the preceding table for Loader. If separators are configured in the conversion steps for the jobs, the separators in the conversion steps will be used. If separators are not configured in the conversion steps, the default separators will be used.