vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/mrs/component-operation-guide-lts/mrs_01_2055.html
Yang, Tong 3f5759eed2 MRS comp-lts 2.0.38.SP20 version 
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com>
Co-authored-by: Yang, Tong <yangtong2@huawei.com>
Co-committed-by: Yang, Tong <yangtong2@huawei.com>
2023-01-19 17:08:45 +00:00

3.4 KiB
Raw Permalink Blame History

Why the Job Information Obtained from the restful Interface of an Ended Spark Application Is Incorrect?

Question

The job information obtained from the restful interface of an ended Spark application is incorrect: the value of numActiveTasks is negative, as shown in Figure 1:

Figure 1 job information

numActiveTasks indicates the number of active tasks.

Answer

The job information can be obtained in either of the following methods:
  • Set spark.history.briefInfo.gather=true and then view the brief JobHistory information.
  • Visit the JobHistory2x page of Spark (URL: https://IP:port/api/v1/<appid>/jobs/).

The value of numActiveTasks in the job information is calculated from the difference between the number of SparkListenerTaskStart events and the number of SparkListenerTaskEnd events in the eventLog file. If some events are not recorded in the eventLog file, the job information obtained from the restful interface is incorrect.