mikacur/MRS-operation-guide-lts
1
0
Fork 0
forked from docs/mapreduce-service
Code Pull Requests Activity
MRS-operation-guide-lts/doc/component-operation-guide/source/using_flink/flink_log_overview.rst
proposalbot 3ccfbdc0f0 Changes to mrs_operation-guide from docs/doc-exports#475 (MRS component operatio 
Reviewed-by: Kacur, Michal <michal.kacur@t-systems.com>
Co-authored-by: proposalbot <proposalbot@otc-service.com>
Co-committed-by: proposalbot <proposalbot@otc-service.com>
2022-12-09 14:50:38 +00:00

6.1 KiB
Raw Blame History

original_name

mrs_01_0596.html

Flink Log Overview

Log Description

Log path:

  • Run logs of a Flink job: ${BIGDATA_DATA_HOME}/hadoop/data${i}/nm/containerlogs/application_${appid}/container_{$contid}

    Note

    The logs of executing tasks are stored in the preceding path. After the execution is complete, the Yarn configuration determines whether these logs are gathered to the HDFS directory.

  • FlinkResource run logs: /var/log/Bigdata/flink/flinkResource

Log archive rules:

  1. FlinkResource run logs:

    • By default, service logs are backed up each time when the log size reaches 20 MB. A maximum of 20 logs can be reserved without being compressed.

      Note

      For versions earlier than MRS 3.x, The executor logs are backed up each time when the log size reaches 30 MB. A maximum of 20 logs can be reserved without being compressed.

    • You can set the log size and number of compressed logs on the Manager page or modify the corresponding configuration items in log4j-cli.properties, log4j.properties, and log4j-session.properties in /opt/client/Flink/flink/conf/ on the client. /opt/client is the client installation directory.

    Table 1 FlinkResource log list
    Type Name Description
    FlinkResource run logs checkService.log Health check log
    kinit.log Initialization log
    postinstall.log Service installation log
    prestart.log Prestart script log
    start.log Startup log

Log Level

Table 2 <mrs_01_0596__table63318572917> describes the log levels supported by Flink. The priorities of log levels are ERROR, WARN, INFO, and DEBUG in descending order. Logs whose levels are higher than or equal to the specified level are printed. The number of printed logs decreases as the specified log level increases.

Table 2 Log levels
Level Description
ERROR Error information about the current event processing
WARN Exception information about the current event processing
INFO Normal running status information about the system and events
DEBUG System information and system debugging information

To modify log levels, perform the following steps:

  1. Go to the All Configurations page of Flink by referring to Modifying Cluster Service Configuration Parameters <mrs_01_2125>.
  2. On the menu bar on the left, select the log menu of the target role.
  3. Select a desired log level.
  4. Save the configuration. In the displayed dialog box, click OK to make the configurations take effect.

Note

  • After the configuration is complete, you do not need to restart the service. Download the client again for the configuration to take effect.
  • You can also change the configuration items corresponding to the log level in log4j-cli.properties, log4j.properties, and log4j-session.properties in /opt/client/Flink/flink/conf/ on the client. /opt/client is the client installation directory.
  • When a job is submitted using a client, a log file is generated in the log folder on the client. The default umask value is 0022. Therefore, the default log permission is 644. To change the file permission, you need to change the umask value. For example, to change the umask value of user omm:
    • Add umask 0026 to the end of the /home/omm/.baskrc file.
    • Run the source /home/omm/.baskrc command to make the file permission take effect.

Log Format

Table 3 Log formats
Type Format Example
Run log <yyyy-MM-dd HH:mm:ss,SSS><Name of the thread that generates the log><Location where the log event occurs> 2019-06-27 21:30:31,778 | INFO | [flink-akka.actor.default-dispatcher-3] | TaskManager container_e10_1498290698388_0004_02_000007 has started. | org.apache.flink.yarn.YarnFlinkResourceManager (FlinkResourceManager.java:368)