Viewing Training Job Logs

On the training job details page, you can preview logs, download logs, search for logs by keyword, and filter system logs in the log pane.

• Previewing logs

  You can preview logs of each compute node, if multiple compute nodes are used, in the training log pane by choosing the target node from the drop-down list on the right.

  If a log file is oversized, the system displays only the latest logs in the log pane. To view all logs, click the link in the upper part of the log pane, which will direct you to a new page.

  Figure 1 Viewing all logs
  
  

  • If the total size of all logs exceeds 500 MB, the log page may be frozen. In this case, download the logs to view them locally.
  • A log preview link can be accessed by anyone within one hour after it is generated. You can share the link with others.

    Ensure that no privacy information is contained in the logs. Otherwise, information leakage may occur.

• Downloading logs

  Training logs are retained for only 30 days. To permanently store logs, click the download icon in the upper right corner of the log pane. You can download the logs of multiple compute nodes in a batch. You can also enable Persistent Log Saving and set a log path when you create a training job. In this way, the logs will be automatically stored in the specified OBS path.

  If a training job is created on an Ascend compute node, certain system logs cannot be downloaded in the training log pane. To obtain these logs, go to the Job Log Path you set when you created the training job.

  Figure 2 Downloading logs
  
• Searching for logs by keyword

  In the upper right corner of the log pane, enter a keyword in the search box to search for logs, as shown in Figure 3.

  Figure 3 Searching for logs by keyword
  

  The system will highlight the keyword and redirect you between search results. Only the logs loaded in the log pane can be searched for. If the logs are not fully displayed (see the message displayed on the page), obtain all the logs by downloading them or clicking the full log link and then search for the logs. On the page redirected by the full log link, press Ctrl+F to search for logs.

• Filtering system logs
  Figure 4 System logs
  

  If System logs is selected, system logs and user logs are displayed. If System logs is deselected, only user logs are displayed.