forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
184 lines
18 KiB
HTML
184 lines
18 KiB
HTML
<a name="mrs_01_0842"></a><a name="mrs_01_0842"></a>
|
|
|
|
<h1 class="topictitle1">Introduction to MapReduce Logs</h1>
|
|
<div id="body1590130629911"><div class="section" id="mrs_01_0842__saaf675d3b66e47babf6fd8f8a9c8a676"><h4 class="sectiontitle">Log Description</h4><p id="mrs_01_0842__en-us_topic_0046736742_p56908160"><strong id="mrs_01_0842__b1476413185474">Log paths:</strong></p>
|
|
<ul id="mrs_01_0842__en-us_topic_0046736742_ul46158236"><li id="mrs_01_0842__en-us_topic_0046736742_li12770940">JobhistoryServer: <strong id="mrs_01_0842__b16693164124719">/var/log/Bigdata/mapreduce/jobhistory</strong> (run log) and <strong id="mrs_01_0842__b7955958114717">/var/log/Bigdata/audit/mapreduce/jobhistory</strong> (audit log)</li><li id="mrs_01_0842__en-us_topic_0046736742_li47829596">Container: <strong id="mrs_01_0842__b159141226124819">/srv/BigData/hadoop/data1/nm/containerlogs/application_${appid}/container_{$contid}</strong></li></ul>
|
|
<div class="note" id="mrs_01_0842__en-us_topic_0046736742_note27813181"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p class="text" id="mrs_01_0842__en-us_topic_0046736742_p48992040">The logs of running tasks are stored in the preceding paths. After the running is complete, the system determines whether to aggregate the logs to an HDFS directory based on the YARN configuration. For details, see <a href="mrs_01_0852.html">Common YARN Parameters</a>.</p>
|
|
</div></div>
|
|
<p id="mrs_01_0842__en-us_topic_0046736742_p38275183"><strong id="mrs_01_0842__b486665312483">Log archive rule</strong>:</p>
|
|
<p id="mrs_01_0842__en-us_topic_0046736742_p13282105">The automatic compression and archive function is enabled for MapReduce logs. By default, a log file is automatically compressed when the size of the log file is greater than 50 MB. The name of the compressed log file is in the following format: <em id="mrs_01_0842__i4281185023920"><Name of the original log>-<yyyy-mm-dd_hh-mm-ss>.[NO.]</em><strong id="mrs_01_0842__b457335611394">.log.zip</strong>. A maximum of 100 latest compressed files are reserved. The number of compressed files can be configured on the parameter configuration page.</p>
|
|
<p id="mrs_01_0842__en-us_topic_0046736742_p52430087">In MapReduce, JobhistoryServer cleans the old log files stored in HDFS periodically. The default storage directory is <strong id="mrs_01_0842__b12204223115111">/mr-history/done</strong>. <strong id="mrs_01_0842__b8917153019516">mapreduce.jobhistory.max-age-ms</strong> is used to set the cleanup interval. The default value of this parameter is 1,296,000,000 ms, which indicates 15 days.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0842__en-us_topic_0046736742_table2108740" frame="border" border="1" rules="all"><caption><b>Table 1 </b>MapReduce log list</caption><thead align="left"><tr id="mrs_01_0842__en-us_topic_0046736742_row48625339"><th align="left" class="cellrowborder" valign="top" width="27.55%" id="mcps1.3.1.8.2.4.1.1"><p id="mrs_01_0842__en-us_topic_0046736742_p46338367">Type</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="38.78%" id="mcps1.3.1.8.2.4.1.2"><p id="mrs_01_0842__en-us_topic_0046736742_p62420240">Name</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="33.67%" id="mcps1.3.1.8.2.4.1.3"><p id="mrs_01_0842__en-us_topic_0046736742_p22874645">Description</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0842__row19786194123116"><td class="cellrowborder" rowspan="14" valign="top" width="27.55%" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p25127389">Run log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.78%" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__p878674133113">jhs-daemon-start-stop.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.67%" headers="mcps1.3.1.8.2.4.1.3 "><p id="mrs_01_0842__p1578813493114">Startup log file of the daemon process</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__row19295124882612"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__p17297104811269">hadoop-<SSH_USER>-jhshadaemon-<hostname>.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__p62977481266">Run log file of the daemon process</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row40906934"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p22052591">hadoop-<SSH_USER>-<process_name>-<hostname>.out</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p41429462">Log that records the MapReduce running environment information</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row37320845"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p3089614">historyserver-<SSH_USER>-<DATE>-<PID>-gc.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p48932183">Log that records the garbage collection of the MapReduce service</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row37736471"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__p14434129183318">jhs-haCheck.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__p15670123443314">Log that records the active and standby status of MapReduce instances</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row18174391"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p62839538">yarn-start-stop.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p56837822">Log that records the startup and stop of the MapReduce service</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row41778354"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p28603481">yarn-prestart.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p35180645">Log that records cluster operations before the MapReduce service startup</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row48190357"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p11104819">yarn-postinstall.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p27075113">Work log before the MapReduce service startup and after the installation</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row42349429"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p7751715">yarn-cleanup.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p23909182">Log that records the cleanup logs about the uninstallation of the MapReduce service</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row13856047"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p48598040">mapred-service-check.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p44127137">Log that records the health check details of the MapReduce service</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row61599913"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p23537029">container_{$contid}</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p27451222">Container log</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__row133344019912"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__p782871941213">hadoop-<SSH_USER>-<process_name>-<hostname>.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__p43221913119">MR run log</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__row3334101591"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__p485483019525">mapred-switch-jhs.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__p28541830165218">MR active/standby switchover log</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__row123351501598"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__p02361024313">env.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__p172361726317">Environment information log before the instance is started or stopped</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row45734414"><td class="cellrowborder" rowspan="2" valign="top" width="27.55%" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p13500073">Audit log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="38.78%" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p19764141">mapred-audit-jobhistory.log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="33.67%" headers="mcps1.3.1.8.2.4.1.3 "><p id="mrs_01_0842__en-us_topic_0046736742_p57391552">MapReduce operation audit log</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__row062314392214"><td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.1 "><p id="mrs_01_0842__p962473915216">SecurityAuth.audit</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" headers="mcps1.3.1.8.2.4.1.2 "><p id="mrs_01_0842__p762483911212">MapReduce security audit log</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<div class="section" id="mrs_01_0842__section66893546516"><h4 class="sectiontitle">Log Level</h4><p id="mrs_01_0842__en-us_topic_0046736742_p18204173"><a href="#mrs_01_0842__tefeb2aa164fe4238b59439b2b0dacf00">Table 2</a> describes the log levels supported by MapReduce The log levels are FATAL, ERROR, WARN, INFO, and DEBUG from high priority to low. Logs whose levels are higher than or equal to the specified level are printed. The number of printed logs decreases as the specified log level increases. </p>
|
|
|
|
<div class="tablenoborder"><a name="mrs_01_0842__tefeb2aa164fe4238b59439b2b0dacf00"></a><a name="tefeb2aa164fe4238b59439b2b0dacf00"></a><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0842__tefeb2aa164fe4238b59439b2b0dacf00" frame="border" border="1" rules="all"><caption><b>Table 2 </b>Log level</caption><thead align="left"><tr id="mrs_01_0842__en-us_topic_0046736742_row48618163"><th align="left" class="cellrowborder" valign="top" width="35.35%" id="mcps1.3.2.3.2.3.1.1"><p id="mrs_01_0842__en-us_topic_0046736742_p45757162"><strong id="mrs_01_0842__b14773162110596">Level</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="64.64999999999999%" id="mcps1.3.2.3.2.3.1.2"><p id="mrs_01_0842__en-us_topic_0046736742_p3865714"><strong id="mrs_01_0842__b1466822705910">Description</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0842__en-us_topic_0046736742_row66642231"><td class="cellrowborder" valign="top" width="35.35%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p29311631">FATAL</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="64.64999999999999%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p25431924">Logs of this level record critical error information about the current event processing.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row27560726"><td class="cellrowborder" valign="top" width="35.35%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p17826361">ERROR</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="64.64999999999999%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p34649116">Logs of this level record error information about the current event processing.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row43406589"><td class="cellrowborder" valign="top" width="35.35%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p26272803">WARN</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="64.64999999999999%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p47722335">Logs of this level record unexpected alarm information about the current event processing.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row26847831"><td class="cellrowborder" valign="top" width="35.35%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p27190732">INFO</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="64.64999999999999%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p54965706">Logs of this level record normal running status information about the system and events.</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row24929307"><td class="cellrowborder" valign="top" width="35.35%" headers="mcps1.3.2.3.2.3.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p6007992">DEBUG</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="64.64999999999999%" headers="mcps1.3.2.3.2.3.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p16885362">Logs of this level record the system information and system debugging information.</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<p id="mrs_01_0842__en-us_topic_0046736742_p35624820">To modify log levels, perform the following operations:</p>
|
|
<ol id="mrs_01_0842__en-us_topic_0046736742_ol52187928"><li id="mrs_01_0842__li214111617211"><span>Go to the <strong id="mrs_01_0842__b35515215184">All Configurations</strong> page of the MapReduce service. For details, see <a href="mrs_01_2125.html">Modifying Cluster Service Configuration Parameters</a>.</span></li><li id="mrs_01_0842__en-us_topic_0046736742_li15574875"><span>On the left menu bar, select the log menu of the target role.</span></li><li id="mrs_01_0842__en-us_topic_0046736742_li5956154"><span>Select a desired log level.</span></li><li id="mrs_01_0842__en-us_topic_0046736742_li53605390"><span>Save the configuration. In the displayed dialog box, click <strong id="mrs_01_0842__b426515176189">OK</strong> to make the configurations take effect.</span><p><div class="note" id="mrs_01_0842__en-us_topic_0046736742_note12686464"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p class="text" id="mrs_01_0842__en-us_topic_0046736742_p47069313">The configurations take effect immediately without restarting the service.</p>
|
|
</div></div>
|
|
</p></li></ol>
|
|
</div>
|
|
<div class="section" id="mrs_01_0842__sa94e55a53af6466f9b2c9feac3284cb5"><h4 class="sectiontitle">Log Format</h4><p id="mrs_01_0842__en-us_topic_0046736742_p54517980">The following table lists the MapReduce log formats.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_0842__en-us_topic_0046736742_table20899776" frame="border" border="1" rules="all"><caption><b>Table 3 </b>Log format</caption><thead align="left"><tr id="mrs_01_0842__en-us_topic_0046736742_row37313023"><th align="left" class="cellrowborder" valign="top" width="24.490000000000002%" id="mcps1.3.3.3.2.4.1.1"><p id="mrs_01_0842__en-us_topic_0046736742_p2455997"><strong id="mrs_01_0842__b754019441120">Type</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="35.709999999999994%" id="mcps1.3.3.3.2.4.1.2"><p id="mrs_01_0842__en-us_topic_0046736742_p45591407"><strong id="mrs_01_0842__b111723509214">Format</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="39.800000000000004%" id="mcps1.3.3.3.2.4.1.3"><p id="mrs_01_0842__en-us_topic_0046736742_p17248297"><strong id="mrs_01_0842__b673975212218">Example</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_0842__en-us_topic_0046736742_row24651527"><td class="cellrowborder" valign="top" width="24.490000000000002%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p50616659">Run log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="35.709999999999994%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p6308675"><<em id="mrs_01_0842__i032002612513">yyyy-MM-dd HH:mm:ss,SSS</em>>|<<em id="mrs_01_0842__i191709301059">Log level</em>>|<<em id="mrs_01_0842__i18184195615210">Name of the thread that generates the log</em>>|<<em id="mrs_01_0842__i191892562214">Message in the log</em>>|<<em id="mrs_01_0842__i161907563212">Location where the log event occurs</em>></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.800000000000004%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_0842__en-us_topic_0046736742_p41240639">2020-01-26 14:18:59,109 | INFO | main | Client environment:java.compiler=<NA> | org.apache.zookeeper.Environment.logEnv(Environment.java:100)</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_0842__en-us_topic_0046736742_row35621438"><td class="cellrowborder" valign="top" width="24.490000000000002%" headers="mcps1.3.3.3.2.4.1.1 "><p id="mrs_01_0842__en-us_topic_0046736742_p66764216">Audit log</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="35.709999999999994%" headers="mcps1.3.3.3.2.4.1.2 "><p id="mrs_01_0842__en-us_topic_0046736742_p39192435"><<em id="mrs_01_0842__i6582194265">yyyy-MM-dd HH:mm:ss,SSS</em>>|<<em id="mrs_01_0842__i199617366">Log level</em>>|<<em id="mrs_01_0842__i11747387315">Name of the thread that generates the log</em>>|<<em id="mrs_01_0842__i14753185314">Message in the log</em>>|<<em id="mrs_01_0842__i12753108136">Location where the log event occurs</em>></p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="39.800000000000004%" headers="mcps1.3.3.3.2.4.1.3 "><p id="mrs_01_0842__en-us_topic_0046736742_p20470679">2020-01-26 14:24:43,605 | INFO | main-EventThread | USER=omm OPERATION=refreshAdminAcls TARGET=AdminService RESULT=SUCCESS | org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger$LogLevel$6.printLog(RMAuditLogger.java:91)</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_0834.html">Using MapReduce</a></div>
|
|
</div>
|
|
</div>
|
|
|