forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
60 lines
6.7 KiB
HTML
60 lines
6.7 KiB
HTML
<a name="mrs_01_24170"></a><a name="mrs_01_24170"></a>
|
|
|
|
<h1 class="topictitle1">Configuring Event Log Rollover</h1>
|
|
<div id="body0000001166803143"><div class="section" id="mrs_01_24170__s2ac2b803845a424da3af09bb2d7cec7e"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_24170__ae1e8268c7d2e48a28968f6942b83f358">When the event log mode is enabled for Spark, that is, <span class="parmname" id="mrs_01_24170__pdff01c0645764d5c99d86eaf11af1c00"><b>spark.eventLog.enabled</b></span> is set to <span class="parmvalue" id="mrs_01_24170__p29033e87cfc2400d841c2a5136c938a1"><b>true</b></span>, events are written to a configured log file to record the program running process. If a program, for example JDBCServer or Spark Streaming, runs for a long period of time and has run many jobs and tasks during this period, many events are recorded in the log file, significantly increasing the file size.</p>
|
|
<p id="mrs_01_24170__a2ae6e6f2ce7c48939d2b0a75f803ff68">When log rollover is enabled, metadata events are written into the log file and job events are written into a new log file (whether a job event is written to the new log file depends on the file size). Metadata events include EnviromentUpdate, BlockManagerAdded, BlockManagerRemoved, UnpersistRDD, ExecutorAdded, ExecutorRemoved, MetricsUpdate, ApplicationStart, ApplicationEnd, and LogStart. Job events include StageSubmitted, StageCompleted, TaskResubmit, TaskStart, TaskEnd, TaskGettingResult, JobStart, and JobEnd. For Spark SQL applications, job events also include ExecutionStart and ExecutionEnd.</p>
|
|
<p id="mrs_01_24170__a20a1bd03686c4ac09beb36f131e2a48f">The UI for the HistoryServer service of Spark is obtained by reading and parsing these log files. The memory size is preset before the HistoryServer process starts. Therefore, when the size of log files is large, loading and parsing these files may cause problems such as insufficient memory and driver GC.</p>
|
|
<p id="mrs_01_24170__ab71e1971e75a42f5828492668f7f59a0">To load large log files in small memory mode, you need to enable log rollover for large applications. Generally, it is recommended that this function be enabled for long-running applications.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_24170__section149351351145714"><h4 class="sectiontitle">Parameters</h4><p id="mrs_01_24170__p18420142117236">Log in to FusionInsight Manager, choose <span id="mrs_01_24170__text1476246141114215"><strong id="mrs_01_24170__b1633203011114215">Cluster</strong> > </span><strong id="mrs_01_24170__b511376378114215">Services</strong> > <strong id="mrs_01_24170__b1122669756114215">Spark2x</strong> > <strong id="mrs_01_24170__b2029973710114215">Configurations</strong>, click <strong id="mrs_01_24170__b1545111097114215">All Configurations</strong>, and search for the following parameters.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_24170__t18af5fbd3aeb4bda8d13de81ed31f812" frame="border" border="1" rules="all"><thead align="left"><tr id="mrs_01_24170__r74395b638f4442e3aab246ff89b4f414"><th align="left" class="cellrowborder" valign="top" width="32.129999999999995%" id="mcps1.3.2.3.1.4.1.1"><p id="mrs_01_24170__a1f516e63d9e642069c06eb1b4f809563"><strong id="mrs_01_24170__a744ab93a07754850a5f6c727a33e62e1">Parameter</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="49.5%" id="mcps1.3.2.3.1.4.1.2"><p id="mrs_01_24170__af9566f38e89e419f83b61605b532f7ad"><strong id="mrs_01_24170__a6645dddca1604ff3a7c65c302142e2da">Description</strong></p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="18.37%" id="mcps1.3.2.3.1.4.1.3"><p id="mrs_01_24170__ac817ca09e627493cb99f5b57d2943bd6"><strong id="mrs_01_24170__adff60dc7bdc24a8aaa6c1b5c6630331e">Default Value</strong></p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_24170__rd52556c4569841b3a5e5a2ebdfcb3b56"><td class="cellrowborder" valign="top" width="32.129999999999995%" headers="mcps1.3.2.3.1.4.1.1 "><p id="mrs_01_24170__p44616111023">spark.eventLog.rolling.enabled</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.5%" headers="mcps1.3.2.3.1.4.1.2 "><p id="mrs_01_24170__p1148143481514">Whether to enable rollover for event log files. If this parameter is set to <strong id="mrs_01_24170__b168154684514">true</strong>, the size of each event log file is reduced to the configured size.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.2.3.1.4.1.3 "><p id="mrs_01_24170__p10461011426">true</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24170__row782249193512"><td class="cellrowborder" valign="top" width="32.129999999999995%" headers="mcps1.3.2.3.1.4.1.1 "><p id="mrs_01_24170__p52001311164">spark.eventLog.rolling.maxFileSize</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.5%" headers="mcps1.3.2.3.1.4.1.2 "><p id="mrs_01_24170__p6831249103510">Maximum size of the event log file to be rolled over when <strong id="mrs_01_24170__b75726519467">spark.eventlog.rolling.enabled</strong> is set to <strong id="mrs_01_24170__b1864725916460">true</strong>.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.2.3.1.4.1.3 "><p id="mrs_01_24170__p15832497350">128M</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24170__rf534b3c540af486495ea96f84d00a611"><td class="cellrowborder" valign="top" width="32.129999999999995%" headers="mcps1.3.2.3.1.4.1.1 "><p id="mrs_01_24170__p12461611923">spark.eventLog.compression.codec</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.5%" headers="mcps1.3.2.3.1.4.1.2 "><p id="mrs_01_24170__p2302328214">Codec used to compress event logs. By default, Spark provides four types of codecs: LZ4, LZF, Snappy, and ZSTD. If this parameter is not specified, <strong id="mrs_01_24170__b17465019516">spark.io.compression.codec</strong> is used.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.2.3.1.4.1.3 "><p id="mrs_01_24170__p4467111127">None</p>
|
|
</td>
|
|
</tr>
|
|
<tr id="mrs_01_24170__row41031841117"><td class="cellrowborder" valign="top" width="32.129999999999995%" headers="mcps1.3.2.3.1.4.1.1 "><p id="mrs_01_24170__p8522173220184">spark.eventLog.logStageExecutorMetrics</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="49.5%" headers="mcps1.3.2.3.1.4.1.2 "><p id="mrs_01_24170__p1946151113217">Whether to write each stage peak value (for each executor) of executor metrics to the event log.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="18.37%" headers="mcps1.3.2.3.1.4.1.3 "><p id="mrs_01_24170__p1685717911125">false</p>
|
|
<p id="mrs_01_24170__p84341337257"></p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
</div>
|
|
<p id="mrs_01_24170__p8060118"></p>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|