forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
46 lines
5.2 KiB
HTML
46 lines
5.2 KiB
HTML
<a name="mrs_01_1945"></a><a name="mrs_01_1945"></a>
|
|
|
|
<h1 class="topictitle1">Configuring the Size of the Event Queue</h1>
|
|
<div id="body1595920206248"><div class="section" id="mrs_01_1945__s7574bad2b00c4d24b0612c8a25e079c5"><h4 class="sectiontitle">Scenarios</h4><p id="mrs_01_1945__a4dae9a2b199f4eb2bfd261df75a5b9bc">Functions such as UI, EventLog, and dynamic resource scheduling in Spark are implemented through event transfer. Events include SparkListenerJobStart and SparkListenerJobEnd, which record each important process.</p>
|
|
<p id="mrs_01_1945__ac6dc5168b00f4c3dad1aea8d91cebd9e">Each event is saved to a queue after it occurs. When creating a SparkContext object, Driver starts a thread to obtain an event from the queue in sequence and sends the event to each Listener. Each Listener processes the event after detecting the event.</p>
|
|
<p id="mrs_01_1945__ac1f6c1688ad543a7b76e90770d967be1">Therefore, when the queuing speed is faster than the read speed, the queue overflows. As a result, the overflow event is lost, affecting the UI, EventLog, and dynamic resource scheduling functions. Therefore, a configuration item is added for more flexible use. You can set a proper value based on the memory size of the driver.</p>
|
|
</div>
|
|
<div class="section" id="mrs_01_1945__sc7dc453d15124448ac121d4c962d514a"><h4 class="sectiontitle">Configuration Description</h4><p id="mrs_01_1945__a94a08bdd53d149cdb7ab8b71d7ce8d72"><strong id="mrs_01_1945__a9d75af7c574c4770bd4552384f4a3251">Navigation path for setting parameters:</strong></p>
|
|
<p id="mrs_01_1945__aa1da597b8b95426cb9fac88023c041a8">Before executing an application, modify the Spark service configuration. On Manager, choose <span class="menucascade" id="mrs_01_1945__m6ca77d75ac3249dca520def4a9f15fd3"><b><span class="uicontrol" id="mrs_01_1945__u259a526d16bf428e8e08be19c5ef90d5"><span id="mrs_01_1945__text11652135610719">Cluster > <em id="mrs_01_1945__i18394133505818">Name of the desired cluster</em> > </span>Service</span></b> > <b><span class="uicontrol" id="mrs_01_1945__u06ce28fb234142acb5bbf2e905b9d179">Spark2x</span></b> > <b><span class="uicontrol" id="mrs_01_1945__ua6560c59db6b4547930b3f91d71ba15b">Configuration</span></b></span> and click <span class="parmvalue" id="mrs_01_1945__p69b6939ef61a4ad4b41b8e1d64f4c2ba"><b>All Configurations</b></span>. Enter a parameter name in the search box.</p>
|
|
|
|
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" id="mrs_01_1945__te4186f213d04444b875444b348663b8a" frame="border" border="1" rules="all"><caption><b>Table 1 </b>Parameter description</caption><thead align="left"><tr id="mrs_01_1945__rdf40be7d301944cb8872ec7da25514fd"><th align="left" class="cellrowborder" valign="top" width="22.430000000000003%" id="mcps1.3.2.4.2.4.1.1"><p id="mrs_01_1945__a7065353c39174288b29607cb592cddd1">Parameter</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="64.54%" id="mcps1.3.2.4.2.4.1.2"><p id="mrs_01_1945__aa46b98292ccb48aebdd6c5945c984d6b">Description</p>
|
|
</th>
|
|
<th align="left" class="cellrowborder" valign="top" width="13.030000000000003%" id="mcps1.3.2.4.2.4.1.3"><p id="mrs_01_1945__adc63113d24b740008c44c7eb082602f6">Default Value</p>
|
|
</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody><tr id="mrs_01_1945__r915d75e37c2246589151851deb466727"><td class="cellrowborder" valign="top" width="22.430000000000003%" headers="mcps1.3.2.4.2.4.1.1 "><p id="mrs_01_1945__a566689de94c841af8726ac565fc4e76f">spark.scheduler.listenerbus.eventqueue.capacity</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="64.54%" headers="mcps1.3.2.4.2.4.1.2 "><p id="mrs_01_1945__a07e63811f69f43c2bdc1e4583d2fd0af">Specifies the size of the event queue. Configure this parameter based on the memory of the driver.</p>
|
|
</td>
|
|
<td class="cellrowborder" valign="top" width="13.030000000000003%" headers="mcps1.3.2.4.2.4.1.3 "><p id="mrs_01_1945__a77dc365770b64874b26b0c6c481fcd04">1000000</p>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<div class="note" id="mrs_01_1945__n184dbd30e32043508da6462079ace617"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1945__a330ee9bf41be42ab8ff2011522ef9773">If the following information is displayed in the Driver log, the queue overflows.</p>
|
|
<ol id="mrs_01_1945__oe31995b9ea5e4db9b859d142013682ec"><li id="mrs_01_1945__lbffa60813d454381a3e8b6b619abaaac">Common application:<pre class="screen" id="mrs_01_1945__s866b3ed6d2fe408b8b3b95f4fe39f3cf">Dropping SparkListenerEvent because no remaining room in event queue.
|
|
This likely means one of the SparkListeners is too slow and cannot keep
|
|
up with the rate at which tasks are being started by the scheduler.</pre>
|
|
</li><li id="mrs_01_1945__l1442273eb4674da1abf41922e54c54f7">Spark Streaming application:<pre class="screen" id="mrs_01_1945__se8dbbfbd29124e3b95a0476af95fcd20">Dropping StreamingListenerEvent because no remaining room in event queue.
|
|
This likely means one of the StreamingListeners is too slow and cannot keep
|
|
up with the rate at which events are being started by the scheduler.</pre>
|
|
</li></ol>
|
|
</div></div>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1941.html">Scenario-Specific Configuration</a></div>
|
|
</div>
|
|
</div>
|
|
|