forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
27 lines
4.2 KiB
HTML
27 lines
4.2 KiB
HTML
<a name="mrs_01_2008"></a><a name="mrs_01_2008"></a>
|
|
|
|
<h1 class="topictitle1">How to Configure Event Queue Size If Event Queue Overflows?</h1>
|
|
<div id="body1595920219446"><div class="section" id="mrs_01_2008__sb922c9a16cb14595842f08df843b660f"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2008__af37d899d382f4d069db74eade2be8535">How to configure the event queue size if the following Driver log information is displayed indicating that the event queue overflows?</p>
|
|
<ul id="mrs_01_2008__u4876f0a67d184d4aa3932db94f522f20"><li id="mrs_01_2008__l596d11085b0e4bfbb1bd5b9316728fc4">Common applications<pre class="screen" id="mrs_01_2008__s2d80139568d44bc8ad9ca03f86397e04">Dropping SparkListenerEvent because no remaining room in event queue.
|
|
This likely means one of the SparkListeners is too slow and cannot keep
|
|
up with the rate at which tasks are being started by the scheduler.</pre>
|
|
</li><li id="mrs_01_2008__l802584341f074a8cb4c30a020c25241c">Spark Streaming applications<pre class="screen" id="mrs_01_2008__sa57642942dfd408a913414f809533310">Dropping StreamingListenerEvent because no remaining room in event queue.
|
|
This likely means one of the StreamingListeners is too slow and cannot keep
|
|
up with the rate at which events are being started by the scheduler.</pre>
|
|
</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_2008__sebca6a3dd7624f57bf940300c8daf22a"><h4 class="sectiontitle">Answer</h4><ol id="mrs_01_2008__o6b1bc6ebb4674ce4b5ef9a55dc32a2db"><li id="mrs_01_2008__l1c4a5d7872624e218aff70f81807933d">Stop the application. Set the configuration option <span class="parmname" id="mrs_01_2008__p08d1e9ada5944bbca7aabf0cb5705983"><b>spark.event.listener.logEnable</b></span> in the Spark configuration file <span class="filepath" id="mrs_01_2008__f6148cc91e3ed4cef82965ee7f911efa3"><b>spark-defaults.conf</b></span> to <span class="parmvalue" id="mrs_01_2008__p9461755d17304d5889f5eeb81fe81526"><b>true</b></span>. And set the configuration option <span class="parmname" id="mrs_01_2008__p85fba62c7cc247d9a21d822cc51ea9cb"><b>spark.eventQueue.size</b></span> to <span class="parmvalue" id="mrs_01_2008__pa8ef80bb81ea494bba9b90d376284a8c"><b>1000W</b></span>. If you need to control the logging rate (in milliseconds), also change the value of the configuration option <span class="parmname" id="mrs_01_2008__p1217e572db3442aa9cae5862d361d95e"><b>spark.event.listener.logRate</b></span>.<p id="mrs_01_2008__a20d39fff48dd409286a4c73ba1ddc2b2">By default, the logging rate is 1000 ms, which means that one log is printed out every 1000 ms.</p>
|
|
</li><li id="mrs_01_2008__l20780d97240a4bfe9bb85d0bf589c63c">Start the application.<div class="p" id="mrs_01_2008__a81cb74b50775488b8cc104b62f56adc0"><a name="mrs_01_2008__l20780d97240a4bfe9bb85d0bf589c63c"></a><a name="l20780d97240a4bfe9bb85d0bf589c63c"></a>The following log information is displayed, including the event consumption rate, event production rate, and <span class="parmname" id="mrs_01_2008__p983cd85e31da41bf95921bbcf1fe5308"><b>MaxSize</b></span> (maximum size of messages in the queue).<pre class="screen" id="mrs_01_2008__saa0af0b62c1943bb86c8880fd35f0698">INFO LiveListenerBus: [SparkListenerBus]:16044 events are consumed in 5000 ms.
|
|
INFO LiveListenerBus: [SparkListenerBus]:51381 events are produced in 5000 ms, eventQueue still has 86417 events, MaxSize: 171764.</pre>
|
|
</div>
|
|
</li><li id="mrs_01_2008__lc0de1e59729c440b900a53713d051815">Change the value of the configuration option <span class="parmname" id="mrs_01_2008__p5a7e108c2ba64f1dad09c00c244e7be8"><b>spark.eventQueue.size</b></span> in the Spark configuration file <span class="filepath" id="mrs_01_2008__f7fe963585f3647fdbddec66cbb1d538f"><b>spark-defaults.conf</b></span> based on the <span class="parmname" id="mrs_01_2008__pe1bb9d46154a4f7da1bb93b041a5d725"><b>MaxSize</b></span> in the log information.<p id="mrs_01_2008__a1bd7852c032e4a359d6eefadca7495b7">For example, if <span class="parmname" id="mrs_01_2008__pe9840f9382144e69bf93c74c881155c8"><b>MaxSize</b></span> is 250000, the appropriate message queue size is 300000.</p>
|
|
</li></ol>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2003.html">Spark Core</a></div>
|
|
</div>
|
|
</div>
|
|
|