forked from docs/doc-exports
Yang, Tong
3f5759eed2
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
20 lines
3.3 KiB
HTML
20 lines
3.3 KiB
HTML
<a name="mrs_01_1593"></a><a name="mrs_01_1593"></a>
|
|
|
|
<h1 class="topictitle1">Summarization</h1>
|
|
<div id="body8662426"><div class="section" id="mrs_01_1593__en-us_topic_0000001219029085_s19ecc6ccab7343ef9b6eb739c889ea75"><h4 class="sectiontitle">Avoiding Data Skew</h4><p id="mrs_01_1593__en-us_topic_0000001219029085_a513c880a7c974e1eb4fea6467b5c9c29">If data skew occurs (certain data volume is large), the execution time of tasks is inconsistent even if no garbage collection is performed.</p>
|
|
<ul id="mrs_01_1593__en-us_topic_0000001219029085_u170bc44cfec246269b529d2e1022aa51"><li id="mrs_01_1593__en-us_topic_0000001219029085_l0901d8a060f241e1b6a5f7996c48f355">Redefine keys. Use keys of smaller granularity to optimize the task size.</li><li id="mrs_01_1593__en-us_topic_0000001219029085_l5877272ecccd453291ad7c6ebbec35bc">Modify the DOP.</li><li id="mrs_01_1593__en-us_topic_0000001219029085_l6a593784db3e4c43abed6b50d4770c24">Call the rebalance operation to balance data partitions.</li></ul>
|
|
</div>
|
|
<div class="section" id="mrs_01_1593__en-us_topic_0000001219029085_s4e06e1d83084415c8fb9fc4e41b67c6a"><h4 class="sectiontitle">Setting Timeout Interval for the Buffer</h4><ul id="mrs_01_1593__en-us_topic_0000001219029085_u6a001fc4bdc848febf9f3cd7addb55fa"><li id="mrs_01_1593__en-us_topic_0000001219029085_l27ee8fbfc42b4340b9d265c8393282c1">During the execution of tasks, data is switched through network switching. You can configure the <span class="parmname" id="mrs_01_1593__en-us_topic_0000001219029085_p978ffe40fd2446a8aaa8026698295f38"><b>setBufferTimeout</b></span> parameter to specify the timeout interval for the buffer.</li><li id="mrs_01_1593__en-us_topic_0000001219029085_l008f63cb69204d1fab72ce14b6b7a8ec">If <span class="parmname" id="mrs_01_1593__en-us_topic_0000001219029085_p1f6a010dfa5146ba84faa2b0f2f021f5"><b>setBufferTimeout</b></span> is set to <span class="parmvalue" id="mrs_01_1593__en-us_topic_0000001219029085_p8ce3a368172548e1aed4bc26eb7572d0"><b>-1</b></span>, the refreshing operation is performed when the buffer full, maximizing the throughput. If <span class="parmname" id="mrs_01_1593__en-us_topic_0000001219029085_p12ff095a03894e8f9927623b2efc306f"><b>setBufferTimeout</b></span> is set to <span class="parmvalue" id="mrs_01_1593__en-us_topic_0000001219029085_p1454fea50462438486c4ced718e380cf"><b>0</b></span>, the refreshing operation is performed each time data is received, minimizing the delay. If <span class="parmname" id="mrs_01_1593__en-us_topic_0000001219029085_pc3e0f617856743c5a9bbfd389adc817a"><b>setBufferTimeout</b></span> is set to a value greater than <span class="parmvalue" id="mrs_01_1593__en-us_topic_0000001219029085_p9086a4e9b7114f50bcb79cf749bf0e94"><b>0</b></span>, the refreshing operation is performed after the butter times out.<p id="mrs_01_1593__en-us_topic_0000001219029085_a0d16a6cdba694725a8e2734bb9669963">The following is an example:</p>
|
|
<pre class="screen" id="mrs_01_1593__en-us_topic_0000001219029085_s57a91b11813b442dbc4a936c8c4381f2">env.setBufferTimeout(timeoutMillis);
|
|
|
|
env.generateSequence(1,10).map(new MyMapper()).setBufferTimeout(timeoutMillis);</pre>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1587.html">Optimization DataStream</a></div>
|
|
</div>
|
|
</div>
|
|
|