forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
20 lines
3.7 KiB
HTML
20 lines
3.7 KiB
HTML
<a name="mrs_01_1588"></a><a name="mrs_01_1588"></a>
|
|
|
|
<h1 class="topictitle1">Memory Configuration Optimization</h1>
|
|
<div id="body1596442815778"><div class="section" id="mrs_01_1588__s644e5930d2e54bf898eecab47bd7f36c"><h4 class="sectiontitle">Scenarios</h4><p class="msonormal" id="mrs_01_1588__a9cff5a5c5e6b4cb59882f130c8192f61">The computing of Flink depends on memory. If the memory is insufficient, the performance of Flink will be greatly deteriorated. One solution is to monitor garbage collection (GC) to evaluate the memory usage. If the memory becomes the performance bottleneck, optimize the memory usage according to the actual situation.</p>
|
|
<p class="msonormal" id="mrs_01_1588__abc0b14d852fb4ada95fde5c61a0976c2">If <span class="uicontrol" id="mrs_01_1588__u4b21843df75f4cbe9a998d1c9831595f"><b>Full GC</b></span> is frequently reported in the Container GC on the Yarn that monitors the node processes, the GC needs to be optimized.</p>
|
|
<div class="note" id="mrs_01_1588__n4c39f10842ff40ed9310830bdf48eb0b"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_1588__a78806b90706148138b9cc51ef1caf4cd">In the <span class="parmname" id="mrs_01_1588__p5d37c6f1a2594dadb1807bba436233ee"><b>env.java.opts</b></span> configuration item of the <span class="parmname" id="mrs_01_1588__p90d98142d6364fff9bab90a18ebbe842"><b>conf/flink-conf.yaml</b></span> file on the client, add the <span class="parmname" id="mrs_01_1588__pf5fe829d16f54d758c2dffd9cdab8ff0"><b>-Xloggc:<LOG_DIR>/gc.log -XX:+PrintGCDetails -XX:-OmitStackTraceInFastThrow -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=20M</b></span> parameter. The GC log is configured by default.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_1588__se9835c745a1044ee80afe327b9a3a620"><h4 class="sectiontitle">Procedure</h4><ul id="mrs_01_1588__u69632b1071774885b72c4be66bb72a52"><li id="mrs_01_1588__l6fa9a7f1251b4b7589b1f18e5a4fef72">Optimize GC.<p id="mrs_01_1588__adb30d0d298cf49c5b4681d2d0439fb6b"><a name="mrs_01_1588__l6fa9a7f1251b4b7589b1f18e5a4fef72"></a><a name="l6fa9a7f1251b4b7589b1f18e5a4fef72"></a>Adjust the ratio of tenured generation memory to young generation memory. In the <span class="parmname" id="mrs_01_1588__pce576bb9caeb4b77ab3820fea9e07e56"><b>conf/flink-conf.yaml </b></span>configuration file on the client, add the <span class="parmname" id="mrs_01_1588__peaa21fb681cf4ee49ad1de30f7808a16"><b>-XX:NewRatio</b></span> parameter to the <span class="parmname" id="mrs_01_1588__pe4d430c63ab9435688321639abca70e5"><b>env.java.opts</b></span> configuration item. For example, <span class="parmname" id="mrs_01_1588__p38cb69f3107e48de817158a5cbf89e0d"><b>-XX:NewRatio=2</b></span> indicates that ratio of tenured generation memory to young generation memory is 2:1, that is, the young generation memory occupies one third and tenured generation memory occupies two thirds.</p>
|
|
</li><li id="mrs_01_1588__lb591081493e746feafa060035b39ef17">When developing Flink applications, optimize the partitioning or grouping operation of DataStream.<ul id="mrs_01_1588__u5e59e82ff0b24f4586e0d390d7997c68"><li id="mrs_01_1588__l9185f76d223e408896776164cc3635bd">If partitioning causes data skew, partitions need to be optimized.</li><li id="mrs_01_1588__ld47ded92f67d48a7842ecccecfc65549">Do not perform concurrent operations, because some operations, WindowAll for example, to DataStream do not support parallelism.</li><li id="mrs_01_1588__lb0bb242e0d9044d2a302d675e4e3d98c">Do not use set keyBy to string type.</li></ul>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1587.html">Optimization DataStream</a></div>
|
|
</div>
|
|
</div>
|
|
|