forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
36 lines
7.2 KiB
HTML
36 lines
7.2 KiB
HTML
<a name="mrs_01_2001"></a><a name="mrs_01_2001"></a>
|
|
|
|
<h1 class="topictitle1">Spark Streaming Tuning</h1>
|
|
<div id="body1595920219241"><div class="section" id="mrs_01_2001__s57a4326aa7f54e5b99c717f9acc62838"><h4 class="sectiontitle">Scenario</h4><p id="mrs_01_2001__a603a88a878c349629200bada11978c5b">Streaming is a mini-batch streaming processing framework that features second-level delay and high throughput. To optimize Streaming is to improve its throughput while maintaining second-level delay so that more data can be processed per unit time.</p>
|
|
<div class="note" id="mrs_01_2001__n7be7c1a219f348429548cc65bb555231"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_2001__ad30ac0e223fe4a8f998a327a2c6b3224">This section applies to the scenario where the input data source is Kafka.</p>
|
|
</div></div>
|
|
</div>
|
|
<div class="section" id="mrs_01_2001__s94c3ee6558af4933b18bd82b6ad25b40"><h4 class="sectiontitle">Procedure</h4><p id="mrs_01_2001__af72b0c00aef443b1946b125654253781">A simple streaming processing system consists of a data source, a receiver, and a processor. The data source is Kafka, the receiver is the Kafka data source receiver of Streaming, and the processor is Streaming.</p>
|
|
<p id="mrs_01_2001__a1ca83dcdd5a246598e7d9b034b5e4ce1">Streaming optimization is to optimize the performance of the three components.</p>
|
|
<ul id="mrs_01_2001__ua3457d70c5394bd0b42dd96f1eaf01a8"><li id="mrs_01_2001__lec073b37150544ada465fa2440c4588a"><strong id="mrs_01_2001__b1641075095513">Data source optimization</strong><p id="mrs_01_2001__ab8263009e6d14fb097730d531ae5d2df">In actual application scenarios, the data source stores the data in the local disks to ensure the error tolerance of the data. However, the calculation results of the Streaming are stored in the memory, and the data source may become the largest bottleneck of the streaming system.</p>
|
|
<p id="mrs_01_2001__a66e982561ef44489a315a0bafa17d81b">Kafka can be optimized from the following aspects:</p>
|
|
<ul id="mrs_01_2001__uca3ff8c4552e4bdfa6b5ff2f11cdd532"><li id="mrs_01_2001__lfb806757ece249f1ba29f1913ad7ef62">Use Kafka-0.8.2 or later version that allows you to use new Producer APIs in asynchronous mode.</li><li id="mrs_01_2001__l0e8f65a42d0f4126ae1697306cb72cb8">Configure multiple Broker directories, multiple I/O threads, and a proper number of partitions for a topic.</li></ul>
|
|
<p id="mrs_01_2001__af7f4f3e2dbed410b87334998ec1a1387">For details, see section <strong id="mrs_01_2001__b10975161013564">Performance Tuning</strong> in the Kafka open source documentation at <a href="http://kafka.apache.org/documentation.html" target="_blank" rel="noopener noreferrer">http://kafka.apache.org/documentation.html</a>.</p>
|
|
</li><li id="mrs_01_2001__li1881842762212"><strong id="mrs_01_2001__b1576718190563">Receiver optimization</strong><p id="mrs_01_2001__aabc4388e926c483f829e2d59de4f26d8">Streaming has multiple data source receivers, such as Kafka, Flume, MQTT, and ZeroMQ. Kafka has the most receiver types and is the most mature receiver.</p>
|
|
<p id="mrs_01_2001__a084d0794d7e1454794f23ab0147bfdbe">Kafka provides three types of receiver APIs:</p>
|
|
<ul id="mrs_01_2001__u4a894b9cbda04a7eadf5af41cc15a463"><li id="mrs_01_2001__lcb1f9711f5144feabf868084fabf7cdc">KafkaReceiver directly receives Kafka data. If the process is abnormal, data may be lost.</li><li id="mrs_01_2001__l8b19c6845ddc467abd0a045d126cd6b1">ReliableKafkaReceiver receives data displacement through ZooKeeper records.</li><li id="mrs_01_2001__l7196edebb0dd46c38b8861d0d47f29e3">DirectKafka reads data from each partition of Kafka through the RDD, ensuring high reliability.</li></ul>
|
|
<p id="mrs_01_2001__a2ac837b54e7f4299b5c3127119e110b9">According to the implementation mechanism and test results, DirectKafka provides better performance than the other two APIs. Therefore, the DirectKafka API is recommended to implement the receiver.</p>
|
|
<p id="mrs_01_2001__ad02cc7a90ba048f0afc026d6bdd8dc34">For details about the Kafka receivers and their optimization methods, see the Kafka open source documentation at <a href="http://kafka.apache.org/documentation.html" target="_blank" rel="noopener noreferrer">http://kafka.apache.org/documentation.html</a>.</p>
|
|
</li><li id="mrs_01_2001__li2469104310226"><strong id="mrs_01_2001__b11553546568">Processor optimization</strong><p id="mrs_01_2001__a3c310a3a2aea4c53ab6fe8ae0ae1f620">The bottom layer of Spark Streaming is executed by Spark. Therefore, most optimization measures for Spark can also be applied to Spark Streaming. The following is an example:</p>
|
|
<ul id="mrs_01_2001__u861ce87bd6d04f89af85273d4198ed19"><li id="mrs_01_2001__ldc3e61769d8047308c4b8dcaa36303ed">Data serialization</li><li id="mrs_01_2001__l5e475d5dd1d149e6a7a3afbc239bfe10">Memory configuration</li><li id="mrs_01_2001__l927c1f69ebd14ee69a0e940d7406eb97">Configuring DOP</li><li id="mrs_01_2001__l3de833930cce479c8d1ea5c5bd07881a">Using the external shuffle service to improve performance</li></ul>
|
|
<div class="note" id="mrs_01_2001__n0757f321eb764bfdb21bc53e43a52329"><img src="public_sys-resources/note_3.0-en-us.png"><span class="notetitle"> </span><div class="notebody"><p id="mrs_01_2001__ac5734d2237234786a2832a0522dd88c1">Higher performance of Spark Streaming indicates lower overall reliability. Examples:</p>
|
|
<p id="mrs_01_2001__a105b7b37cd834f1fa960f6bae06c86a5">If <span class="parmname" id="mrs_01_2001__p3e8a185fc98348f6ab275bd1c30094c1"><b>spark.streaming.receiver.writeAheadLog.enable</b></span> is set to <span class="parmvalue" id="mrs_01_2001__p9285cbc7a7ff4943a1c9b06622d3f311"><b>false</b></span>, disk I/Os are reduced and performance is improved. However, because WAL is disabled, data is lost during fault recovery.</p>
|
|
<p id="mrs_01_2001__a2aa4e376693c4d03a739b6680477f392">Therefore, do not disable configuration items that ensure data reliability in production environments during Spark Streaming tuning.</p>
|
|
</div></div>
|
|
</li><li id="mrs_01_2001__li8135712233"><strong id="mrs_01_2001__b1333685411310">Log archive optimization</strong><p id="mrs_01_2001__ad6da9d90c1494c488a8772cefdca0a55">The <span class="parmname" id="mrs_01_2001__p464467f58fd2406a8f8b784bff7e5109"><b>spark.eventLog.group.size</b></span> parameter is used to group <strong id="mrs_01_2001__b4353175810312">JobHistory</strong> logs of an application based on the specified number of jobs. Each group creates a file recording log to prevent <strong id="mrs_01_2001__b163535585319">JobHistory</strong> reading failures caused by an oversized log generated during the long-term running of the application. If this parameter is set to <span class="parmvalue" id="mrs_01_2001__p02d04a30d7154fce9f0e0df35862b9ba"><b>0</b></span>, logs are not grouped.</p>
|
|
<p id="mrs_01_2001__a64db11554001461cbae26011c1034622">Most Spark Streaming jobs are small jobs and are generated at a high speed. As a result, frequent grouping is performed and a large number of small log files are generated, consuming disk I/O resources. You are advised to increase the parameter value to, for example, <span class="parmvalue" id="mrs_01_2001__peb958f78057d40b79832b06220256a35"><b>1000</b></span> or greater.</p>
|
|
</li></ul>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1974.html">Spark2x Performance Tuning</a></div>
|
|
</div>
|
|
</div>
|
|
|