forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
71 lines
6.3 KiB
HTML
71 lines
6.3 KiB
HTML
<a name="mrs_01_2007"></a><a name="mrs_01_2007"></a>
|
|
|
|
<h1 class="topictitle1">Why Does FetchFailedException Occur When the Network Connection Is Timed out</h1>
|
|
<div id="body1595920219445"><div class="section" id="mrs_01_2007__s01a93fe2ff4342e987900cb61fa33fbd"><h4 class="sectiontitle">Question</h4><p id="mrs_01_2007__aa530a08ffe424d5b92927c698ac51232">On a large cluster of 380 nodes, run the ScalaSort test case in the HiBench test that runs the 29T data, and configure Executor as <span class="parmname" id="mrs_01_2007__pd791bb85f11b41fc86aa058cbcfac9e2"><b>--executor-cores 4</b></span>. The following abnormality is displayed:</p>
|
|
<pre class="screen" id="mrs_01_2007__s5cfeec3660b04fb6b6be016b27e2bde5">org.apache.spark.shuffle.FetchFailedException: Failed to connect to /192.168.114.12:23242
|
|
at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:321)
|
|
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:306)
|
|
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
|
|
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
|
|
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
|
|
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
|
|
at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
|
|
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
|
|
at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:217)
|
|
at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:102)
|
|
at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:90)
|
|
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:301)
|
|
at org.apache.spark.rdd.RDD.iterator(RDD.scala:265)
|
|
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
|
|
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:301)
|
|
at org.apache.spark.rdd.RDD.iterator(RDD.scala:265)
|
|
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
|
|
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:301)
|
|
at org.apache.spark.rdd.RDD.iterator(RDD.scala:265)
|
|
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
|
|
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:301)
|
|
at org.apache.spark.rdd.RDD.iterator(RDD.scala:265)
|
|
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
|
|
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
|
|
at org.apache.spark.scheduler.Task.run(Task.scala:87)
|
|
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
|
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
|
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
|
|
at java.lang.Thread.run(Thread.java:745)
|
|
Caused by: java.io.IOException: Failed to connect to /192.168.114.12:23242
|
|
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:214)
|
|
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
|
|
at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:91)
|
|
at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
|
|
at org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
|
|
at org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170)
|
|
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
|
|
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
|
|
... 3 more
|
|
Caused by: java.net.ConnectException: Connection timed out: /192.168.114.12:23242
|
|
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
|
|
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
|
|
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
|
|
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
|
|
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
|
|
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
|
|
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
|
|
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
|
|
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
|
|
... 1 more</pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_2007__s77095240a026459e932a9f1f2f98a8e8"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_2007__a901139d299cc4490a02497f76006f5b0">When an application is run, configure the Executor parameter as <span class="parmname" id="mrs_01_2007__p9cbf89ce555b4560990e4df4d5a4ce05"><b>--executor-cores 4</b></span>. The degree of parallelism (DOP) is high in a single process, resulting in that the IO is highly occupied and the task works slowly.</p>
|
|
<pre class="screen" id="mrs_01_2007__s1516eefd58704f39a2d4a0a4b90ff420">16/02/26 10:04:53 INFO TaskSetManager: Finished task 2139.0 in stage 1.0 (TID 151149) in 376455 ms on 10-196-115-2 (694/153378)</pre>
|
|
<p id="mrs_01_2007__a4bba4ae9b3a34585ac4560fce628c510">Because running a single task takes more than 6 minutes. The network connection is timed out and the running task fails.</p>
|
|
<p id="mrs_01_2007__ae6b1eb81d40146dea791f6d468f5d175">Set the number of cores as 1, which is <span class="parmname" id="mrs_01_2007__pb2d7b3f56eb1405bbbca0d1c03072903"><b>--executor-cores 1</b></span>. A task is executed smoothly in proper time (within 15s).</p>
|
|
<pre class="screen" id="mrs_01_2007__s07f2edb307ef471db675cf549ed8a6ae">16/02/29 02:24:46 INFO TaskSetManager: Finished task 59564.0 in stage 1.0 (TID 208574) in 15088 ms on 10-196-115-6 (59515/153378)</pre>
|
|
<p id="mrs_01_2007__a26a36305fbd44c94837237d9ce9bd7e6">Therefore, to process the task of network connection timed out and avoid such error, you can reduce the core number of a single Executor.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_2003.html">Spark Core</a></div>
|
|
</div>
|
|
</div>
|
|
|