forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
54 lines
5.1 KiB
HTML
54 lines
5.1 KiB
HTML
<a name="mrs_01_1705"></a><a name="mrs_01_1705"></a>
|
|
|
|
<h1 class="topictitle1">Why Is "java.net.SocketException: No buffer space available" Reported When Data Is Written to HDFS</h1>
|
|
<div id="body1597735020370"><div class="section" id="mrs_01_1705__sf1bffdf522534ef49385deefa780a9d6"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1705__a691e81a7bc704787a145fbece7f67be9">Why is an "java.net.SocketException: No buffer space available" exception reported when data is written to HDFS?</p>
|
|
<p id="mrs_01_1705__a4f6bedf2bc054b8b95d85bcfa41c3cb8">This problem occurs when files are written to the HDFS. Check the error logs of the client and DataNode.</p>
|
|
<p id="mrs_01_1705__a97f0892a7dd84b3599376f1cc2838758">The client logs are as follows:</p>
|
|
<div class="fignone" id="mrs_01_1705__f4e463f46c2eb4068a75effe4abc284dd"><span class="figcap"><b>Figure 1 </b>Client logs</span><br><span><img id="mrs_01_1705__icb7726e9a0c249f9a2118ccb51f07aa8" src="en-us_image_0000001295930408.jpg"></span></div>
|
|
<p id="mrs_01_1705__aac863f013a6d4388a4de47a6247bb848">DataNode logs are as follows:</p>
|
|
<pre class="screen" id="mrs_01_1705__sd3ecad5a11e44d94b60cfc15c561d43f">2017-07-24 20:43:39,269 | ERROR | DataXceiver for client DFSClient_NONMAPREDUCE_996005058_86
|
|
at /192.168.164.155:40214 [Receiving block BP-1287143557-192.168.199.6-1500707719940:blk_1074269754_528941 with io weight 10] | DataNode{data=FSDataset{dirpath='[/srv/BigData/hadoop/data1/dn/current, /srv/BigData/hadoop/data2/dn/current, /srv/BigData/hadoop/data3/dn/current, /srv/BigData/hadoop/data4/dn/current, /srv/BigData/hadoop/data5/dn/current, /srv/BigData/hadoop/data6/dn/current, /srv/BigData/hadoop/data7/dn/current]'}, localName='192-168-164-155:9866', datanodeUuid='a013e29c-4e72-400c-bc7b-bbbf0799604c', xmitsInProgress=0}:Exception transfering block BP-1287143557-192.168.199.6-1500707719940:blk_1074269754_528941 to mirror 192.168.202.99:9866: java.net.SocketException: No buffer space available | DataXceiver.java:870
|
|
2017-07-24 20:43:39,269 | INFO | DataXceiver for client DFSClient_NONMAPREDUCE_996005058_86
|
|
at /192.168.164.155:40214 [Receiving block BP-1287143557-192.168.199.6-1500707719940:blk_1074269754_528941 with io weight 10] | opWriteBlock BP-1287143557-192.168.199.6-1500707719940:blk_1074269754_528941 received exception java.net.SocketException: No buffer space available | DataXceiver.java:933
|
|
2017-07-24 20:43:39,270 | ERROR | DataXceiver for client DFSClient_NONMAPREDUCE_996005058_86
|
|
at /192.168.164.155:40214 [Receiving block BP-1287143557-192.168.199.6-1500707719940:blk_1074269754_528941 with io weight 10] | 192-168-164-155:9866:DataXceiver error processing WRITE_BLOCK operation src: /192.168.164.155:40214 dst: /192.168.164.155:9866 | DataXceiver.java:304 <strong id="mrs_01_1705__a8d2b0b938abf4a76952c9ca12e94726e">java.net.SocketException: No buffer space available</strong>
|
|
at sun.nio.ch.Net.connect0(Native Method)
|
|
at sun.nio.ch.Net.connect(Net.java:454)
|
|
at sun.nio.ch.Net.connect(Net.java:446)
|
|
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
|
|
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:192)
|
|
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
|
|
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
|
|
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:800)
|
|
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:138)
|
|
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
|
|
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:265)
|
|
at java.lang.Thread.run(Thread.java:748)</pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_1705__sab5f82cdf20940ecaa267d1738e8582b"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1705__af6f767124ba14658944e2232c33230d3">The preceding problem may be caused by network memory exhaustion.</p>
|
|
<p id="mrs_01_1705__ac5618cf7b50444e582d0ce7d528921ad">You can increase the threshold of the network device based on the actual scenario.</p>
|
|
<p id="mrs_01_1705__a610096cf576a4d6db7f3ea6022c632c7">Example:</p>
|
|
<pre class="screen" id="mrs_01_1705__sfb7a32b155f14cc892398a5f42bb756d">[root@xxxxx ~]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh*
|
|
128
|
|
512
|
|
1024
|
|
[root@xxxxx ~]# echo 512 > /proc/sys/net/ipv4/neigh/default/gc_thresh1
|
|
[root@xxxxx ~]# echo 2048 > /proc/sys/net/ipv4/neigh/default/gc_thresh2
|
|
[root@xxxxx ~]# echo 4096 > /proc/sys/net/ipv4/neigh/default/gc_thresh3
|
|
[root@xxxxx ~]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh*
|
|
512
|
|
2048
|
|
4096</pre>
|
|
<p id="mrs_01_1705__a7245840b108f40d4a1d36c4b6bdc7c85">You can also add the following parameters to the <strong id="mrs_01_1705__b86043948722533">/etc/sysctl.conf</strong> file. The configuration takes effect even if the host is restarted.</p>
|
|
<pre class="screen" id="mrs_01_1705__s44e6a0637e0b415692c057cf05d33607">net.ipv4.neigh.default.gc_thresh1 = 512
|
|
net.ipv4.neigh.default.gc_thresh2 = 2048
|
|
net.ipv4.neigh.default.gc_thresh3 = 4096</pre>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1690.html">FAQ</a></div>
|
|
</div>
|
|
</div>
|
|
|