forked from docs/doc-exports
Yang, Tong
6182f91ba8
Reviewed-by: Hasko, Vladimir <vladimir.hasko@t-systems.com> Co-authored-by: Yang, Tong <yangtong2@huawei.com> Co-committed-by: Yang, Tong <yangtong2@huawei.com>
26 lines
2.5 KiB
HTML
26 lines
2.5 KiB
HTML
<a name="mrs_01_1700"></a><a name="mrs_01_1700"></a>
|
|
|
|
<h1 class="topictitle1">Why Does Array Border-crossing Occur During FileInputFormat Split?</h1>
|
|
<div id="body1597735020141"><div class="section" id="mrs_01_1700__sd6cfe94a8277481e9cebd708c59d735f"><h4 class="sectiontitle">Question</h4><p id="mrs_01_1700__ac1d0f85c88c549a19f990308df4feef7">When HDFS calls the FileInputFormat getSplit method, the ArrayIndexOutOfBoundsException: 0 appears in the following log:</p>
|
|
<pre class="screen" id="mrs_01_1700__s7645825abdf242d89795e426b4163991"><span id="mrs_01_1700__ph261215351772">java.lang.ArrayIndexOutOfBoundsException: 0</span>
|
|
<span id="mrs_01_1700__ph1061283511712">at org.apache.hadoop.mapred.FileInputFormat.identifyHosts(FileInputFormat.java:708)</span>
|
|
<span id="mrs_01_1700__ph7612935177">at org.apache.hadoop.mapred.FileInputFormat.getSplitHostsAndCachedHosts(FileInputFormat.java:675)</span>
|
|
<span id="mrs_01_1700__ph86131735674">at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:359)</span>
|
|
<span id="mrs_01_1700__ph861315356710">at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:210)</span>
|
|
<span id="mrs_01_1700__ph261317359710">at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)</span>
|
|
<span id="mrs_01_1700__ph061414357710">at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)</span>
|
|
<span id="mrs_01_1700__ph13614435872">at scala.Option.getOrElse(Option.scala:120)</span>
|
|
<span id="mrs_01_1700__ph10614535375">at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)</span>
|
|
<span id="mrs_01_1700__ph86149351572">at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)</span></pre>
|
|
</div>
|
|
<div class="section" id="mrs_01_1700__sbb1b6b67f3d8491f8579feeb968fbd5c"><h4 class="sectiontitle">Answer</h4><p id="mrs_01_1700__a159ffeeaa3a84a83b3d03bac23ce2660">The elements of each block correspondent frame are as below: /default/rack0/:,/default/rack0/datanodeip:port.</p>
|
|
<p id="mrs_01_1700__a3f193adfb98e4d8ba2e63a645109fb87">The problem is due to a block damage or loss, making the block correspondent machine ip and port become null. Use <strong id="mrs_01_1700__b13503171315181">hdfs fsck</strong> to check the file blocks health state when this problem occurs, and remove damaged block or restore the missing block to re-computing the task.</p>
|
|
</div>
|
|
</div>
|
|
<div>
|
|
<div class="familylinks">
|
|
<div class="parentlink"><strong>Parent topic:</strong> <a href="mrs_01_1690.html">FAQ</a></div>
|
|
</div>
|
|
</div>
|
|
|