vpruthi/doc-exports
1
0
Fork 0
forked from docs/doc-exports
Code Pull Requests Activity
doc-exports/docs/dli/umn/dli_03_0234.html
Su, Xiaomeng 12dd64efc7 dli_umn_20240430 
Reviewed-by: Pruthi, Vineet <vineet.pruthi@t-systems.com>
Co-authored-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
Co-committed-by: Su, Xiaomeng <suxiaomeng1@huawei.com>
2024-05-15 11:56:22 +00:00

4.2 KiB
Raw Blame History

Why Does the Flink Submission Fail Due to Hadoop JAR File Conflict?

Symptom

Flink Job submission failed. The exception information is as follows:

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider not found
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2664)
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
 ... 31 common frames omitted
 Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.obs.metrics.OBSAMetricsProvider not found
 at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2568)
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2662)
 ... 32 common frames omitted

Cause Analysis

Flink JAR files conflicted. The submitted Flink JAR file conflicted with the HDFS JAR file of the DLI cluster.

Procedure

  1. Configure hadoop-hdfs in the POM file as follows:

     <dependency>
 <groupId>org.apache.hadoop</groupId>
 <artifactId>hadoop-hdfs</artifactId>
 <version>${hadoop.version}</version>
 <scope> provided </scope>
 </dependency>

    Alternatively, use the exclusions tag to exclude the association.

  2. To use HDFS configuration files, change core-site.xml, hdfs-site.xml, and yarn-site.xml to mrs-core-site.xml, mrs-hdfs-site.xml and mrs-hbase-site.xml, respectively.

     conf.addResource(HBaseUtil.class.getClassLoader().getResourceAsStream("mrs-core-site.xml"), false);
 conf.addResource(HBaseUtil.class.getClassLoader().getResourceAsStream("mrs-hdfs-site.xml"), false);
 conf.addResource(HBaseUtil.class.getClassLoader().getResourceAsStream("mrs-hbase-site.xml"), false);

Parent topic: Flink Jar Jobs