Getting this error due to Hadoop dependency:
I hav...
# troubleshootings
Sunil Srinivasa
08/31/2023, 11:52 AMGetting this error due to Hadoop dependency:
I have followed this post but still no luck. I’m using K8 operator for deployments.
Copy code
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at com.smaato.nrt.kafka2s3streamsync.JobFlowGraph.getS3FileSink(JobFlowGraph.scala:86) ~[?:?]
at com.smaato.nrt.kafka2s3streamsync.JobFlowGraph.sinkToKafka(JobFlowGraph.scala:98) ~[?:?]
at com.smaato.nrt.kafka2s3streamsync.JobFlowGraph.build(JobFlowGraph.scala:111) ~[?:?]
at com.smaato.nrt.kafka2s3streamsync.DataStreamJob$.main(DataStreamJob.scala:12) ~[?:?]
at com.smaato.nrt.kafka2s3streamsync.DataStreamJob.main(DataStreamJob.scala) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]g
Giannis Polyzos
08/31/2023, 12:17 PMtry adding
flink-shaded-hadoop-2-uber-2.8.3-10.0.jarGiannis Polyzos
08/31/2023, 12:18 PMs
Sunil Srinivasa
08/31/2023, 12:21 PMok, I’ll give a try
Sunil Srinivasa
08/31/2023, 12:31 PMstill got the same error
Sunil Srinivasa
08/31/2023, 12:32 PMI used the given lib along with
Copy code
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-hadoop-compatibility_${scala.binary.version}</artifactId>
<version>${flink.version}</version>
</dependency>Sunil Srinivasa
08/31/2023, 12:34 PMalso enabled built-in plugin:
flink-s3-fs-hadoop-1.17.1.jars
Sricharan
09/01/2023, 7:15 AMs
Sunil Srinivasa
09/04/2023, 8:33 AMalso enabled built-in plugin:@Sricharan the link which you shared, I’ve tried this as wellflink-s3-fs-hadoop-1.17.1.jar
g
Giannis Polyzos
09/04/2023, 8:35 AMcan u verify these jars are available on the cluster? never encountered this exception when the jar was available on the
/libss
Sunil Srinivasa
09/04/2023, 11:50 AM@Giannis Polyzos you are right! it is not copied to libs folder
Sunil Srinivasa
09/04/2023, 11:50 AMSunil Srinivasa
09/04/2023, 11:50 AMI’m following this document
Sunil Srinivasa
09/04/2023, 11:51 AMBut I’m using k8 opeartor with CRD.
Copy code
containers:
# Do not change the main container name
- name: flink-main-container
env:
# <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/resource-providers/standalone/docker/#using-filesystem-plugins>
- name: ENABLE_BUILT_IN_PLUGINS
value: flink-s3-fs-hadoop-1.17.1.jarSunil Srinivasa
09/04/2023, 11:51 AMbut this setting is not copying the jar file
g
Giannis Polyzos
09/04/2023, 11:52 AMYou could create a simple dockerfile, like:
Copy code
FROM flink:1.17.1
ADD jars/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar /opt/flink/lib/🙌 1
👍 1
s
Sunil Srinivasa
09/04/2023, 11:53 AMsure I’ll try this
s
Sricharan
09/04/2023, 5:49 PM@Sunil Srinivasa You have to add all the dependencies in the docker file to be available, like Ginnis shared.
s
Sunil Srinivasa
09/04/2023, 7:24 PM@Giannis Polyzos /@Sricharan issue was resolved after adding the Jar in the image. Thanks for the help!
🙏 2
6 Views