Hello everyone, I'm quite new to the Java world, s...
# troubleshooting
r
Hello everyone, I'm quite new to the Java world, so please don't be too hard on me if my question is stupid. πŸ™‚ I’m working on a pipeline job to process Parquet files stored in an S3-like storage. I followed guides on how to set up the project but adapted it to use Kotlin and build with Gradle, which worked well for another job. However, in this case, I ran into dependency issues. Flink Parquet requires some dependencies from Hadoop to work. For the other job, I added
hadoop-common:3.4.1
, and that was sufficient. But for this pipeline, it also requires
hadoop-mapreduce-core
. And that's where my troubles started. As soon as I add the MapReduce core dependency, I get a Java linkage error:
Copy code
Caused by: java.lang.LinkageError: loader constraint violation: when resolving method 'org.apache.hadoop.conf.Configuration org.apache.flink.runtime.util.HadoopUtils.getHadoopConfiguration(org.apache.flink.configuration.Configuration)' the class loader org.apache.flink.util.ChildFirstClassLoader @162afc80 of the current class, org/apache/flink/formats/parquet/avro/AvroParquetRecordFormat, and the class loader 'app' for the method's defining class, org/apache/flink/runtime/util/HadoopUtils, have different Class objects for the type org/apache/hadoop/conf/Configuration used in the signature (org.apache.flink.formats.parquet.avro.AvroParquetRecordFormat is in unnamed module of loader org.apache.flink.util.ChildFirstClassLoader @162afc80, parent loader 'app'; org.apache.flink.runtime.util.HadoopUtils is in unnamed module of loader 'app')
I've tried to play around with different versions of Hadoop dependencies and tried to relocate org.apache.hadoop but without success. Finally I ran out of ideas what to try and how to fix this. I'd much appreciate if someone gives me direction.
r
what is it you're wanting to do with the data? Have you considered Flink SQL?
r
Basically, I have to process records (~300M) stored in Parquet files, enrich them with some extra data stored as raw logs on another S3 storage, and supply them for further analysis. I haven't looked into Flink SQL yet, to be honest. Would you recommend looking into it?
r
if Java isn't your thing, then yes πŸ™‚ (speaking as someone for whom Java is their thing either)
r
I'm fine with Kotlin πŸ™‚
But back to the original topic, could you suggest something I could try to fix this? I believe I missed some small, stupid piece.