Unless you've compiled Flink yourself with a different Scala version

Yes

Then it would be best to reach out to Delta

@Krzysztof Chmielewski ?

also is Delta connector in your case a part of your Uber Jar or did you added this as provided dep and shipped to the cluster?

I myself was also running Flink test jobs with Delta connector on various Flink versions. Currently (connector version 0.6.0) has dependency on Flink 1.15 but I know people are using it also with Flink 1.16. You can double check your dependencies comparing with: https://github.com/delta-io/connectors/blob/master/examples/flink-example/pom.xml

Let me know @Neha if any questions

delta-standalone_2.12-0.6.0. is build based on Scala 2.12.8 but we dont ship scala with this library. So maybe there is binary incompatibility between 2.12.7 that is on Flink and 2.12.8 that is expected by delta-standalone_2.12-0.6.0 lib. I'm thinning that maybe when uber jar is created with Delta Connector, scala is added to that jar and this is why there is no exception like that when submitting uber jar [1]. Is submitting an uber jar a ok strategy in your case? I will take a look at this case and let you know. BTW you can open an issue here [2] and tag me @kristoffSC [1] https://github.com/delta-io/connectors/blob/master/examples/flink-example/pom.xml [2] https://github.com/delta-io/connectors/issues

yeah its a good question - maybe when submitting an uber jar, Flink uses different class loader that puts classes on different class path? So Flink Job sees classes that are coming from Flink's class path + from uber jar. The classes from uber jar can override classes from Flink -> on Job's class path ofc.

So Flink job for Delta Connector uses class path with its dependencies (with scala from uber Jar) But when you submit individual dependencies on Cluster then standalone library has Flink's Scala on classpath hence the issue.

I'm not 100% sure if that's the case but I would think it might be, @Martijn Visser could you comment on my theory about class loaders and class paths?

please check for "AllDependency" uber jar

@Neha Sory, I asked before but I dont think you answered it. Is your use case allow for submitting your jobs using "uberJar" strategy instead shipping delta connector to the cluster? If yest then you could use the example pom.xml Ive send in previous message.

Just for me, could you share why you cannot use uberJar Strategy?

I would say that what was removed was different versions of scala for flink dependencies for example Flink table-planner before Flink 1.15 was shipped for scala 2.11 and 2.12 - two artifacts. Now there is only one artefact that expects scala 2.12 runtime. So flink still uses scala for its internal stuff (Table planner for example) and it expects Scala runtime. Scala is shipped with flink runtime (I think) so client does not need to bother with that.

Hi @Neha I think I have something for you. Please follow steps below: 1. remove all libraries that you added to flink while trying to make connector to work. You have send list of them in one of the first messages. I believe it was this https://apache-flink.slack.com/archives/C03G7LJTS2G/p1677595893063609?thread_ts=1677578890.965759&cid=C03G7LJTS2G 2. Create a new java project. If you are using maven, then pom.xml is attached to this message. Please double check versions if they match the one you are using. 3. Execute

mvn clean package

on this new project. This should give you a new artifact

DeltaAllDep-1.0-SNAPSHOT-shaded.jar

. Add this jar to Flink cluster (under flink's lib folder) 4. In your Flink Job project, where you want to use Delta Connector mark delta-standalone and delta-flink dependencies as <scope>provided</scope> Something like this:

<dependency>
 <groupId>io.delta</groupId>
 <artifactId>delta-flink</artifactId>
 <version>${connectors.version}</version>
 <scope>provided</scope>
</dependency>
<dependency>
 <groupId>io.delta</groupId>
 <artifactId>delta-standalone_${scala.main.version}</artifactId>
 <version>${connectors.version}</version>
 <scope>provided</scope>
</dependency>

Build Flink Job jar. 5. Submit jar from point 4 to Flink cluster. It should not contain delta libraries. I've tried this on locally on Docker based Flink cluster and it seems to do the job. I was able to submit "vanilla" Delta Job.

@Neha did it worked for you?

FYI I've created an Issue for this on Delta's Github. I will be discussing this today. I will propose to add this solution to the examples and README.md https://github.com/delta-io/connectors/issues/512