Is it possible to add very common dependencies (ja...
# troubleshooting
m
Is it possible to add very common dependencies (jars) to the "/lib" directory of a Flink session cluster and forgo bundling these in application jars? I've been trying for some time to include the flink-connector-kafka this way but when I submit my jar:ed application to the cluster I get a ClassNotFound exception for KafkaSource.
Notably: https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/configuration/maven/#packaging-the-application
If you want to create a JAR for a Flink Job and use external dependencies not built into the Flink distribution, you can either add them to the classpath of the distribution or shade them into your uber/fat application JAR.
If I'm reading this correctly it should be possible to add jars into the "/lib" directory (which I assume is the classpath of the distribution)
I think I figured it out! This is what I did before to add the dependencies:
Copy code
FROM flink:1.20.1-java17

ADD https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/3.4.0-1.20/flink-connector-kafka-3.4.0-1.20.jar /opt/flink/lib/ext/.
This added the jars but it uses the root user to do so, so the flink user does not have access rights to them! This is how it looks now (and the classloaders can successfully pick up the jars!)
Copy code
FROM flink:1.20.1-java17

ADD --chown=flink --chmod=644 https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka/3.4.0-1.20/flink-connector-kafka-3.4.0-1.20.jar /opt/flink/lib/ext/.