Hi everyone, we were trying to use s3 in jarURI for flinkSessionJob. But it always used to fail saying no s3 scheme implementation found for downloading jar, even though we put the hadoop-s3-fs jar in plugins directory & set HADOOP_CLASSPATH env var as well in pod. After doing all this, we saw that flink operator on startup used to log, no Hadoop dependency found on classpath. So we looked upon docker-entrypoint.sh for flink-operator and realized that hadoop_classpath env var was not added to java classpath argument. In order to fix it, we edited docker-entrypoint.sh to add out s3 jar path in classpath(-cp argument). After that we could see the hadoop-s3 jar was added on classpath(confirmed via logs) & it worked fine. I was wondering, is there any easier way to put a jar(s) on classpath in flink operator? Modifying the docker-entrypoint.sh might lead to more bugs if we upgrade operator and there are any changes in the entrypoint file. Thanks.

I have just copied all the dependencies jars in /opt/flink/lib in my Dockerfile and it worked for me (v1.17.0)

Yes for flink it works. Everything specified in lib dir is added to classpath, but I am talking about flink k8s operator. It doesn't have any lib directory afaik

I am also using flink-k8s-operator. You can build custom container image for your FlinkDeployment and copy the jar files in your Dockerfile like so

COPY lib/* /opt/flink/lib/

FROM apache/flink-kubernetes-operator:1.4.0

ARG hadoop_plugins="/opt/flink/lib"

RUN mkdir -p "${hadoop_plugins}" && cd ${hadoop_plugins} && curl -O <https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.16.1/flink-s3-fs-hadoop-1.16.1.jar>

I have this docker file, where hadoop fs jar is present in /opt/flink/lib directory. But on startup, it logs o.a.f.c.f.FileSystem [INFO ] Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available