Hi folks, I'm under migration to Apache Flink Ope...
# troubleshooting
о
Hi folks, I'm under migration to Apache Flink Operator with FlinkDeployments and everything is working fine in case when all jars are located in /lib folder of docker image (I'm not using fat jar), but when I moving job's and dependency jars into /usrlib - it doesn't work because couldn't found job's jar. Old deployment way (helm chart with deployment) is working fine. Maybe I need to tell somehow to operator (or in flink config) that jars are divided between /lib and /usrlib folders? Any suggestions, please In logs, I see that classpath contains only jars from /lib, but no one from /usrlib. Classpath: /opt/flink/lib/flink-cep-1.19.1.jar/opt/flink/lib/flink connector files 1.19.1.jar/opt/flink/lib/flink-csv-1.19.1.jar/opt/flink/lib/flink json 1.19.1.jar/opt/flink/lib/flink-scala_2.12-1.19.1.jar/opt/flink/lib/flink table api java uber 1.19.1.jar/opt/flink/lib/flink-table-planner-loader-1.19.1.jar/opt/flink/lib/flink table runtime 1.19.1.jar/opt/flink/lib/jackson-annotations-2.15.3.jar/opt/flink/lib/jackson core 2.15.3.jar/opt/flink/lib/jackson-core-asl-1.9.2.jar/opt/flink/lib/jackson databind 2.15.3.jar/opt/flink/lib/log4j-1.2-api-2.17.1.jar/opt/flink/lib/log4j api 2.17.1.jar/opt/flink/lib/logback-classic-1.2.11.jar/opt/flink/lib/logback core 1.2.11.jar/opt/flink/lib/logstash-logback-encoder-5.3.jar/opt/flink/lib/flink dist 1.19.1.jar:/etc/hadoop/conf Flink version: 1.19.1 Apache Flink Operator version: 1.11.0 Application & Standalone Mode
p
Perhaps you need to add /usrlib on Flink's classpath.
👀 1
о
I tried (one a time) to set via flinkConfiguration: 1. env.java.opts.all: "--classpath .../usrlib/" ; 2. env.java.opts.all: "-Denv.classpath=.../usrlib/" ; 3. pipeline.classpaths: "file:///opt/flink/usrlib/" but it didn't work
p
What is your deployment model? You need to set the image on the operator as well. Perhaps your TM/JM does not have those classes under /usrlib. Can you check this?
о
I build job's docker image and added usrlib folder with bunch of jars including job's jar with main class inside. Using it for FlinkDeployment k8s resource. in FlinkDeployment set jarURI and entryClass as well and they are inside running pod in right path. jarURI: file:///opt/flink/usrlib/my-job-1.0.1.jar className: com.jobs.myjob.MyJob for mode I'm also using mode: standalone and it's Application deployment (not Session cluster)
only one thing.. usrlib has root:root owner (not flink:flink as lib) but with read permissions.
p
Have you checked if the task manager is being spin up with the expected image? I had a similar issue like this one before.
Perhaps also the owner/group may be a lead
о
it works if to change mode to native - in this case it starts jobmanager first, looks like copying job's jar into artifactory folder and than runs taskmanager.
but doesn't work with mode: standalone. Maybe native mode will be fine for me, but if any ideas how to make work in standalone - will be great!
j
Class loading is controlled by the constructClassPath function in the /opt/flink/bin/config.sh and I don't see /usrlib there. To my understanding, it is just a custom folder to keep application jars.
👍 1