Niro
02/19/2025, 4:31 AMkubectl exec -it example-cluster -n flink-operator -- \
flink run -d -p 2 -c org.example.MainClass s3://zzzzz/StateMachineExample/StateMachineExample.jar
And I'm getting following error
ERROR StatusLogger Reconfiguration failed: No configuration found for '25f38edc' at 'null' in 'null'
ERROR StatusLogger Reconfiguration failed: No configuration found for '6c40365c' at 'null' in 'null'
03:57:45.270 [main] ERROR org.apache.flink.client.cli.CliFrontend - Invalid command line arguments.
org.apache.flink.client.cli.CliArgsException: Could not get job jar and dependencies from JAR file: JAR file does not exist: s3:/zzzzz/StateMachineExample/StateMachineExample.jar
at org.apache.flink.client.cli.CliFrontend.getJobJarAndDependencies(CliFrontend.java:260) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:237) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) [flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189) [flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157) [flink-dist-1.17.2.jar:1.17.2]
Caused by: java.io.FileNotFoundException: JAR file does not exist: s3:/zzzzz/StateMachineExample/StateMachineExample.jar
at org.apache.flink.client.cli.CliFrontend.getJarFile(CliFrontend.java:904) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.client.cli.CliFrontend.getJobJarAndDependencies(CliFrontend.java:256) ~[flink-dist-1.17.2.jar:1.17.2]
... 6 more
Could not get job jar and dependencies from JAR file: JAR file does not exist: s3:/zzzzz/StateMachineExample/StateMachineExample.jar
I have the correct IAM configuration in place (confirmed by testing with another pod using the same service account, which can retrieve the JAR without any issues).
You can also check the pod startup logs to get insight into what’s happening during the initialization process.
Any help would be really appreciatedNiro
02/19/2025, 4:34 AMapiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
name: {{ .Release.Name }}
namespace: {{ .Release.Namespace }}
spec:
image: flink:1.17
flinkVersion: v1_17
serviceAccount: flink
flinkConfiguration:
logging.level: DEBUG
taskmanager.numberOfTaskSlots: "2"
fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
fs.s3a.aws.credentials.provider: "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
jobManager:
resource:
memory: "2048m"
cpu: 0.5
podTemplate:
spec:
serviceAccount: flink
nodeSelector:
org/app: flink
org/node-type: flink-worker
volumes:
- name: flink-plugins
emptyDir: {}
initContainers:
- name: s3-plugin-downloader
image: busybox:uclibc
command:
- /bin/sh
- -c
- |
mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
wget -O /opt/flink/plugins/flink-s3-fs-hadoop/flink-s3-fs-hadoop-1.17.0.jar \
https://repo.maven.apache.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.17.0/flink-s3-fs-hadoop-1.17.0.jar
volumeMounts:
- name: flink-plugins
mountPath: /opt/flink/plugins/flink-s3-fs-hadoop
containers:
- name: flink-main-container
volumeMounts:
- name: flink-plugins
mountPath: /opt/flink/plugins/flink-s3-fs-hadoop
taskManager:
resource:
memory: "2048m"
cpu: 0.5
podTemplate:
spec:
serviceAccount: flink
nodeSelector:
org/app: flink
org/node-type: flink-worker
volumes:
- name: flink-plugins
emptyDir: {}
initContainers:
- name: s3-plugin-downloader
image: busybox:uclibc
command:
- /bin/sh
- -c
- |
mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
wget -O /opt/flink/plugins/flink-s3-fs-hadoop/flink-s3-fs-hadoop-1.17.0.jar \
https://repo.maven.apache.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.17.0/flink-s3-fs-hadoop-1.17.0.jar
volumeMounts:
- name: flink-plugins
mountPath: /opt/flink/plugins/flink-s3-fs-hadoop
containers:
- name: flink-main-container
volumeMounts:
- name: flink-plugins
mountPath: /opt/flink/plugins/flink-s3-fs-hadoopPedro Mázala
02/19/2025, 11:03 AMBased on the logs, it's not being able to load the jar file. It's weird because it should beJAR file: JAR file does not exist: s3:/zzzzz/StateMachineExample/StateMachineExample.jar
s3://...
This can also lead to a problem with hadoop. Check
2025-02-19 03:55:33,968 INFO org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider ] - Hadoop FS is not available (not packaged with this application): NoClassDefFoundError : "org/apache/hadoop/conf/Configuration".
Have you tried using s3a://... based on this config from your yaml fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
I'd try to build my own image as well. To add the required jars you have.
THose would be my leads 😄Niro
02/19/2025, 11:36 AMs3:/, but assumed it was just a logging thing 😀Niro
02/19/2025, 11:37 AMfs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
fs.s3a.aws.credentials.provider: "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
should I include both in my case ?Raghuram Onti Srinivasan
02/19/2025, 2:49 PM