Hi everyone, I'm facing issues while running a job...
# troubleshooting
n
Hi everyone, I'm facing issues while running a job on Flink deployed in an EKS cluster using the flink-kubernetes-operator. My setup is fairly simple, and I'm attempting to run a job using the following command.
kubectl exec -it example-cluster -n flink-operator -- \
flink run -d -p 2 -c org.example.MainClass s3://zzzzz/StateMachineExample/StateMachineExample.jar
And I'm getting following error
Copy code
ERROR StatusLogger Reconfiguration failed: No configuration found for '25f38edc' at 'null' in 'null'
ERROR StatusLogger Reconfiguration failed: No configuration found for '6c40365c' at 'null' in 'null'
03:57:45.270 [main] ERROR org.apache.flink.client.cli.CliFrontend - Invalid command line arguments.
org.apache.flink.client.cli.CliArgsException: Could not get job jar and dependencies from JAR file: JAR file does not exist: s3:/zzzzz/StateMachineExample/StateMachineExample.jar
	at org.apache.flink.client.cli.CliFrontend.getJobJarAndDependencies(CliFrontend.java:260) ~[flink-dist-1.17.2.jar:1.17.2]
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:237) ~[flink-dist-1.17.2.jar:1.17.2]
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095) ~[flink-dist-1.17.2.jar:1.17.2]
	at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189) ~[flink-dist-1.17.2.jar:1.17.2]
	at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) [flink-dist-1.17.2.jar:1.17.2]
	at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189) [flink-dist-1.17.2.jar:1.17.2]
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157) [flink-dist-1.17.2.jar:1.17.2]
Caused by: java.io.FileNotFoundException: JAR file does not exist: s3:/zzzzz/StateMachineExample/StateMachineExample.jar
	at org.apache.flink.client.cli.CliFrontend.getJarFile(CliFrontend.java:904) ~[flink-dist-1.17.2.jar:1.17.2]
	at org.apache.flink.client.cli.CliFrontend.getJobJarAndDependencies(CliFrontend.java:256) ~[flink-dist-1.17.2.jar:1.17.2]
	... 6 more
Could not get job jar and dependencies from JAR file: JAR file does not exist: s3:/zzzzz/StateMachineExample/StateMachineExample.jar
I have the correct IAM configuration in place (confirmed by testing with another pod using the same service account, which can retrieve the JAR without any issues). You can also check the pod startup logs to get insight into what’s happening during the initialization process. Any help would be really appreciated
and my helm chart like below
Copy code
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: {{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
spec:
  image: flink:1.17
  flinkVersion: v1_17
  serviceAccount: flink
  flinkConfiguration:
    logging.level: DEBUG
    taskmanager.numberOfTaskSlots: "2"
    fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
    fs.s3a.aws.credentials.provider: "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
  jobManager:
    resource:
      memory: "2048m"
      cpu: 0.5
    podTemplate:
      spec:
        serviceAccount: flink
        nodeSelector:
          org/app: flink
          org/node-type: flink-worker
        volumes:
          - name: flink-plugins
            emptyDir: {}
        initContainers:
          - name: s3-plugin-downloader
            image: busybox:uclibc
            command:
              - /bin/sh
              - -c
              - |
                mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
                wget -O /opt/flink/plugins/flink-s3-fs-hadoop/flink-s3-fs-hadoop-1.17.0.jar \
                https://repo.maven.apache.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.17.0/flink-s3-fs-hadoop-1.17.0.jar
            volumeMounts:
              - name: flink-plugins
                mountPath: /opt/flink/plugins/flink-s3-fs-hadoop
        containers:
          - name: flink-main-container
            volumeMounts:
              - name: flink-plugins
                mountPath: /opt/flink/plugins/flink-s3-fs-hadoop
  taskManager:
    resource:
      memory: "2048m"
      cpu: 0.5
    podTemplate:
      spec:
        serviceAccount: flink
        nodeSelector:
          org/app: flink
          org/node-type: flink-worker
        volumes:
          - name: flink-plugins
            emptyDir: {}
        initContainers:
          - name: s3-plugin-downloader
            image: busybox:uclibc
            command:
              - /bin/sh
              - -c
              - |
                mkdir -p /opt/flink/plugins/flink-s3-fs-hadoop
                wget -O /opt/flink/plugins/flink-s3-fs-hadoop/flink-s3-fs-hadoop-1.17.0.jar \
                https://repo.maven.apache.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.17.0/flink-s3-fs-hadoop-1.17.0.jar
            volumeMounts:
              - name: flink-plugins
                mountPath: /opt/flink/plugins/flink-s3-fs-hadoop
        containers:
          - name: flink-main-container
            volumeMounts:
              - name: flink-plugins
                mountPath: /opt/flink/plugins/flink-s3-fs-hadoop
p
JAR file: JAR file does not exist: s3:/zzzzz/StateMachineExample/StateMachineExample.jar
Based on the logs, it's not being able to load the jar file. It's weird because it should be
s3://...
This can also lead to a problem with hadoop. Check
2025-02-19 03:55:33,968 INFO  org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvider ] - Hadoop FS is not available (not packaged with this application): NoClassDefFoundError : "org/apache/hadoop/conf/Configuration".
Have you tried using
s3a://...
based on this config from your yaml
fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
I'd try to build my own image as well. To add the required jars you have. THose would be my leads 😄
n
I tried using s3a:// but got the same error. I also noticed
s3:/
, but assumed it was just a logging thing 😀
just wanted to clarify on following configuration entries
Copy code
fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
    fs.s3a.aws.credentials.provider: "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
should I include both in my case ?
r
I'd suggest the same as @Pedro Mázala. Have you tried it with the built into the image to check if it's only the s3 issue and everything else is setup correctly?
🙌 1