https://flink.apache.org/ logo
Join Slack
Powered by
Team, We've baked JM and TM pod image to have jar...
# troubleshooting
j
Team, We've baked JM and TM pod image to have jar files which are required for flinksessionjob submission. We're submitting jobs using 
jarURI: "file:///opt/flink/integration_testing_ui_demo.jar".
 Operator is failing to find the jar inside the JM/TM pod to bring up TM pods even though jar is present within image. I could exec and see the jar. Can someone help to find the cause Operator logs:
Copy code
2025-11-06 06:36:55,168 o.a.f.k.o.r.ReconciliationUtils [WARN ][ncm-sc/nc-int] Attempt count: 11, last attempt: false
2025-11-06 06:36:55,259 o.a.f.r.u.c.m.ProcessMemoryUtils [INFO ][ncm-sc/fsc-flink-cluster] The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2025-11-06 06:36:55,362 o.a.f.r.u.c.m.ProcessMemoryUtils [INFO ][ncm-sc/fsc-flink-cluster] The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2025-11-06 06:36:55,459 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][ncm-sc/fsc-flink-cluster] Resource fully reconciled, nothing to do...
2025-11-06 06:36:55,469 i.j.o.p.e.EventProcessor       [ERROR][ncm-sc/nc-int] Error during event processing ExecutionScope{ resource id: ResourceID{name='nc-int', namespace='ncm-sc'}, version: 48211671}
org.apache.flink.kubernetes.operator.exception.ReconciliationException: java.io.FileNotFoundException: /opt/flink/integration_testing_ui_demo.jar (No such file or directory)
	at org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:130)
	at org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:58)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:153)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:111)
Exec to JM Pod: I have jar inside
Copy code
❯ kubectl exec -it -n ncm-sc fsc-flink-cluster-6c47f5964b-t5jhb -- bash
Defaulted container "flink-main-container" out of: flink-main-container, setup-certs-and-plugins (init)
root@fsc-flink-cluster-6c47f5964b-t5jhb:/opt/flink# ls
'${sys:log.file}'   bin    examples                          lib       licenses   NOTICE   plugins        README.txt
 artifacts          conf   integration_testing_ui_demo.jar   LICENSE   log        opt      pod-template
Describe flinksession job
Copy code
❯ kubectl describe flinksessionjob nc-int -n ncm-sc | tail -n 20

Status:
  Error:  {"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.io.FileNotFoundException: /opt/flink/integration_testing_ui_demo.jar (No such file or directory)","additionalMetadata":{},"throwableList":[{"type":"java.io.FileNotFoundException","message":"/opt/flink/integration_testing_ui_demo.jar (No such file or directory)","additionalMetadata":{}}]}
  Job Status:
    Checkpoint Info:
      Last Periodic Checkpoint Timestamp:  0
    Job Id:                                f72f53c6355212276b75452aa2dc376e
    Savepoint Info:
      Last Periodic Savepoint Timestamp:  0
      Savepoint History:
  Lifecycle State:      UPGRADING
  Observed Generation:  2
  Reconciliation Status:
    Last Reconciled Spec:      {"spec":{"job":{"jarURI":"file:///opt/flink/integration_testing_ui_demo.jar","parallelism":1,"entryClass":"com.beam.screaltime.worker.ncm.process_functions.NcmEventsJob","args":["--jobType","cdc","--cloudProvider","nx","--buildId","dev1-6404","--operatorParallelism","{\"default\":1}"],"state":"suspended","savepointTriggerNonce":null,"initialSavepointPath":null,"checkpointTriggerNonce":null,"upgradeMode":"stateless","allowNonRestoredState":null,"savepointRedeployNonce":null,"autoscalerResetNonce":null},"restartNonce":1,"flinkConfiguration":null,"deploymentName":"fsc-flink-cluster"},"resource_metadata":{"apiVersion":"flink.apache.org/v1beta1","firstDeployment":true}}
    Reconciliation Timestamp:  1762411304062
    State:                     UPGRADING
Events:
  Type     Reason               Age                  From  Message
  ----     ------               ----                 ----  -------
  Warning  SessionJobException  7m5s (x19 over 23m)  Job   /opt/flink/integration_testing_ui_demo.jar (No such file or directory)
  Normal   Submit               7m5s (x19 over 23m)  Job   Starting deployment
l
Hello, Please try with jarURI: local:///opt/flink/integration_testing_ui_demo.jar
j
That also failed. So, kept jar within operator image in same location by rebuilding from the given flink operator image. That worked with 
file://
Operator was looking jar in its local path
Copy code
2025-11-06 06:36:54,080 o.a.f.k.o.c.FlinkSessionJobController [INFO ][ncm-sc/nc-int] Starting reconciliation
2025-11-06 06:36:54,093 o.a.f.k.o.r.ReconciliationUtils [INFO ][ncm-sc/nc-int] Clearing last reconciled spec after initial deploy failure
2025-11-06 06:36:54,093 o.a.f.k.o.o.s.FlinkSessionJobObserver [INFO ][ncm-sc/nc-int] Skipping observe before resource is deployed during upgrade
2025-11-06 06:36:54,178 o.a.f.k.o.l.AuditUtils         [INFO ][ncm-sc/nc-int] >>> Status[Job]      | Error   | FAILED          | {"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.io.FileNotFoundException: /opt/flink/integration_testing_ui_demo.jar (No such file or directory)","additionalMetadata":{},"throwableList":[{"type":"java.io.FileNotFoundException","message":"/opt/flink/integration_testing_ui_demo.jar (No such file or directory)","additionalMetadata":{}}]}
2025-11-06 06:36:54,258 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][ncm-sc/nc-int] Deploying for the first time
2025-11-06 06:36:54,365 o.a.f.r.u.c.m.ProcessMemoryUtils [INFO ][ncm-sc/fsc-flink-cluster] The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2025-11-06 06:36:54,458 o.a.f.k.o.l.AuditUtils         [INFO ][ncm-sc/nc-int] >>> Status[Job]      | Info    | UPGRADING       | The resource is being upgraded
2025-11-06 06:36:54,560 o.a.f.r.u.c.m.ProcessMemoryUtils [INFO ][ncm-sc/fsc-flink-cluster] The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2025-11-06 06:36:54,564 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][ncm-sc/fsc-flink-cluster] Resource fully reconciled, nothing to do...
2025-11-06 06:36:54,658 o.a.f.k.o.l.AuditUtils         [INFO ][ncm-sc/nc-int] >>> Event[Job]       | Info    | SUBMIT          | Starting deployment
2025-11-06 06:36:54,763 o.a.f.r.u.c.m.ProcessMemoryUtils [INFO ][ncm-sc/fsc-flink-cluster] The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2025-11-06 06:36:54,958 o.a.f.k.o.a.ArtifactManager    [INFO ][ncm-sc/nc-int] Created dir: /opt/flink/artifacts/ncm-sc/fsc-flink-cluster/nc-int
2025-11-06 06:36:54,959 o.a.f.r.u.c.m.ProcessMemoryUtils [INFO ][ncm-sc/fsc-flink-cluster] The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
Above operator logs with error So, we kept jar in operator image. That worked. Can someone let us know if this is Okay to use this way
First Approach: Built the JobManager/TaskManager pod image with the JAR included. Deployment failed to create the FlinkSessionJob, as the operator attempted to locate the JAR within its own local path. Second Approach: Used the standard Flink operator image and rebuilt it to include the JAR inside the operator pod image. After redeploying the operator, the FlinkSessionJob came up successfully when the JAR was present in both the operator image and the JM/TM pod images. Third Approach: Rebuilt the JM/TM images without the JAR and redeployed the cluster while keeping the JAR only within the operator image. The FlinkSessionJob was still successfully created, even though the JM/TM pod images did not contain the JAR.
Conclusion: This confirms that when using the filesystem-based approach, the JAR must be present within the operator image to successfully create and launch new FlinkSessionJobs.
l
I've worked before with flink-operator to run multiple jobs. flink-operator and flink images are two different things. My setup was: • Deploy flink-operator with flink-operator upstream image. • Deploy the CR FlinkDeployment with my own flink image. CR configured with a initContainer to pull the job jar from s3 bucket, mount a volume in the container. Then run flink job using that jar and flink image, using jarURI: local:///
j
Here is the error for local
Copy code
Last Reconciled Spec:      {"spec":{"job":{"jarURI":"local:///opt/flink/integration_testing_ui_demo.jar","parallelism":1,"entryClass":"com.beam.screaltime.worker.ncm.process_functions.NcmEventsJob","args":["--jobType","cdc","--cloudProvider","nx","--buildId","dev1-6404","--operatorParallelism","{\"default\":1}"],"state":"suspended","savepointTriggerNonce":null,"initialSavepointPath":null,"checkpointTriggerNonce":null,"upgradeMode":"stateless","allowNonRestoredState":null,"savepointRedeployNonce":null,"autoscalerResetNonce":null},"restartNonce":1,"flinkConfiguration":null,"deploymentName":"fsc-flink-cluster"},"resource_metadata":{"apiVersion":"<http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>","firstDeployment":true}}
    Reconciliation Timestamp:  1762409488972
    State:                     UPGRADING
Events:
  Type     Reason               Age                  From  Message
  ----     ------               ----                 ----  -------
  Warning  SessionJobException  23s (x5 over 3m23s)  Job   Could not find a file system implementation for scheme 'local'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please see <https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/>.
  Normal   Submit               23s (x5 over 3m23s)  Job   Starting deployment
Open in Slack
PreviousNext
Powered by