Given the context and the error details, it appears that the root cause of your issue lies in the inability to locate the JAR file necessary for executing one of the Flink jobs on your cluster. Here are some targeted steps to troubleshoot and potentially resolve this issue in your production environment: Immediate Steps 1. Check the Temporary Directory: Given the path

/tmp/flink-web-b7c8bf9d-504f-4c17-bf1e-b10ce5f8b242/flink-web-upload/

, verify if the directory exists and if there are any recent changes in your system’s cleanup policies that might be deleting files from

/tmp

. Some systems automatically clean up

/tmp

directories, which could be causing the issue. 2. Job Submission Process: Investigate how jobs are being submitted to the Flink cluster. Ensure that the JAR file is correctly uploaded and referenced during job submission. If you’re using a web interface or REST API, check the logs for any hints about failed uploads or misconfigurations. 3. Disk Space: Confirm that there is enough disk space on the nodes where Flink is running. A full disk can prevent file uploads and cause similar errors. Long-term Resolutions & Best Practices 1. Customize Flink’s Blob Server Path: Instead of using the default

/tmp

directory, consider configuring a more persistent storage location for the Flink blob server. You can do this by setting the

blob.server.base-dir

property in your Flink configuration file (typically

flink-conf.yaml

). Choose a directory that is less likely to be cleaned up automatically and has ample space. 2. Job Management: Implement checks before job submission to ensure that the required JAR files are present and accessible. This can be done as a pre-flight check in your job submission scripts or application logic. 3. Monitoring & Logging: Enhance monitoring around the job submission process and the blob server activity. Use Flink’s built-in metrics and consider integrating with your monitoring system to get alerts when the disk space is running low or when job submissions fail due to missing files. 4. Resource Management: Review your resource allocation, particularly for the JobManager and TaskManagers. Insufficient resources can lead to failures in managing or executing jobs. 5. Version Compatibility: Ensure that the version of Flink used for compiling the JAR matches the version running in the cluster. Mismatched versions can sometimes cause unexpected behavior.

Hey Draco!! Thanks for responding. I am using https://pkg.go.dev/k8s.io/api (k8s.io/client-go v0.27.2) in a go service by creating a FlinkSessionJob object and submit it to the k8sClient as a resource. The job submission is picked up by our Flink Operator in the cluster and then it downloads the jar from the jarUri mentioned in the SessionObject and submits it to the cluster. It’s not every time we are running into this issue. Sometimes for the same jar it goes through after few retries but sometimes it never goes through resulting in permanent failure of job execution. For points 2,3 : yes there is enough disk space and no failures in job submission because I can see the resource flinkSessionJob getting created. (so job submission isn’t failing) Point 5 : I have this check already in place Point 7 : There is enough memory at least 50% of it is available of JM. Point 8 : It is the same version in both places.

Ok, Given your additional context and the measures you’ve already taken, I would adjust the troubleshooting steps a bit. 1. Jar Retrieval and Network: Since the JAR is fetched from a

jarUri

specified in the FlinkSessionJob object, ensure that the network path or storage where the JAR is hosted is reliable and consistently accessible. Check for intermittent connectivity issues or any potential rate limits that may be causing sporadic failures in fetching the JAR. 2. Concurrency and Resource Contention: With 250 jobs per day, it’s possible that there are moments of high concurrency where multiple jobs are attempting to upload or access their respective JARs simultaneously. This could lead to race conditions, especially if the underlying storage or network infrastructure has limitations. Look into the patterns of job submissions and see if there are specific time windows where failures are more common, indicating a resource contention issue. 3. Operator and Flink Cluster Health: Verify the health and log files of your Flink Operator. If the operator is experiencing any issues like restarts, backpressure, or delays in processing the FlinkSessionJob objects, it could contribute to the observed behavior. Ensure the operator has sufficient resources and is not overwhelmed during peak submission times. 4. Blob Server Configuration in Kubernetes: While you’re looking into customizing Flink’s blob server path, also ensure that Kubernetes volume configurations for the Flink cluster are set up to handle the load and are resilient. If you’re using persistent volumes, check their reclaim policy and availability to avoid data loss. 5. Flink JobManager Logs: Dive deeper into the JobManager logs for any related errors or warnings that might occur just before or after these failures. Specifically, look for any clues related to the blob server, network timeouts, or file system operations. 6. Retry Logic and Backoff Strategies: Since you mentioned retries help sometimes, consider implementing a more sophisticated retry strategy in your submission process with exponential backoff. This can help mitigate temporary issues without overwhelming the system. 7. Temporary File Cleanup Policy: If the Flink operator or any external process is responsible for cleaning up temporary files, ensure that it’s not prematurely deleting files needed for currently running or about-to-be-submitted jobs. Adjust the cleanup policies if necessary. 8. Flink Operator Version Compatibility: Double-check not only the Flink version compatibility between client and cluster but also ensure that the Flink operator version you’re using is compatible with both your Kubernetes version and the Flink cluster version. By systematically looking into these areas, you might be able to identify the root cause