Hello all brave Flink developers and users!
I am wondering if it is only me/us, but we are facing an intermittent issue with Flink Operator and Flink Session Jobs.
Versions used:
Flink image version: 1.18.1 (same with 1.16.1)
Flink Kubernetes Operator version: 1.8.1 (same with 1.2)
Platform: OCP 4.12 (same with 4.10)
Setup:
We have a flink session cluster that holds 3 Flink Session jobs that pull exactly the same jar. Same code, but different configuration (topics, source/sinks)
Deployment is done massively with a big helm. Configmaps, FlinkDeployments, and FlinkSessionJobs resources are installed massively into the cluster via kubectl apply -f app.yml.
The problem:
In roughly 7/10 of deployment attempts, sessionJob1 takes the configuration of sessionJob2 and vice versa. If we delete the flink-session-jobs and flink-deployment, sessionJobs are assigned with the correct config file. Otherwise we repeat until we observe the jobs having correct source/sinks .
Question:
1. Any suggestion how to troubleshoot this?
2. Is it related to flink-operator or job-manager that may do the nasty job?
3. Can be related to our code?