Hello all brave Flink developers and users! I am ...
# troubleshooting
n
Hello all brave Flink developers and users! I am wondering if it is only me/us, but we are facing an intermittent issue with Flink Operator and Flink Session Jobs. Versions used: Flink image version: 1.18.1 (same with 1.16.1) Flink Kubernetes Operator version: 1.8.1 (same with 1.2) Platform: OCP 4.12 (same with 4.10) Setup: We have a flink session cluster that holds 3 Flink Session jobs that pull exactly the same jar. Same code, but different configuration (topics, source/sinks) Deployment is done massively with a big helm. Configmaps, FlinkDeployments, and FlinkSessionJobs resources are installed massively into the cluster via kubectl apply -f app.yml. The problem: In roughly 7/10 of deployment attempts, sessionJob1 takes the configuration of sessionJob2 and vice versa. If we delete the flink-session-jobs and flink-deployment, sessionJobs are assigned with the correct config file. Otherwise we repeat until we observe the jobs having correct source/sinks . Question: 1. Any suggestion how to troubleshoot this? 2. Is it related to flink-operator or job-manager that may do the nasty job? 3. Can be related to our code?
p
Hi, I think I'm facing the same issue (Flink 1.17.1, Flink k8s operator 1.5.0). I'm starting pyflink jobs though, ~20 at a time. The created FlinkSessionJob resources are correct, some of them are starting the wrong job though (different topics). So I end up not having some jobs at all, and some are duplicated. Did you find a fix for this issue?
n
Hi Peter, Already it feels better I am not the only one here. πŸ™‚ Not really, we switched to deployment waves where each job is deployed to a separate wave. This looks like it mitigates the issue. But I don't consider it a solution.
p
I see, thanks