Hey folks,
I wanted to share an issue we recently encountered and see if anyone has experienced something similar.
We have around 30 Flink jobs running in a Kubernetes cluster using the Flink Kubernetes Operator. The setup for these jobs is a standalone application using the reactive scheduler, with high availability (HA) configured as "Kubernetes," and the HA storage directory is on S3. We’ve been running this setup in production for almost a year without any issues.
The problem started the same month we introduced the
HA configuration for the Flink Kubernetes Operator. Multiple jobs started going down after a few days, with their config maps getting deleted in Kubernetes, leaving the jobs in a failed state with no ability to recover.
Unfortunately, we couldn’t find any logs directly pointing to the issue being related to the FKO HA configuration. The only clue we have is the timing of the incidents. Since we removed the HA configuration from production, we haven't experienced any further problems. The only logs we found mentioned missing config maps and checkpoint information.
We’ve tried to replicate the issue in our development environment, but so far, we haven’t had any luck reproducing it.
Has anyone experienced something similar?