Hi Everyone!
tldr: Where can we report bugs related to Flink Kubernetes Operator?
Operator version: 1.5.0
Background: We are using FlinkSessionJob for submitting jobs to a session cluster.
Bug: We frequently encounter a problem where the job gets stuck in CREATED/RECONCILING state. On checking flink operator logs we see the error
Job could not be found. Full trace
here.
This happens when:
1. We submit a flink session job and the flink operator submit the job to Flink Cluster.
2. Now if the job manager restarts, the job no longer exists in JM.
3. When the operator now reconciles the job, the JM rest API gives 404 for that given jobID and the operator keeps logging the error. This leads to the job getting stuck.
The bigger issue is that
1. now this job can neither be restarted using restartNonce nor be suspended, because if we do so, the operator still calls the REST API and the same error is thrown.
2. If I delete the flink session job, then deletionTimestamp is added to it, but it get's stuck there. I am assuming that operator is again not able to do it's cleanup part(i.e. say cancel/delete the job) and hence not removing the finalizer from the FlinkSessionJob. PS: not very sure if this theory is correct.
Expected Behaviour: In my understanding, the operator should mark the job as failed/deleted or some other appropriate status instead of being stuck infinitely for a given job if it no longer exists in Session Cluster.