Hello team, we are running into a correctness bug caused by flink k8s operator + autoscaling. Specifically, when kubernetes.operator.job.drain-on-savepoint-deletion is set to true, the operator uses a draining savepoint to stop the pipeline, then resumes from it with modified parallelism overrides. I think this is wrong behavior. Per Flink document, "If you want to resume the job at a later point in time, then do not drain the pipeline because it could lead to incorrect results when the job is resumed". I just want to confirm this is indeed unintended behavior. I can contribute a PR.

I have not encountered this behavior. It should only trigger this way when you delete the flinkdeployment - not when it changes scale. which flink + operator version are you using? which scaler? (reactive/adaptive)

Operator 1.13, Flink 2.0.1 . I am using

jobmanager.scheduler: default

(because we need a bunch of features not implemented by reactive/adaptive), therefore the operator needs to realize scaling by taking/restoring savepoints.

There's not a lot of documentation on the default scheduler - is it mature for prod workloads? Is it actively maintained? Do you know what's missing in the adaptive/reactive that you need? it might have been resolved in 1.14 or 1.15

I think the default scheduler is as vanilla as you can get with Flink, so I would assume it is mature since very long time ago. Specifically the most important feature we are missing from adaptive/reactive is fine grained resource management, it's a restriction on Flink side (not operator side), and the ticket hasn't got much traction: issues.apache.org/jira/browse/FLINK-29344 . Happy to provide more context.