👋, we are testing Flink's reactive mode deployment and it seems to scale up and down fine when a pipeline is fresh with a few checkpoints, but after a couple of days (more than 100? checkpoints), when we try to scale up or down, the pipeline gets into a weird state where the Task Manager will get into a "Running" state briefly (a couple of seconds) before getting canceled. Combing through the logs and I can't find anything to support my hypothesis that this has to do with the number of checkpoints. But I did find this log in the Job Manager:

{
  "instant": {
    "epochSecond": 1674595600,
    "nanoOfSecond": 114000000
  },
  "thread": "flink-akka.actor.default-dispatcher-17",
  "level": "INFO",
  "loggerName": "org.apache.flink.runtime.jobmaster.JobMaster",
  "message": "Disconnect TaskExecutor .... because: The TaskExecutor is shutting down.",
  "endOfBatch": false,
  "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
  "contextMap": {},
  "threadId": 103,
  "threadPriority": 5
}

My answer to most reactive mode related questions is that probably the reactive mode is not what you want , but we have an autoscaler in the Flink Kubernetes operator now which is really nice and works without reactive 🙂

Yes! We are also in the process of moving over to the Flink Operator and long term will leverage the auto scaling efforts there :). In the meantime and in parallel testing out the reactive mode.

My answer to most reactive mode related questions is that probably the reactive mode is not what you want

We thought that pipelines would benefit regardless from bumping first to reactive mode as it allows one to easily scale up or down. It would also allow us to test the waters with more dynamic resource allocations and see if we need to make any other adjustments to our pipelines.

All in all I dont have much prod experience with the reactive mode. I personally prefer the active Kubernetes integration and “fixed” job parallelism. This also means I cannot really answer why it doesn’t work as expected for you.

Got it thank you Gyula!

thanks for the followup!

@Gyula Fóra does autoscaler with 1.17 restart JM and TM? Is it a prod ready version yet ?

The version on current operator main is more or less prod ready :) it still needs to be released as part of 1.5.0 but you can build your own operator image

that’s great news!