We have long-running ingestion tasks (1 hour) that...
# general
d
We have long-running ingestion tasks (1 hour) that run on EC2 instances in an Autoscaling Group. AWS frequently rebalances our instances across AZs after a large deployment, resulting in early termination of some instances and a bit of cleanup work as the new tasks catch back up. Is there any sort of way to hand off a subgroup of tasks early - as in, tell a specific node to complete its tasks while also taking it out of rotation - as a sort of 'shutdown hook' that doesn't require up to an hour (next handoff period) to resolve, or is handoff strictly a whole-supervisor action?
j
In any version of the product, if you resubmit the Supervisor spec with no changes, it will roll all of the ingestion tasks. If you want to roll only specific tasks (or task groups if you are doing replicated ingestion), there is a new API call that will allow you to do that: https://github.com/apache/druid/pull/16310
🙌 1