Hi Druid experts! Wanted to get some feedback arou...
# general
a
Hi Druid experts! Wanted to get some feedback around a potential behavior change for the /druid/indexer/v1/supervisor endpoint. The current behavior when an ingestion spec is submitted is to stop then start the supervisor. Regardless of whether the underlying spec actually changed, the supervisor is always restarted. The behavior change we wanted to introduce is to first check if the spec has changed. If it hasn't, then do nothing. Otherwise just proceed with the existing behavior. Our use case is that we have some automation which hits the supervisor endpoint to update the schema in the ingestion spec. It's often unclear which specs have changed, so we would like to be able to just submit all of them to the supervisor endpoint and have it be restarted only if the spec has changed. Please let me know your thoughts, and happy to contribute if we think this is valid behavior
s
I upvote this feature. Actually I wrote and maintain a script on my app that checks the changes in the spec and post it only if it has changes.
j
I have on occasion resubmitted an unchanged Supervisor in order to roll tasks ... so if you do this then I request that you add a mechanism to allow us to force the original behavior ... e.g. "suspended: restart"?
a
Agreed @John Kowtko I'm thinking we can just add a QueryParam that allows us to keep the default behavior the same, but optionally enable the behavior to not restart if the spec is the same
m
We also maintain our own script/logic to check if spec has changed and only post it if it has changes.
h
My pipeline deploys all everytime. Didn't bother too much, as I have a super aggressive compacting pipeline (first stage after 6h). But yeah, that's a nice feature. Can we get a flag to force the restart in case the old behavior is still desired (I used it yesterday for a rebalance after changing the number of instances)
a
In case folks are interested I've put up a first draft for this change here: https://github.com/apache/druid/pull/17707 Will be adding unit tests next week. Please let me know your thoughts!