Hello team, I'm looking for some suggestions on how to setup graceful shutdown properly, on brokers and servers, similar as how trino/presto works. Out Pinot cluster is deployed in K8S, to avoid downtime when a pod is restarted, • a server node need to commit any consuming segments, and mark as inactive to avoid new queries coming; • a broker is marked as inactive to avoid new queries, and wait until active queries are finished. cc @Grace Lu

The existing shutdown of components should already be graceful and rolling restart will guarantee zero-downtime. You don’t have to explicitly orchestrate this.

• Servers don’t need to commit consuming segments, they will re-consume from previous checkpoint when it comes up.

I'm thinking that adding a pre-stop hook to commit consuming segments forcefully would make it more stable. Currently our k8s cluster run daily node churn so pod restart event happens quite often. During the minutes we notice that table is shown unhealthy, data staleness and large consuming spikes which we want to avoid, queries are getting slow as well. The flush settings are:

"realtime.segment.flush.threshold.time": "4h",
      "realtime.segment.flush.threshold.size": "100M",
      "realtime.segment.flush.desired.size": "50M",

Why do you think force flush will make it more stable? Consumption spike can be reduced by rate limit of consumption