This message was deleted.

sorry v24.0 actually

You can try to increase the Readiness probe time or try tailing the historical log during a restart

I'm actually having the same issue. The problem is that since it is a statefulset, the old pod is deleted before the new one comes up and so I can't see what the issue was. The logs make it look like a graceful shutdown is being forced, but I don't know by what and there is nothing in the coordinator logs to indicate that it is responsible. A failing health check could be the result of it being shut down.

you have to determine if the shutdown is due to K8s or if it is due to something at the druid level

For me its due to k8s, we saw a clear event

Mine is likely k8s too. I will figure it out, even if I have to set the readiness to 8 hours and check periodically and catch it in the act.

No it is not the same, if we incease the number of

failureThreshold

that solve the issue but it is not a good workaround...

I would say test the health endpoint yourself using kubectl port-forward, and determine if the node itself is even functioning. If it is responding to normal requests but not health checks, that's odd.

you can drop the terminationGracePeriod i think it defaults to 30, we changed all ours to something really small like low single digit seconds because we don’t have any PreStop hooks

The health endpoint responds but after a long time (few minutes)... and meanwhile we see in the log that the Historical is loading the segments... it is like if the health is not ready while all segments are not fully loaded... and as it took few minutes, k8s decided to restart the pod after the 3 default failures of the probes... Do you know what are the behavior of theses probes ? when Druid considers the live probe ok and the ready ok ?

I think you can implement a startup probe so that the other probes are not run/checked until this completes

We have something like this which works well:

startUpProbe:
        httpGet:
          path: /druid/historical/v1/readiness
          port: 8088
        # up to 20 minutes to startup before other probes run
        periodSeconds: 10
        failureThreshold: 180

    livenessProbe:
        periodSeconds: 5
        failureThreshold: 3
        httpGet:
          path: /status/health
          port: 8088

    readinessProbe:
        periodSeconds: 5
        failureThreshold: 3
        httpGet:
          path: /druid/historical/v1/readiness
          port: 8088

Good, thank you I'll look at this startUpProbe. "`readinessProbe` is not needed and redundant.", you mean we can simply remove it ?

Yeah. I'd edit it out of my message, but this slack doesn't seem to allow editing.

Ok, what is under

/druid/historical/v1/readiness

? which condition will validate this endpoint if you know ?

It is documented here https://druid.apache.org/docs/latest/operations/api-reference.html But honestly I took it from an example kubernetes config from one of the people who maintains druid-operator.