Hi team,
I’m looking for some suggestions on managing Pinot Broker replicas in a Kubernetes environment.
I initially deployed 3 broker replicas and then scaled it down to 1 replica directly via Kubernetes. However, in the Pinot UI, I’m seeing that the status for the two removed brokers is marked as dead, while enabled is still set to true. Additionally, these dead brokers are still being returned via service discovery, which is causing Trino to attempt connections to them—resulting in intermittent query failures.
• What is the recommended or correct procedure to scale down brokers safely in Kubernetes so they are removed cleanly from the cluster metadata?
• I came across a mention of the untag option as a possible clean-up mechanism. Is that the only way to cleanup?
Appreciate any insights or best practices on this. Thanks in advance!