You have to untag that server then rebalance: https://docs.pinot.apache.org/operators/operating-pinot/rebalance/rebalance-servers

Thanks @Neha Pawar this fixed the issue. Although wondering why it didn't resolve automatically?

Rebalance is not designed to adjust automatically

On deployment update?

If the server is still in zk, and tagged with the same tag used by the table, then it will continue to be used. @Xiang Fu is there a way to achieve automatic removal of the server or rebalance when using k8 deployment? I would guess not?

no, pinot cannot figure out if this server is dead for a long time or should be recycled, it has to be human intervention, however, user can build some script or tooling to do periodically check and perform the action

This is the case where after deployment the pod ip changes

but service name doesn’t change right?

Server_192.168.18.47_8098 the instance name has the ip of the pod. Is there a way to resolve this?

you can add below config to your controller/broker/server config files

pinot.set.instance.id.to.hostname=true

@Xiang Fu this conf was already present. Any other way to avoid ip in instance names?

this is the only way

I had to add this to the server config file as well. Is it resolved now. Previously had added only in controller config