We scaled up our brokers and saw that for about 15...
# troubleshootinge
Elon
03/22/2021, 10:26 PMWe scaled up our brokers and saw that for about 15 minutes some brokers were returning "RESOURCE_MISSING_ERROR" - when I looked in the zk browser for INSTANCES/broker-<x>/CURRENTSTATES it appears like the END_TIME values spanned that 15 minute interval. To avoid this should users get their client by contacting zookeeper, i.e. brokers for table?
Elon
03/22/2021, 10:27 PMOr should this not have happened?
Elon
03/22/2021, 10:28 PMCurrently users just set the pinot broker hostname to the kubernetes service, and there is only 1 broker tenant.
m
Mayank
03/22/2021, 10:29 PM
Typically, there should be a load-balancer/vip in front of the brokers that should listen to external view to ensure that brokers are ONLINE before it can route requests.
Mayank
03/22/2021, 10:29 PM
But 15 minutes seems like a long time
e
Elon
03/22/2021, 10:47 PMBut if the broker is up but not yet ready to serve requests how do we tell? We just use the readiness probe from the k8s chart.
m
Mayank
03/22/2021, 10:47 PM
Broker is ready when it is ONLINE on external view.
e
Elon
03/22/2021, 11:01 PMThanks - so maybe we are getting our clients incorrectly then? i.e. just using the k8s service (which only adds endpoints when they are online). Judging from the messages in zk browser:INSTANCES/<broker>/CURRENT_STATES the earliest table online occurred 17 mins before the last one