Apache Pinot

Hi team, we’re facing a bit of a weird problem in our production setup. We call Pinot from another service (lets call it Query Service) using an ALB, which hits the broker nodeport svc. Around 50% of the time, Query Service ends up seeing a 410 error code from the broker. However, we do not get this error even once when we query the tables from Pinot’s query console.

To mitigate this, we changed the Query Service to hit only a specific broker using the K8s svc (`pinot-broker-3.broker-headless.pinot.cluster.local:8099`) and it stop seeing any errors. The rationale behind doing this was we had assigned only a single broker for the tables yet query was going to one other broker (which wasn’t a part of the tenant). Therefore, we pinned it only to the broker we wanted to use.

We faced this issue last week too and was circumvented by recreating the broker sts. However, it appeared again this week.

There are no errors in the broker resource on Zookeeper.

<@U024J35MLLV> are you using the java client for queries? Is that initialized via the ZK url or broker list?

<@U024ET8HGUQ> Yes, Java client using broker list.

As far as I understand the java client,
```Connection connection = ConnectionFactory.fromHostList(broker1, broker2)```
will load balance across the list of brokers provided. It does not check whether or not the broker actually belongs to the tenant / table being queried

Using
```Connection connection = ConnectionFactory.fromZookeeper(zkurl)```
However, will use ZK meta to maintain an updated map of tableName -&gt; broker lists and always query the right broker. This ofcourse implies that all brokers must be either publically accesible, or your app should be deployed in the same VPC as pinot

<@U011R2VMS4S> for more

Yes, in prod environments, .fromHostList should only be used when your brokers are not accessible by host:port and you need to make sure the queries being forwarded are for correct tenant.

<@U011R2VMS4S> the broker list in that case is the ALB’s endpoint. So that might route queries to broker that are not part of the tenant and therefore end up in 410?

Yeah the ALB will distribute the requests across all brokers. If your query app is deployed within the same VPC as pinot (and zookeeper), using ZK url to initialise the client may work