Hi Team, I’m facing an issue with the Trino Pinot...
# troubleshooting
s
Hi Team, I’m facing an issue with the Trino Pinot connector in a Kubernetes environment, and I could use some assistance. Setup: • Pinot Cluster: Deployed via Helm with ZooKeeper, initially with 3 broker pods, but I’ve scaled it down to 1 broker pod. • Trino Configuration: Using the Pinot connector with both
pinot.controller-urls
and
pinot.broker-url
set in the catalog properties. Problem: Although I’ve pointed the
pinot.broker-url
to the active Pinot service (which should resolve to the single active broker), Trino seems to ignore this setting and continues to try connecting to the brokers via the headless service, including the brokers that have been scaled down. Has anyone encountered a similar issue where Trino continues to use a headless service instead of the defined broker-url? Any advice on ensuring Trino only connects to the active broker after scaling down? Thanks in advance for your help! Configuration:
Copy code
pinot.controller-urls=pinot-controller.pinot.svc.cluster.local:9000
pinot.broker-url=pinot-broker.pinot.svc.cluster.local:8000
Error:
Copy code
trino> select * from pinot.default.airlinestats;
Query 20250429_210757_00092_aich9 failed: Failed communicating with server: http://pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStats

trino> select * from pinot.default.airlinestats;
Query 20250429_210805_00094_aich9 failed: Failed communicating with server: http://pinot-broker-1.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStats
x
Is Trino deployed in same k8s cluster? If not, you need to make sure all internal controller/broker/servers dns resolvable to external trino.
it’s clear from the logs
Copy code
Failed communicating with server: http://pinot-broker-1.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStats
that trino cannot access to
pinot-broker-1.pinot-broker-headless.pinot.svc.cluster.local
If you are using StarTree cloud, there is a way to use pinot proxy for Trino: https://dev.startree.ai/docs/get-started-with-startree-cloud/connect-to-startree-cloud/connect-to-trino
s
Trino and Pinot are deployed in the same cluster. Trino is not able to communicate with
pinot-broker-1.pinot-broker-headless.pinot.svc.cluster.local
because the pod is scaled down and not running any more. Even when I gave the broker-url it is not using that. It is going back to using the service discovery.
x
then use service name should work
you can config just controller
s
Exactly, I am using the broker service name for
broker-url
and controller service for
controller-urls
as you can see below. But if you see the error it is not using the broker service url I am providing, it is using headless service of the broker.
Copy code
pinot.controller-urls=pinot-controller.pinot.svc.cluster.local:9000
pinot.broker-url=pinot-broker.pinot.svc.cluster.local:8000
Error:
Copy code
trino> select * from pinot.default.airlinestats;
Query 20250429_210757_00092_aich9 failed: Failed communicating with server: http://pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStats

trino> select * from pinot.default.airlinestats;
Query 20250429_210805_00094_aich9 failed: Failed communicating with server: http://pinot-broker-1.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStats
x
but even for this if your trino and k8s are in the same cluster, then dns resolution should work
can you check your coredns
when you call pinot broker service url, are you going to hit broker-1 or broker-2?
can you do
kubectl get all -A
and paste the response
s
I am working with a service example:
pinot-broker.pinot.svc.cluster.local:8000
. Initially, I deployed the service with three broker pods: -
pinot-broker-0.pinot-broker.pinot.svc.cluster.local
-
pinot-broker-1.pinot-broker.pinot.svc.cluster.local
-
pinot-broker-2.pinot-broker.pinot.svc.cluster.local
In this configuration, everything worked as expected. I was able to retrieve data successfully for every query submitted. The service distributed requests across the three pods seamlessly. I scaled down the deployment to a single replica. This removed two of the pods, leaving only: -
pinot-broker-0.pinot-broker.pinot.svc.cluster.local
After scaling down, some queries are still attempting to route to the removed pods (
pinot-broker-1
and
pinot-broker-2
) intermittently. This behavior results in errors, as these pods are no longer active or reachable.
x
have you tried to restart trino pods?
is it due to dns cache locally?
if when you call broker service healthcheck endpoint internally and you can always get OK, then it means the service dns resolution is good
what could go wrong is trino side dns cache
or it caches some thing internally like broker discovery etc
s
Yes, I did uninstall and re-install Trino. When I run the query using curl from the Trino pod it is working without any issues. Example:
curl http://pinot-broker-headless.pinot.svc.cluster.local:8000/query/sql -H 'Authorization: Basic xxxxxx  -d '{"sql":"SELECT * FROM xxxxxxx"}'
x
ok
then it must be trino side issue then
do you see issue when you remove broker-url config
s
It is behaving the same with or without broker-url config.
x
ok
s
We upgraded to version 474 to address this issue, as a fix for it was released but we are still encountering the same problem.
x
it might be possible that controller tried to get all the broker instances not all online instances, you might need to check the trino pinot connector code to validate
I somehow feel this
broker-url
config is not honored?
I cannot tell without reading the code
s
I feel the same.
x
seems pinot side is ok, and it’s trino side caching info
better to quickly check the trino side source code, that would be the fastest way
s
I am looking into it now.