SP
04/29/2025, 10:00 PMpinot.controller-urls and pinot.broker-url set in the catalog properties.
Problem: Although I’ve pointed the pinot.broker-url to the active Pinot service (which should resolve to the single active broker), Trino seems to ignore this setting and continues to try connecting to the brokers via the headless service, including the brokers that have been scaled down.
Has anyone encountered a similar issue where Trino continues to use a headless service instead of the defined broker-url? Any advice on ensuring Trino only connects to the active broker after scaling down?
Thanks in advance for your help!
Configuration:
pinot.controller-urls=pinot-controller.pinot.svc.cluster.local:9000
pinot.broker-url=pinot-broker.pinot.svc.cluster.local:8000
Error:
trino> select * from pinot.default.airlinestats;
Query 20250429_210757_00092_aich9 failed: Failed communicating with server: http://pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStats
trino> select * from pinot.default.airlinestats;
Query 20250429_210805_00094_aich9 failed: Failed communicating with server: http://pinot-broker-1.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStatsXiang Fu
Xiang Fu
Failed communicating with server: http://pinot-broker-1.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStats
that trino cannot access to pinot-broker-1.pinot-broker-headless.pinot.svc.cluster.localXiang Fu
SP
04/30/2025, 2:57 PMpinot-broker-1.pinot-broker-headless.pinot.svc.cluster.local because the pod is scaled down and not running any more. Even when I gave the broker-url it is not using that. It is going back to using the service discovery.Xiang Fu
Xiang Fu
Xiang Fu
SP
04/30/2025, 3:06 PMbroker-url and controller service for controller-urls as you can see below. But if you see the error it is not using the broker service url I am providing, it is using headless service of the broker.
pinot.controller-urls=pinot-controller.pinot.svc.cluster.local:9000
pinot.broker-url=pinot-broker.pinot.svc.cluster.local:8000
Error:
trino> select * from pinot.default.airlinestats;
Query 20250429_210757_00092_aich9 failed: Failed communicating with server: http://pinot-broker-2.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStats
trino> select * from pinot.default.airlinestats;
Query 20250429_210805_00094_aich9 failed: Failed communicating with server: http://pinot-broker-1.pinot-broker-headless.pinot.svc.cluster.local:8000/debug/routingTable/airlineStatsXiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
kubectl get all -A and paste the responseSP
04/30/2025, 4:42 PMpinot-broker.pinot.svc.cluster.local:8000.
Initially, I deployed the service with three broker pods:
- pinot-broker-0.pinot-broker.pinot.svc.cluster.local
- pinot-broker-1.pinot-broker.pinot.svc.cluster.local
- pinot-broker-2.pinot-broker.pinot.svc.cluster.local
In this configuration, everything worked as expected. I was able to retrieve data successfully for every query submitted. The service distributed requests across the three pods seamlessly.
I scaled down the deployment to a single replica. This removed two of the pods, leaving only:
- pinot-broker-0.pinot-broker.pinot.svc.cluster.local
After scaling down, some queries are still attempting to route to the removed pods (pinot-broker-1 and pinot-broker-2) intermittently. This behavior results in errors, as these pods are no longer active or reachable.Xiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
SP
04/30/2025, 4:45 PMcurl http://pinot-broker-headless.pinot.svc.cluster.local:8000/query/sql -H 'Authorization: Basic xxxxxx -d '{"sql":"SELECT * FROM xxxxxxx"}'Xiang Fu
Xiang Fu
Xiang Fu
SP
04/30/2025, 4:47 PMXiang Fu
SP
04/30/2025, 4:49 PMXiang Fu
Xiang Fu
broker-url config is not honored?Xiang Fu
SP
04/30/2025, 4:50 PMXiang Fu
Xiang Fu
SP
04/30/2025, 4:51 PM