Hi team. I have Pinot deployed on AWS EKS with of...
# troubleshooting
t
Hi team. I have Pinot deployed on AWS EKS with official helm chart. I am facing authentication problems between Pinot component when the Server tries to complete segments and cannot call the Controller.
2025/04/30 06:17:46.386 ERROR [ServerSegmentCompletionProtocolHandler] [player_sessions_active_minutes__9__0__20250430T0525Z] Could not send request *http://pinot-controller-1.pinot-controller-headless.pinot.svc.cluster.local:9000/segment* │
│ Consumed?reason=rowLimit&streamPartitionMsgOffset=4446773&instance=Server_10.65.77.10_8098&name=player_sessions_active_minutes__9__0__20250430T0525Z&rowCount=100000&memoryUsedBytes=5248627                                                  │
│ org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 401 (Unauthorized) with reason: "HTTP 401 Unauthorized" while sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=4446773&instance= │
│ Server_10.64.78.10_8098&name=player_sessions_active_minutes__9__0__20250430T0525Z&rowCount=100000&memoryUsedBytes=5248627 to controller: pinot-controller-1.pinot-controller-headless.pinot.svc.cluster.local, version: Unknown               │
│     at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:476) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
I have enabled pinotAuth in my helm chart values override
pinotAuth:
enabled: true
controllerFactoryClass: org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory
brokerFactoryClass: org.apache.pinot.broker.broker.BasicAuthAccessControlFactory
configs:
- access.control.principals=admin,user,viewer
- access.control.principals.admin.password=${admin_pass}
- access.control.principals.user.password=${user_pass}
- access.control.principals.viewer.password=${viewer_pass}
- access.control.principals.user.permissions=READ,WRITE
- access.control.principals.viewer.permissions=READ
However I cannot understand where in the helm chart I can configure basic auth access control to the controller my reference point is the documentation here -> https://docs.pinot.apache.org/operators/tutorials/authentication/basic-auth-access-control I tried passing extra configs to the helm chart like so
server:
extra:
configs: |-
pinot.server.segment.fetcher.auth.token=Basic ${admin_pass}
pinot.server.segment.uploader.auth.token=Basic ${admin_pass}
pinot.server.instance.auth.token=Basic ${admin_pass}
or in jvmOpts but neither worked. 🆘
x
I think those
Copy code
server:
  extra:
    configs: |-
      pinot.server.segment.fetcher.auth.token=Basic ${admin_pass}
      pinot.server.segment.uploader.auth.token=Basic ${admin_pass}
      pinot.server.instance.auth.token=Basic ${admin_pass}
should work can you validate those in your k8s configMap ?
t
Yes, values are propagated to the configmap. Managed to get it to work, between all experiments I did, no idea exactly what was going wrong but it works now, thank you
x
ok, this config
pinot.server.segment.uploader.auth.token=Basic ${admin_pass}
will drive the token used to upload a segment to controller
t
Hi @Xiang Fu Bit of a side question, but is it possible to use single stage query engine when querying Pinot from Superset? I have a chart that throws a timeout error
│ 2025/05/01 14:24:55.493 ERROR [MultiStageBrokerRequestHandler] [jersey-server-managed-async-executor-0] Caught exception executing request 1679471305000000197: SELECT active_minute_ts AS active_minute_ts, status AS status, COUNT(DISTINCT │
│ FROM "default".player_sessions_active_minutes                                                                                                 │
│ WHERE active_minute_ts >= '2025-05-01 11:24:45.000000' AND active_minute_ts < '2025-05-01 14:24:45.000000' GROUP BY active_minute_ts, status ORDER BY COUNT(DISTINCT player_uid) DESC                             │
│  LIMIT 50000, Received error query execution result block: {250=ExecutionTimeoutError                                                                             │
│ ProcessingException(errorCode:250, message:ExecutionTimeoutError)                                                                                       │
│   at org.apache.pinot.common.exception.QueryException.<clinit>(QueryException.java:116)                                                                           │
│   at org.apache.pinot.common.response.broker.BrokerResponseNative.<clinit>(BrokerResponseNative.java:61)                                                                  │
│   at org.apache.pinot.core.query.reduce.BrokerReduceService.reduceOnDataTable(BrokerReduceService.java:73)                                                                 │
│   at org.apache.pinot.broker.requesthandler.SingleConnectionBrokerRequestHandler.processBrokerRequest(SingleConnectionBrokerRequestHandler.java:149)}                                            │
│ org.apache.pinot.query.service.dispatch.QueryDispatcher.runReducer(QueryDispatcher.java:462)                                                                         │
│ org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:133)                                                                       │
│ org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:255)                                                         │
│ org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:168)
I can see Superset envokes the multistage engine here
MultiStageBrokerRequestHandler
When i run the same query in Pinot Query Console, it completes under a second. I tried passing
{"connect_args":{"use_multistage_engine":"false"}}
to the database connection Engine Parametes in Superset, but the requests keep going through the multistage broker
x
Copy code
I think in this case you need to either configure longer timeout or use distinctCountHll kind of estimation for the unique counting
the stacktrace shows it’s using single stage for your query
t
I do have distinctcounthll in starTreeIndexConfigs in the table definition "functionColumnPairs": [ "DISTINCTCOUNTHLL__player_uid" but the query still times out in superset. I don't understand why it doesn't time out in the query console in Pinot, it only times out there if I toggle multiquery engine on.