Tsvetan
04/30/2025, 12:39 PM2025/04/30 06:17:46.386 ERROR [ServerSegmentCompletionProtocolHandler] [player_sessions_active_minutes__9__0__20250430T0525Z] Could not send request *http://pinot-controller-1.pinot-controller-headless.pinot.svc.cluster.local:9000/segment* │
│ Consumed?reason=rowLimit&streamPartitionMsgOffset=4446773&instance=Server_10.65.77.10_8098&name=player_sessions_active_minutes__9__0__20250430T0525Z&rowCount=100000&memoryUsedBytes=5248627 │
│ org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 401 (Unauthorized) with reason: "HTTP 401 Unauthorized" while sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=4446773&instance= │
│ Server_10.64.78.10_8098&name=player_sessions_active_minutes__9__0__20250430T0525Z&rowCount=100000&memoryUsedBytes=5248627 to controller: pinot-controller-1.pinot-controller-headless.pinot.svc.cluster.local, version: Unknown │
│ at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:476) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
I have enabled pinotAuth in my helm chart values override
pinotAuth:
enabled: true
controllerFactoryClass: org.apache.pinot.controller.api.access.BasicAuthAccessControlFactory
brokerFactoryClass: org.apache.pinot.broker.broker.BasicAuthAccessControlFactory
configs:
- access.control.principals=admin,user,viewer
- access.control.principals.admin.password=${admin_pass}
- access.control.principals.user.password=${user_pass}
- access.control.principals.viewer.password=${viewer_pass}
- access.control.principals.user.permissions=READ,WRITE
- access.control.principals.viewer.permissions=READ
However I cannot understand where in the helm chart I can configure basic auth access control to the controller
my reference point is the documentation here -> https://docs.pinot.apache.org/operators/tutorials/authentication/basic-auth-access-control
I tried passing extra configs to the helm chart like so
server:
extra:
configs: |-
pinot.server.segment.fetcher.auth.token=Basic ${admin_pass}
pinot.server.segment.uploader.auth.token=Basic ${admin_pass}
pinot.server.instance.auth.token=Basic ${admin_pass}
or in jvmOpts but neither worked.
🆘Xiang Fu
server:
extra:
configs: |-
pinot.server.segment.fetcher.auth.token=Basic ${admin_pass}
pinot.server.segment.uploader.auth.token=Basic ${admin_pass}
pinot.server.instance.auth.token=Basic ${admin_pass}
should work
can you validate those in your k8s configMap ?Tsvetan
04/30/2025, 5:01 PMXiang Fu
pinot.server.segment.uploader.auth.token=Basic ${admin_pass} will drive the token used to upload a segment to controllerTsvetan
05/01/2025, 2:30 PM│ 2025/05/01 14:24:55.493 ERROR [MultiStageBrokerRequestHandler] [jersey-server-managed-async-executor-0] Caught exception executing request 1679471305000000197: SELECT active_minute_ts AS active_minute_ts, status AS status, COUNT(DISTINCT │
│ FROM "default".player_sessions_active_minutes │
│ WHERE active_minute_ts >= '2025-05-01 11:24:45.000000' AND active_minute_ts < '2025-05-01 14:24:45.000000' GROUP BY active_minute_ts, status ORDER BY COUNT(DISTINCT player_uid) DESC │
│ LIMIT 50000, Received error query execution result block: {250=ExecutionTimeoutError │
│ ProcessingException(errorCode:250, message:ExecutionTimeoutError) │
│ at org.apache.pinot.common.exception.QueryException.<clinit>(QueryException.java:116) │
│ at org.apache.pinot.common.response.broker.BrokerResponseNative.<clinit>(BrokerResponseNative.java:61) │
│ at org.apache.pinot.core.query.reduce.BrokerReduceService.reduceOnDataTable(BrokerReduceService.java:73) │
│ at org.apache.pinot.broker.requesthandler.SingleConnectionBrokerRequestHandler.processBrokerRequest(SingleConnectionBrokerRequestHandler.java:149)} │
│ org.apache.pinot.query.service.dispatch.QueryDispatcher.runReducer(QueryDispatcher.java:462) │
│ org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:133) │
│ org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:255) │
│ org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:168)
I can see Superset envokes the multistage engine here
MultiStageBrokerRequestHandler
When i run the same query in Pinot Query Console, it completes under a second.
I tried passing {"connect_args":{"use_multistage_engine":"false"}}
to the database connection Engine Parametes in Superset, but the requests keep going through the multistage brokerXiang Fu
I think in this case you need to either configure longer timeout or use distinctCountHll kind of estimation for the unique countingXiang Fu
Tsvetan
05/01/2025, 3:23 PM