Hi Folks, Has anyone faced random issues while querying with tracing? Ran a simple query with tracing from pinot ui

select sum(amount) as amt from reward
where user_id = 'some_id'

Following exception occurred

[
  {
    "message": "InternalError:\njava.lang.NullPointerException\n\tat org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:188)\n\tat org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:223)\n\tat org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:151)",
    "errorCode": 450
  },
  {
    "message": "InternalError:\njava.lang.NullPointerException\n\tat org.apache.pinot.core.util.trace.TraceContext.getTraceInfo(TraceContext.java:188)\n\tat org.apache.pinot.core.query.executor.ServerQueryExecutorV1Impl.processQuery(ServerQueryExecutorV1Impl.java:223)\n\tat org.apache.pinot.core.query.executor.QueryExecutor.processQuery(QueryExecutor.java:60)\n\tat org.apache.pinot.core.query.scheduler.QueryScheduler.processQueryAndSerialize(QueryScheduler.java:151)",
    "errorCode": 450
  },
  {
    "message": "1 servers [11.4.7.172_O] not responded",
    "errorCode": 427
  }
]

Whats more interesting the servers become unstable and randomly timeout

"errorCode": 427

, queries (without tracing) are randomly processed successfully or timeout Timeout log in server

ERROR [BaseCombineOperator] [pqr-1] Timed out while polling results block, numBlocksMerged: 0 (query: QueryContext{_tableName='reward_OFFLINE', _selectExpressions=[*], _aliasList=[null], _filter=null, _groupByExpressions=null, _havingFilter=null, _orderByExpressions=null, _limit=10, _offset=0, _queryOptions={responseFormat=sql, groupByMode=sql, timeoutMs=10000}, _debugOptions=null, _brokerRequest=BrokerRequest(querySource:QuerySource(tableName:reward_OFFLINE), pinotQuery:PinotQuery(dataSource:DataSource(tableName:reward_OFFLINE), selectList:[Expression(type:IDENTIFIER, identifier:Identifier(name:*))], orderByList:[], limit:10, queryOptions:{responseFormat=sql, groupByMode=sql, timeoutMs=10000}))})

Restarting all the servers fixed the issue Using recent nightly docker image (digest a6c14285abf4)

Do you have indexing setup? Also seems like you could sort on user id

yes data is sorted by user id. The query usually have < 20ms resp time. The issue only happens if I try tracing and the exception occurs. Then queries randomly time out

Cc: @Richard Startin for tracing

yes https://github.com/apache/pinot/issues/7502