https://pinot.apache.org/ logo
Join Slack
Powered by
I’m running into this issue where the servers are ...
# troubleshooting
l
I’m running into this issue where the servers are starting to ingest many and many records ( a spike on ingestion) but when i see the throughput from a kafkastreams perspective i don’t see any spikes, and then weirdly enough i see the servers restarting for some reason, while checking the logs I do see this:
Copy code
Caught exception while processing query: QueryContext{_tableName='ads_metrics_REALTIME', _selectExpressions=[listing_id, sum(click_count), sum(impression_count), sum(cost), sum(order_count), sum(revenue)], _aliasList=[null, null, null, null, null, null], _filter=(shop_id = '25746445' AND serve_time BETWEEN '1637125200' AND '1639717199'), _groupByExpressions=[listing_id], _havingFilter=null, _orderByExpressions=null, _limit=6000, _offset=0, _queryOptions={responseFormat=sql, groupByMode=sql, timeoutMs=9999}, _debugOptions=null, _brokerRequest=BrokerRequest(querySource:QuerySource(tableName:ads_metrics_REALTIME), pinotQuery:PinotQuery(dataSource:DataSource(tableName:ads_metrics_REALTIME), selectList:[Expression(type:IDENTIFIER, identifier:Identifier(name:listing_id)), Expression(type:FUNCTION, functionCall:Function(operator:SUM, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:click_count))])), Expression(type:FUNCTION, functionCall:Function(operator:SUM, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:impression_count))])), Expression(type:FUNCTION, functionCall:Function(operator:SUM, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:cost))])), Expression(type:FUNCTION, functionCall:Function(operator:SUM, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:order_count))])), Expression(type:FUNCTION, functionCall:Function(operator:SUM, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:revenue))]))], filterExpression:Expression(type:FUNCTION, functionCall:Function(operator:AND, operands:[Expression(type:FUNCTION, functionCall:Function(operator:EQUALS, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:shop_id)), Expression(type:LITERAL, literal:<Literal longValue:25746445>)])), Expression(type:FUNCTION, functionCall:Function(operator:BETWEEN, operands:[Expression(type:IDENTIFIER, identifier:Identifier(name:serve_time)), Expression(type:LITERAL, literal:<Literal longValue:1637125200>), Expression(type:LITERAL, literal:<Literal longValue:1639717199>)]))])), groupByList:[Expression(type:IDENTIFIER, identifier:Identifier(name:listing_id))], orderByList:[], limit:6000, queryOptions:{responseFormat=sql, groupByMode=sql, timeoutMs=9999}))}
java.lang.ArrayIndexOutOfBoundsException: null
this started happening out of nowhere so i’m unsure as to what’s happening has anyone gotten this similar kind of error? I also don’t have the stack trace sadly i don’t know why it isn’t being logged 😞 also in general our p99 response times have been impacted
e
Do you see any direct memory oom's in the logs? That can cause the server to restart.
l
not really but i did see the servers getting super busy
e
The log message would be something like
Copy code
java.lang.OutOfMemoryError: Direct buffer memory
l
let me look for that message
doing some more digging it seems like all the apps got restarted (most likely a change in our end made all the apps in the gke cluster to restart) i wonder if the fact that they didn’t restarted in the right order made pinot come back in a funky state
seems like that’s what caused everything to restart 😄
👍 1
e
You can set the direct memory explicitly in the jvm opts
by default it's the same as Xmx
l
hmm ok I think my only lingering question is what’s this error i’m getting when executing that query lol I guess all the other questions have been answered
e
I've seen strange errors when a server crashes due to oom, if it doesn't occur after increasing direct memory and/or scaling up the # of servers that could indicate the source of the error is due to the server crash.
👀 1
x
how long does the ingestion spike last?
do you see messages been double consumed
l
i think the spikes have been solved for now
what worries me more is that when i execute a query i get that error ^
Copy code
select product_id, SUM(click_count), SUM(impression_count), SUM(cost), SUM(order_count), SUM(revenue) from ads_metrics 
WHERE user_id = 8471315
AND serve_time BETWEEN 1634515200 AND 1634601599
GROUP BY product_id
i’m looking into a month’s data
in the query console i get this error
Copy code
[
  {
    "message": "QueryExecutionError:\njava.lang.ArrayIndexOutOfBoundsException",
    "errorCode": 200
  }
]
and what the logs show is what i pasted above in the main thread
i have found that the more data i look the error is present
if i play with the dates I can get it to work for last 2 days
if i look at past 3 days i get the error
weird
x
Hmm, what if you just query on -3rd day
l
it returns data
x
hmm, so just 3rd day work but past 3 days doesn't work
Interesting
l
I don't have a stack trace :(
but yea I get these errors
the only clue is the time span
that I have for now
Open in Slack
PreviousNext
Powered by