Hi folks! I'm having an issue which is, my queries...
# troubleshooting
d
Hi folks! I'm having an issue which is, my queries are timing out because they run past the default timeout set for queries to Pinot; I know that it's possible to use for example
option(timeoutMs=60000)
after a table name to increase the timeouts, but the problem is, I'm using the SQLAlchemy library in my Python project and haven't yet found a way to make it compile that into the query. Is there some other way to increase the timeouts on a per-query basis? Something like a
SET TIMEOUT=60000
query I can execute before my normal query?
m
I’d first try to improve the query latency before increasing timeout
Is there a set of expensive queries that always timeout even when no load on cluster? If so can you share query and response metadata
d
Ideally that's what I want to do, but I need to increase the timeout as a stop-gap for now, otherwise I'll have to spend days or weeks until I can get my latency increased, because more serious changes will have to be made probably
What I need to do is to get a truckload of data up to 100M rows and then aggregate that inside my application (because the aggregation logic is more complex than what Pinot is able to do on its own), and because of the amount of data Pinot is timing out
m
I think there are broker level time out configs you can set. However I am worried increasing timeout may adversely impact the cluster because less number of queries will run in parallel (due to expensive queries), and may cause more backlog
What’s your read qps
d
My read QPS is really low, hardly ever there will be multiple people doing queries actually, so increasing latency a bit is not a big issue for us. Even if it takes like 1 minute to run.
But I'll try to increase the broker timeout config then, I think it might be enough for now. Later on I want to run some database recommendations so that I can find out how I can optimize it further, but this is for the future, I can't work on that now.
👍 1
(At the moment I'm trying to wrap up my stuff so that I can finally have some analytics UI available for our internal users)