We have a pinot cluster, some of our users are run...
# troubleshootingy
Yash Agarwal
11/15/2021, 6:49 AMWe have a pinot cluster, some of our users are running very heavy queries which results in
Copy code
java.lang.OutOfMemoryError: Java heap space Copy code
{
"_code": 404,
"_error": "ZKPath /PinotCluster/LIVEINSTANCES/Server_node_8098 does not exist:"
}k
Kishore G
11/15/2021, 7:05 AM
you can set a limit for maxQueryLimit and maxGroupBy limits
y
Yash Agarwal
11/15/2021, 7:21 AMSure. but even then there are cases when the limits are quite small but it is doing a count distinct on a large column.
Yash Agarwal
11/15/2021, 7:23 AMideally we are preventing all these queries .. in our middle layer .. and converting them to optimized versions .. but just in case we want to avoid our nodes from going down.
k
Kishore G
11/15/2021, 7:23 AM
how large is it?
Kishore G
11/15/2021, 7:23 AM
did you try partitionedDistinct?
y
Yash Agarwal
11/15/2021, 7:25 AMYes we have that implemented .. but these are very fringe cases hence trying to understand how to avoid from getting the node down.
Yash Agarwal
11/15/2021, 1:10 PMOur memory settings are Xms4G Xmx8G on 16 G nodes. Should we bump down our Xmx even further ?
k
Kishore G
11/15/2021, 3:14 PM
Two options
• increase the memory to ensure that distinct values fit in memory
add configuration to limit max distinct values or Enhance distinct operator to start using HLL when the number of unique goes beyond a certain size.. this will require code change
y
Yash Agarwal
11/15/2021, 3:51 PMI am more worried about making sure the node is able to fix itself after such a query.
Yash Agarwal
11/15/2021, 3:51 PMCurrently the only option for us is to restart the server instance
k
Kishore G
11/15/2021, 3:52 PM
Can you please file an issue
y
Yash Agarwal
11/15/2021, 4:00 PMI think there is a similar issue already created. https://github.com/apache/pinot/issues/5627. Hence not creating a duplicate.
2 Views