https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
g
It's pretty workload-dependent so the information about your specific workload will be helpful! A tip about data-gathering: in cases where the Historicals aren't dying, but are just getting a lot of allocation pressure, heap dumps (as big chunky point-in-time snapshots) don't always tell the story very well. A couple of other options: 1) Better: heap histograms, from 
jmap -histo <pid>
, taken repeatedly. These are much faster to take than heap dumps and show the same high-level info. Not the full object graph, but you get the same high-level object counts. 2) Best: a JVM allocation profiler. These hook into the JVM and track allocations continuously. Definitely the best way to figure out what's pressuring the allocator.
Another note: not sure what version you're using, but note that in recent versions (starting from 24) we improved the allocation profile of groupBys: https://github.com/apache/druid/pull/12474, https://github.com/apache/druid/pull/12468 This stuff will be helpful if you're doing a lot of groupBys and are on a version < 24
l
Thanks Gian, that all helped! Histograms definitely were of use. and I'm working to figure out more about connecting a profiler to get some live allocation rate information After some analysis, I'm starting to be suspicious of queries who have tons of IN filters, some of which are surprisingly large. I'm seeing ThreadLocalMap objects that are pretty huge on query threads named as 
QUERYTYPE_DATASOURCE_INTERVAL
Within this map are SpillingGroupers who have filtered aggregators and the InDimFilter seems crazy large. Measured in MBs. I'm a bit concerned that these are coming onto heap as Humongous Objects, making heap become fragmented. I see that you added a performance improvement in druid 24 that provides the filter values as UTF-8 out of the box. In my head this lines up with the timeline of this problem becoming more common for us after upgrading to Druid 24. My question for you now is - what part of the query are these threads for? I'm thinking if we have many segments involved in a query located on a single node, if that node takes a query with these crazy IN filters, there could be a lot of these DimInFilter objects being created at the same time, causing my problems. Does that hypothesis have any merit in your opinion? Unfortunately I'm only on the platform team, and our query services team would be the ones to put the clamps down on what kind of filters our users can come up with 😅 If I had the power I'd probably put some ground rules in place when it comes to the number of values in these filters to see if it helps at all.
another observation is that I feel like I'm seeing spilling groupers referenced by these processing threads (
QUERYTYPE_DATASOURCE_INTERVAL
) who are for what I assume are previous queries. For example, I see a reference to a grouper for a group by on datasource Y in a processing thread that is actually a timeseries for datasource x. I'm trying to wrap my head around the historical query execution code and how the processing thread pool is releasing resources, but these references stood out to me
g
the utf8 change does increase the memory requirement of IN filters, however, the utf8 version of the value set is shared across all segments for the query (so the # of segments on your server doesn't affect memory usage)
i expect the utf8 change would increase the memory requirement by about 50% (for IN filter)
if you wanted to test it, you could deploy a build to one server that omits the 
valuesUtf8
stuff. i.e. delete the variable and also delete the code that uses it:
Copy code
final Utf8ValueSetIndex utf8ValueSetIndex = <http://indexSupplier.as|indexSupplier.as>(Utf8ValueSetIndex.class);
      if (utf8ValueSetIndex != null) {
        return utf8ValueSetIndex.forSortedValuesUtf8(valuesUtf8);
      }
With that code deleted, we'll fall back to using the utf16 
values
l
Interesting idea! Although now that you point out that the utf8 version is shared, I'm now thinking this was a false flag due to my assumption that there was no re-use there. My search continues. I appreciate your inputs!
I think the problem is identified... there was a client out there running 
none
granularity 
timeseries
queries. I was so dialed in on large group by's that I never even noticed these popping up in request logs whenever we had instability. I got tipped off when looking at a heap histogram taken while a node was in the process of dying and there were millions of 
DateTime
objects which got me wondering who the heck was doing insanely granular workloads. bet part is that it isn't even some type of sensor data or financial data where milliseconds even make a difference
g
Oh, yeah, those are gross. 
none
basically means 
millisecond
.
Note: if you do the queries through SQL, we plan these queries more safely since https://github.com/apache/druid/pull/13206
Although if you do them through native you can still foot gun
l
that's a great feature in #13206! We are still on native as our application team has built a whole services layer around that query language which is how our users interact with the cluster as of now. But if SQL is getting most of the attention going forward, it could be time for us to start scoping out what it would look like to migrate over. I appreciate the help you've provided in this search. The tip on histograms certainly made the difference!
g
glad that helped! Certainly our way of thinking going forward (and also, just in general) is that SQL is the path for "I want the database to try to run the query the best way it can" and native is the path for "I know what I am doing"
In general we expect most new users would be using SQL, and existing users would migrate unless they really like the control they get with native
2 Views
Open in Slack
PreviousNext
Powered by