This message was deleted Apache Druid #troubleshooting

Join Slack

This message was deleted.

# troubleshooting

Slackbot

03/13/2023, 10:26 AM

This message was deleted.

John Kowtko

03/13/2023, 10:52 AM

Hi Khajjiar, is there a reason not to use SQL then?

khajjiar trip

03/13/2023, 11:00 AM

i think the sql will be converted to a groupBy query in which we will use HLLSketch with lgk=12. which will fail on our cluster. I am converting a groupBy query to a topN query.

John Kowtko

03/13/2023, 8:12 PM

Okay, fishing around for hacks to get the native query to not be a GroupBy ...

Copy code

with tmp as (
 select floor(__time to HOUR) Hr, 
        count(*) cnt 
   from "wikipedia_hr"
  group by 1
)
 select t.* 
   from tmp t
  cross join (select 1 val)
  where t.cnt > 1200

Adding a benign join on the final select forces a query type of "scan" ... I don't know if you can use your sketches with a scan query, but if not then I suggest following this train of logic and trying post-agg operations that will not allow the "having" filter to push back into your group by. Let us know what you find out. Thx.

Gian Merlino

03/13/2023, 11:09 PM

topN doesn't support HAVING on aggregations b/c it isn't really compatible with the topN optimization of pushing down limits to the segment: HAVING logically applies before LIMIT, so if we push down LIMIT, we must also push down HAVING, but if we push down HAVING, then it cannot see the full aggregation results! (because they are not available at the individual segment level.)

Gian Merlino

03/13/2023, 11:10 PM

there is sort of a logical incompatibility there

Gian Merlino

03/13/2023, 11:12 PM

btw, i know you're trying topN because you had issues with groupBy performing. one thing i wanted to ask is, could you try your groupBy query with

mergeThreadLocal: true

in your context, using Druid 25? It's an experimental parameter that changes groupBy's merge on historicals to be thread-local instead of using a shared buffer, which reduces thread contention. We found it useful in certain cases and are wondering if we should make this on by default. Knowing if it helps you would help us make that decision 🙂

khajjiar trip

03/15/2023, 4:44 AM

Thanks Gian. i would have loved to try that param with Druid 25. The issue is that i don’t have much control over the builds/version on our dev cluster. I would see whats possible over the weekend.

Gian Merlino

03/15/2023, 11:27 PM

ah, okay. It's available on 24.0 too, if you use that

Gian Merlino

03/15/2023, 11:27 PM

Not 0.23, though

Open in Slack

Previous Next