This message was deleted.
# general
s
This message was deleted.
v
Try to aggregate before join. Another option is to use lookup instead of join. Depending on the query you can first filter by the join condition. Paste the query here and I will try to be more detailed
a
what are the sizes of join sides? How many rows on the left and how many does right side has?
🙌 1
u
left size is maximum 30million maximum(but large colum numbers) and right size can be much larger than right side right table is keep being larger (using kafka ingestion by druid) i think can we try ingestion tuning for this case any advice is welcome and thank you
a
if the right size is a large druid table, then try to flip the join. so larger table is on the left
🙌 1
u
i want to try but this case we have to keep format like this
Copy code
# DELETED
a
here table2 is the larger one?
🙌 1
u
yes table 2 is streaming data
r
I did know read the query , but if you are returning just counts, maybe consider using DataSketch for filter/union/intersection, will be much faster and scale for billion of 'rows'
u
we tried but not worked well maybe in druid 0.25 version filter has been huge upgraded i would check it out and want to use druid for more …
r
what version are you running? the improvements in 0.25 was mostly focused on the MSQ, not the classic query runner
👍 1
or maybe https://github.com/apache/druid/pull/13133 but even that is only tested for (already very large, 1M in values)
👍 1
u
ver 21 -> 25 now i would check it out
Maybe I need tuning for data partition ...
upgrading to 25 and using semi join
in
performance same… maybe cuz of partition?