https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
v
what clustering (secondary partitioning) are you using for that data source?
i.e. if you look at one of the segments in the console segments view what do you see under 
Shard type
?
a
This One
v
oh boy, well get ready to be lifted up on peoples arms and be treated as a hero. Forget paying for your own beer for I have good news. You can likely get a ton of performance out of your datasource for "free" by switching to range partitioning
you will likely want to range partition on 
type, site
or maybe 
site, type
?
is one of your dimensions frequently being filtered on? Like looking at sensors for a specific type or site?
how you get there depends on what Druid version you are on, and how you are doing ingestion (I am guessing streaming?)
a
The dimension that is always used for filtering in all queries is the tag field (we have about 60000 tags (sensors).
v
then 
tag, type, site
?
What Druid version are you using?
you can find that info on the home page (click the logo in the corner)
a
0.23v and yes streaming ingestion
v
well good news that is the first version that has range partitioning
a
And how can I apply this change Vladim?
v
you can do it as part of the compaction config
but wait... before you jump into it... you asked a very good question! "Could you tell me how I can evaluate if the performance has improved"
a
yes, that is the big question 🙂
v
In Druid 24 an amazing feature called SQL based ingestion was added (it was GA in Druid 25). With SQL based ingestion it would be trivial to copy a section (like a day or a month) of your main datasource into a new datasource to play with different partitioning types. You could try different partitioning types and evaluate which one works best. This is one of the main values of SQL based ingestion: it greatly lowers the cost of experimentation with data arrangements. I would say this is THE main value if you are using streaming ingest.
👍 1
It is so easy that I can even write the query for you (I have not tested this query, it might have typos):
So I would recommend you upgrade to Druid 25!
just to run my cool query
but if you don't want to do it... you can still do effectively the same thing by using the native batch system
And doing 
Reindex from Druid
it is much less fun though
and then you can play the the secondary partitioning in the 
Partition
step
after you figure out what partitioning you want you would add it to your compaction spec
a
ou yeah Vadim! I get to work on it!!
ReIndex Task is running but i don't watch the test table in druid schema, maybe at the end of task?
v
yeah task has to finish and the segments have to load
another cool reason to upgrade and use SQL based ingestion is that it comes with really detailed progress reporting! (look at them progress bars!)
👏 1
That there is the Multi Stage Query system at work 🙂
🙌 1
a
@Vadim hi!! I have installed version 25 of druid but I don't see the sql-msq-task engine in the query console...
The console
v
a
Maybe i need restart middle managers?
2 min in loading....
v
Did you restart all the nodes after adding the extension?
a
no
Amazing!!
Another question, my sentence is
INSERT INTO "tabla_msq" SELECT * FROM "eparksignals-sanroque" where __time >='2023-04-11' and __time < '2023-04-12' PARTITIONED BY HOUR CLUSTERED BY tag
As you can see I have added the "tag" column as a secondary partition (CLUSTERED BY), but if I go to see the segments it seems to have been included right?
@Vadim
v
You need to use REPLACE not INSERT, see first bullet point in https://druid.apache.org/docs/latest/multi-stage-query/concepts.html#clustering
a
Nice! but what is "d1"?
My sentence only goes with CLUSTER BY tag column
is a boost?
v
yeah the d1 is a boost and is used to make sure things are distributed better, future versions of the web console will not show it at all as people find it confusing
a
nice Vadim!!
Open in Slack
PreviousNext
Powered by