Hi , for deep storage query if the segments are pa...
# generalk
Krishna
11/26/2024, 8:40 AMHi , for deep storage query if the segments are partitioned based on particular column and if you use that column in deep storage query filter then it should just pull only those segments right ? but we are seeing it is pulling all the segments present for that hour .
j
John Kowtko
11/26/2024, 2:24 PMHi Krishna,
I am seeing "range pruning" when I use MSQ.
As an example you can test, create this range partitioned version of wikipedia:
Copy code
replace into wiki_range overwrite all select * from wikipedia partitioned by day clustered by channel, flags Copy code
{
"rowsPerSegment": 2000
}John Kowtko
11/26/2024, 2:26 PMThen run a query with filter:
Copy code
select count(*) from wiki_range where channel = '#fr.wikipedia'John Kowtko
11/26/2024, 2:38 PMMake sure you are creating the sharded segments properly, i.e. if using MSQ then you must use REPLACE and not INSERT for your create statement.
If you have done all of the above and it is still not working for you, then please send your create statement, a screenshot of the resulting segment shards, and the query you are running against them so we can look at the filters you are using.
And let us know what version of Druid you are using.
k
Krishna
11/26/2024, 4:43 PMThanks @John Kowtko we r using native ingestion and using msq for deep storage querying
j
John Kowtko
11/26/2024, 6:40 PMThen as long as you have the shards set up something like the above, and you MSQ filter expressions are index-friendly, then you should get some pruning. if you want to post your WHERE conditions on the MSQ SQL we can take a look.
k
Krishna
11/26/2024, 10:05 PMsure