Is it possible to query data from a specific parti...
# general
a
Is it possible to query data from a specific partition? I have a datasource partitioned using a hash-based strategy, and I want to query data from partition #10 corresponding to a particular segment. Can this be accomplished using SQL, native queries, or MSQ in Druid? While I know I can inspect segment records via the Druid console or by using the dump-segment tool, I need a way to query this data directly.
j
Hi Alex, Segment pruning happens automatically when executing a query, whether in SQL or JSON format. The pruning is done based on the filter conditions of the query. Generally the segment sharding process and query pruning process are handled automatically by Druid, so presumably no reason to have to consciously direct the query to specific segments. Just curious what your use case that would require you to want to query from a specific segment. If it is for "surgical update" purposes, there is a better way to do this using mixed segment granularity overwrites. Thanks. John
a
Hi. My use case is to select data for performance tests. I need to evaluate worst scenario - when requested data belongs to the all different shards, medium - 50% of segments need to be scanned, 25% and so on
j
I would suggest using range partitioning then, you can see the shard boundary values to determine how any segments a given query would access based on the filter conditions that you specify.
a
Thank you, but I'm already use hash based partition key with a lot of loaded segments and changing the partition type is not an option for me. I just want to find a "good" data for the performance tests. That's why I need to query this data with more complex filter and belonging to the same segments is only one of criteria that I can't calculate during the query time
p
hi Alex, I'm not familiar with a method to query a partition, but if you hack a scenario where only specific partition is queried - you can manually create a new datasource with only the required segment (you"ll need to mingle segment metadata a bit, or use another druid and direct it to same segment(s), which is maybe a better approach anyway for performance test)
a
Hi Pavel, Thank you for your suggestion regarding the "hack." I plan to use it to select data for the "best" scenario, where all queried data belongs to a single partition. However, I still have a problem with the worst scenario, where all data resides in different partitions.