This message was deleted.
# generals
Slackbot
11/22/2023, 6:07 PMThis message was deleted.
s
Sergio Ferragut
11/22/2023, 7:31 PMHave you enabled metrics? You can see how many segments each of them scanned in the query metrics. Specifically in broker's
query/segments/countquery/segmentAndCache/timeSergio Ferragut
11/22/2023, 7:36 PMAlso, how do you expect to query this? Hash is typically better to minimize segments needed when using an equality condition at query time on the partitioning column. It is more susceptible to skew in the data and can therefore create more imbalance in the segment sizes. With range partitioning, particularly in MSQ, segment size balancing is better, but you do have the possibility of needing to scan more segments in the equality condition case because a value can span more than one partition.
I'm very curious to hear more about your results.
a
Abhishek Agarwal
11/23/2023, 6:54 AMMetrics is a good way to do that comparison. I don't think that scanning more segments is necessarily a problem. Even if the request lands on a data server, data server has on-disk indices to skip most of the rows.
a
Anshu Makkar
11/23/2023, 7:09 AMScanning more segments has become an issue, that's why I am considering range dim partitioning. Right now cpu reaches 100% in historicals, querying last 7days of data replication =2.
There are 900 segments in a day near 500-700 MB.
Anshu Makkar
11/23/2023, 7:21 AMCurrently with hashed based partitioning there is no pruning happening, which is why cpu is reaching upto 100%.
s
Sergio Ferragut
11/24/2023, 11:05 PMAny common filtering columns among the queries? In the hash case did you filter on the partitioning column? I know the obvious answer is yes. But just verifying 😁 it seems very odd to me that it wouldn’t prune. Is the filter an equality?
a
Anshu Makkar
11/27/2023, 3:03 PMYes we had equality condition and filtering on partitioning column.
Anshu Makkar
11/27/2023, 3:04 PM@Sergio Ferragut
query/segments/count