This message was deleted.

Hi Younes, There is partition "boosting" which introduces a high cardinality pseudo-column to the range key set to even out segment distribution. This PR: https://github.com/apache/druid/pull/15474#discussion_r1448388430 indicates it is being added to Group By queries, which suggests it should already be in a (more recent?) release of the product. I don't have information on the base "clustering" behavior, other than I am aware that MSQ should be adding something to the range key to increase cardinatlity in an effort to create more even resulting segment sizes.

I'm using the latest version 28.0.1. I'll check to see if this feature is included...

It is expected that a partition is processed by a single worker (otherwise we’d be “partitioning” it further). What is not expected is that the single partition is too large, that it is a pain point for the user. Since it’s an issue with the segment generation, it is highly likely that what John mentioned is the issue, since we haven’t seen inaccuracy in other regions of generating partitions. To see if it’ll help to upgrade to 29.0.0 (which is going to be released soon) - check if the query is a Group By query, and also has a Clustered by

Thank you, I'll try that

Umm, that’s really bad. That’ll mean that the segment will have 218M rows

I added a high cardinality column to the clustering. And that worked nicely!!!!

@Younes Naguib You should try the previous query which generated large segments with druid 29.0.0 and see if it changes something. It would be a good validation for the druid community. Also druid 29.0.0 will be out this week.