Hi, we have a datasource which was ingested via MSQ with Range partitioning . we have kept maxrows per segment as 5M but we do see segments have more than that. So, we are trying to run Manual compaction to see but Manual compaction is using dynamic partitioning not Range any suggestions how to get segment rows to 5M while ingesting via MSQ or how to make Manual compaction works . Below is the error from task log

2024-05-07T19:22:13,940 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
java.lang.UnsupportedOperationException: null
        at org.apache.druid.msq.util.PassthroughAggregatorFactory.getMaxIntermediateSize(PassthroughAggregatorFactory.java:166) ~[?:?]
        at org.apache.druid.query.aggregation.AggregatorFactory.factorizeWithSize(AggregatorFactory.java:90) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.incremental.OnheapIncrementalIndex.factorizeAggs(OnheapIncrementalIndex.java:312) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.incremental.OnheapIncrementalIndex.addToFacts(OnheapIncrementalIndex.java:241) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.incremental.IncrementalIndex.add(IncrementalIndex.java:517) ~[druid-processing-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.realtime.plumber.Sink.add(Sink.java:184) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.add(AppenderatorImpl.java:308) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.append(BaseAppenderatorDriver.java:411) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.add(BatchAppenderatorDriver.java:115) ~[druid-server-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:108) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:922) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:526) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1199) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:532) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.common.task.CompactionTask.runTask(CompactionTask.java:507) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:169) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) ~[druid-indexing-service-25.0.0.jar:25.0.0]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:834) ~[?:?]
2024-05-07T19:22:13,940 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in BUILD_SEGMENTS.
java.lang.UnsupportedOperationException: null

ah, that error you are seeing was fixed in druid 28

ok .. so we can use Range partitioning in Druid 28 ? and any suggestions why MSQ is not using maxrowspersegment parameter to store exact no.of rows ?

it should respect (roughly) the

rowsPerSegment

context param

ok