This message was deleted.

Generally auto-compaction cannot run against a time chunk that is currently being actively ingested into. The conflict is against the entire time chunk, not just individual segments. I am dealing with this issue on anothe cluster right now, I have been told to be careful about mixing segment granularities with real-time ingestion. So either: a) set up your auto-compaction with an offset that is old enough to never operate on a time chunk that is being ingested into (as limited by lateMessageRejectionPeriod). E.g. set the compaction offset to P31D and set lateMessageRejectionPeriod to 30D ... or set compaction offset to 2D and lateMessageRejectionPeriod to 1D or b) Leave your auto-compaction at HOUR granularity, with offset 2H or so, which should allow it to run when it finds a time when that hour is not being ingested into. I may not have gotten this 100% right as I am still learning about this, hopefully others will confirm or correct. -- John

Option b is what we had before the auto-compaction could change the granularity and indeed the compaction works. But then it is meaningless since what we want is to rollup the data in less segments as it makes queries more than a hundred times faster by our benchmarks with our real data. Now for option a... I haven't set anything on

lateMessageRejectionPeriod

since we deal with late messages by ourselves way before it reaches Druid. I had no idea that it could interfere with auto compaction so I'll try that. Thanks

Also, there is one more thing to keep in mind ... Auto-compaction offset is from the LATEST TIME CHUNK in the datasource, not the CURRENT DATETIME ... so if you are ingesting any data with future event dates, this will affect which time chunk compaction tries to start with. Check the START/END dates of your segments to see if you have any with future dates in them. If you do, then ... ... the earlyMessageRejectionPeriod parameter allows you to limit the early data in this case.

That is the second problem. The start and end date for the Kafka ingested segments are the stars and end of the month. Even though they only have data regarding a single hour

Your auto-compaction offset is set to 0S, so no delay, it will compact the current time chunk from HOUR to MONTH. I would try changing that to P1M so that it clears the current month and will only start compacting the data from HOUR to MONTH after you move into the next month.

But if I set the offset to one month then I'll have an entire month worth of uncompacted segments, which is exactly what I am trying to prevent

If you are concerned about fragmentation within the hour timechunks, then I think you can run manual compaction with HOUR granularity and specify the interval for the current month. Currently I don't think you can create multiple levels of auto-compaction (there is an upcoming "life-cycle management" feature that is supposed to address that) ... so the manual compaction of HOUR segments for the current month you may have to set up on a cron job.