This message was deleted.

The recommendation is to use it. Auto compaction on all data sources using real-time ingestion with multiple tasks is a good/best practice. Each parallel task generates segments independently so using auto-compaction to merge them is an important technique in segment size optimization.

And here's a consideration from the docs:

To set

skipOffsetFromLatest

, consider how frequently you expect the stream to receive late arriving data. If your stream only occasionally receives late arriving data, the auto-compaction system robustly compacts your data even though data is ingested outside the

skipOffsetFromLatest

window. For most realtime streaming ingestion use cases, it is reasonable to set

skipOffsetFromLatest

to a few hours or a day.