Hi team. Wanted to understand how compactions and ...
# general
u
Hi team. Wanted to understand how compactions and ingestions in compacted datasources are expected to be working. 1. I have a datasource with data from 2024. Granularity set to Day. I set up its auto compaction with granularity set to Month. Offset day set to 10. Even so on Aug 19 : The segments seem to be month granularity up till July and then Day granularity till Aug 19. Is this expected behaviour? Are these Day level segments going to be there till Aug end? 2. Now I run an ingestion for a time period July 25 - Aug 5. Firstly this ingestion breaks as OVERWRITE WHERE clause identified interval which is not aligned with
PARTITIONED BY granularity.
This I figure is because the date range is split between month level segments and day level segments. So I break the ingestion into 2 : Before the month level change and after the month level change. So I run an ingestion July 25 - July 31. This works but only with DAY granularity. So this makes me uncertain about whether or not the ingestion earlier was breaking because of the underlying segment granularity. 3. Now the ingestion for July 25 - July 31. creates 7 day level segments : But they are not getting compacted : Compaction is saying 100% compacted except for last 10 days. Not seeing these uncompacted segments. Shouldn't these segements be relevant for compaction? If anybody who understands compaction well, can help with this. Would be appreciated.