@kapa.ai we run Iceberg Compaction but when we run Airbyte Incremental Sync after that, Airbyte does not respect the last compacted state and is still based on old partitions. How can it be fixed?

The knowledge sources do not contain enough information to fully address your specific scenario, but I can share what is documented about the interaction between Iceberg compaction and Airbyte syncs. Key Warning: Compaction and Airbyte Syncs The Airbyte documentation explicitly warns about running compaction alongside Airbyte syncs, particularly for truncate refresh syncs: > Do not run compaction during a truncate refresh sync to prevent data loss. During a truncate refresh sync, the system deletes all files that don't belong to the latest generation. This includes files without generation IDs (compacted files) and files from previous generations. [S3 Data Lake - Compaction] [GCS Data Lake - Compaction] This means that compacted files (which lack generation IDs) can be deleted by Airbyte during a truncate sync, and conversely, compaction can interfere with Airbyte's ability to track its own file generations. What the Knowledge Sources Don't Cover The knowledge sources do not explicitly document: • How Airbyte's incremental sync state interacts with externally compacted Iceberg partitions • A supported fix for the scenario where Airbyte ignores the post-compaction state Recommendations Based on Available Information 1. Avoid running compaction concurrently or between Airbyte syncs if you are using truncate refresh mode, as this is explicitly unsupported. 2. For incremental syncs specifically, the issue of small files and equality deletes after compaction is a known open issue [S3 Data Lake Issue], and the community has noted that external compaction is currently the only option but comes with these caveats. I'd recommend opening a GitHub issue or checking the Airbyte GitHub discussions for more targeted guidance on your specific compaction + incremental sync scenario, as the knowledge sources don't provide a definitive fix for this case. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.