I am using Airbyte cloud for half years, and facing lot of issue due to cdc(change data capture) method. We are struggling to find best method how to effectively adjust data ingestion based on date update at. For example, in paid media domains (google ads, tiktok marketing) some stream doesn't provide cdc or incremental append that cause to spiking our compute use in cloud and our storage as well. its also has breaking change from facebook on feb that our compute hours double its cost cause of retry attempts. It cost our storage (clickhouse) too. I wonder if there are anyone facing the same issue and is there any sustainable solution for this, since we are using airbyte for seamless integration for our saas multi tenant and assurance us for cdc is always available instead reingest data from date to now 😞.

Your frustrations are well-documented and shared by others in the Airbyte community. Let me address the key issues based on what the knowledge sources cover: CDC-Specific Challenges WAL retention and sync frequency trade-off is a real pain point. For CDC sources (like PostgreSQL), Airbyte recommends: • Sync frequency must be shorter than your WAL retention period • 7-day WAL retention is recommended as a buffer against unexpected delays • For near real-time needs: sync every 1–4 hours; for daily reporting: once daily [CDC best practices] Invalid CDC position behavior is another critical setting. For production environments with large databases, Airbyte recommends setting this to "Fail Sync" rather than automatic re-sync, specifically to prevent surprise costs from unexpected full re-snapshots. [CDC best practices] Retry Cost Spikes (e.g., Facebook Breaking Change) This is a known gap. There is an open feature request in the Airbyte community asking for configurable retry limits and cost-control circuit breakers, motivated by exactly the kind of incident you describe — a connector retrying 23 times on a large dataset, multiplying costs unexpectedly. Currently, retry logic is hardcoded per connector with no user-level override. [Feature Request] Paid Media Streams Without CDC/Incremental For sources like Google Ads or TikTok Marketing that don't support CDC or incremental append, this is a known limitation. The knowledge sources do not provide a specific workaround for these connectors beyond what's already documented. The lack of CDC support in these connectors means full refreshes are unavoidable for those streams, which directly contributes to compute and storage spikes. BigQuery/Destination Cost from Incremental Syncs There is also an open GitHub issue about the

MAX(_airbyte_extracted_at)

query pattern used by Airbyte's destination connectors causing high scan costs, which some users have worked around using custom DBT setups or federated queries — though these are community workarounds, not official Airbyte solutions. [GitHub Issue] ------------------------- In summary, the knowledge sources confirm these are real, open issues. The most actionable steps based on official guidance are: 1. Set "Fail Sync" for invalid CDC position behavior in production to avoid runaway re-snapshots. 2. Align your sync frequency with WAL retention to avoid LSN loss. 3. For connectors without CDC, the knowledge sources do not offer a built-in sustainable alternative — this remains an open limitation. 4. The retry cost control issue is currently an open feature request with no released solution yet. I'd recommend upvoting the relevant GitHub issues linked above to signal priority to the Airbyte team. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

How to manage fail sync for every stream or connection ?