Hey @kapa.ai can I safely delete the airbyte_internal dataset in BigQuery? This dataset has greatly increased in size and seems to also keep inactive sync data. It doesn't clean itself for some reason.

The answer depends on which destination connector version you're using. If you're using the newer connector versions (Direct Load) For Postgres destination v3.0.0+ and Snowflake destination v4.0.0+, these connectors use a Direct Load architecture that writes data directly to final tables without using raw tables. In this case, you can safely delete the

airbyte_internal

schema/dataset, as new syncs will no longer write to it. [Postgres Migration Guide] For BigQuery, the situation is less clear-cut. The knowledge sources do not explicitly confirm that BigQuery has fully migrated to Direct Load in the same way. Why airbyte_internal grows and doesn't clean up This is a known issue. The raw tables in

airbyte_internal

are not automatically truncated or consolidated after syncs. Every time a record is re-read from the source (e.g., because a modified timestamp was touched), a new copy is appended to the raw table — even if the data is identical. This can result in significant duplication (e.g., 5× or more storage overhead). [Excessive duplication issue] Before deleting airbyte_internal in BigQuery ⚠️ Do not delete it blindly if your syncs are still actively using it. The raw tables serve as the source of truth for the typing & deduplication process. If you delete them while syncs depend on them, you could break your pipelines. Safe steps to consider: 1. Verify your BigQuery connector version — check if it supports Direct Load (which eliminates the need for raw tables). 2. If still on the older architecture, you can manually trigger a soft reset instead of deleting: drop the final table and reset

_airbyte_loaded_at

to NULL in the raw table, then re-sync. [Typing & Deduping] 3. If you've confirmed Direct Load is active and syncs are completing successfully without using raw tables, you can safely drop the old raw tables from

airbyte_internal

to reclaim storage. [Postgres Migration Guide] Also note: there is a separate known issue where BigQuery destinations in different regions can conflict over the

airbyte_internal

dataset name — so if you have multiple BigQuery destinations, be cautious. [BigQuery region issue] In summary: check your connector version first. If you're on a Direct Load-capable version and syncs are working correctly, deleting

airbyte_internal

is safe. Otherwise, proceed with caution. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.