@kapa.ai I am investigatiment a problem I has with Airbyte OS and the Salesforce connector. Scenario: I setup a couple dozens Streams to be synced incrementally from Salesforce to my Snowflake datawarehouse. The synchronization generally looks fine, and comparing records with those present at the source, there were no discrepancies. At some point, there was a Schema change on one of the streams. The schema change was the addition of a new field. The connection is setup to "Propagate field changes only". The schema change was incorporated automatically during the night, but in the morning I observed a few unexpected and erroneous behaviours: • several records were missing in the destination table: roughly 200k over a total 500k • records were missing also on the corresponding internal raw table, the one maintained by Airbyte, from where the final data is written in the destination table ◦ this is particularly surprising, as I thought this table is append only • I tried to re-ingest that stream from scratch, into another destination, using a separate connection, and it fetched all the records from the source How could have this happened? How can I prevent similar episodes to happen again?

This is an interesting and concerning scenario. Let me address what the knowledge sources can and cannot explain about your situation. What the Knowledge Sources Can Explain Schema Change Handling in Incremental Sync The Airbyte documentation explicitly warns about this: > "The current behavior of Incremental is not able to handle source schema changes yet, for example, when a column is added, renamed or deleted from an existing table etc. It is recommended to trigger a Full refresh - Overwrite to correctly replicate the data to the destination with the new schema changes." [Incremental Append - Known Limitations] This suggests that schema changes and incremental sync do not mix well, and the addition of a new field could have triggered unexpected behavior. Schema Propagation Behavior With

Propagate field changes only

, Airbyte automatically propagates column additions to the destination. The documentation notes: > "New Column/Field: The new field will be created in the destination. Values for the field will be filled in for the updated rows. If you are missing values for rows not updated, a backfill can be done by manually triggering a refresh or through the

Backfill new or renamed columns

option." [Schema change management] This means rows that were not updated at the source after the schema change would not automatically have the new field backfilled — but this alone doesn't explain 200k missing records. What the Knowledge Sources Cannot Fully Explain The knowledge sources do not provide enough information to definitively explain: • Why ~200k records would go missing (not just have a null new field) from both the destination table and the internal raw table • Why the raw/internal table — which is expected to be append-only — would have missing records These are significant anomalies that go beyond what is documented in the provided sources. Recommendations Based on the Documentation 1. Avoid relying solely on incremental sync after schema changes. The docs explicitly recommend triggering a Full Refresh - Overwrite when schema changes occur to ensure correct replication. [Incremental Append - Known Limitations] 2. Use the backfill option for the new column to ensure historical rows are populated. [Schema change management] 3. Consider CDC (Change Data Capture) if your source supports it, as it is more robust for capturing changes without relying on cursor fields. [Incremental Append - Known Limitations] 4. For the missing records issue specifically, I would strongly recommend opening a support ticket with Airbyte, as the raw table data loss in particular is not explained by the available documentation and warrants direct investigation by the Airbyte team. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai What do you mean by "trigger a Full Refresh overwrite" ? I want the stream to stay incremental. I can surely restart from scratch (just for that stream!), but I want that subsequent refreshes continue incremtally

Great clarification! You're right that "trigger a Full Refresh - Overwrite" doesn't mean you have to permanently switch the stream to that mode. Here's what it means in practice: A one-time Full Refresh - Overwrite means running a single sync that wipes the destination table and re-ingests all records from scratch — just for that stream. After that one-time reset, you can switch the stream back to incremental mode for all subsequent syncs. However, the knowledge sources do not explicitly describe a step-by-step UI workflow for doing this on a per-stream basis (i.e., temporarily switching one stream to Full Refresh - Overwrite, running it, then switching back to Incremental). What the documentation does confirm is: • Full Refresh - Overwrite replaces all existing data in the destination table with fresh data from the source, without creating new tables. [Full Refresh - Overwrite] • Schema changes are problematic for incremental sync, and a full refresh is the recommended remedy for getting the destination back to a correct state. [Incremental Append - Known Limitations] • When a breaking schema change is detected, Airbyte may prompt you to refresh or clear the affected streams — which is essentially this same concept applied per-stream. [Schema change management] So your intended approach — re-ingest that one stream from scratch once, then continue incrementally — is conceptually aligned with what Airbyte recommends. For the exact mechanics of how to trigger a per-stream refresh in the UI without permanently changing the sync mode, I'd recommend consulting Airbyte support or the community, as the knowledge sources don't provide enough detail on that specific workflow. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Here some more details. Running this query against the internal table for the connection I used to experiment:

select * from raw."airbyte_internal"."salesforce_spool_raw__stream_Opportunity"
where "_airbyte_data":Id = '...omissis...'

I got two records, with two different

_airbyte_generation_id

. Running the same query but against

raw."airbyte_internal"."salesforce_raw__stream__Opportunity

(which is the internal table corresponding to my "official" connection, I get no records at all for that Id. Can you explain this?

The

_airbyte_generation_id

column is directly relevant here. Based on the knowledge sources: > "Every time a Refresh Sync (of either type) occurs, the generation increases.

_airbyte_generation_id

is a monotonically increasing counter which maps to this notion of 'generation' of data." [Refreshing your data] So the two records you found in your experimental connection's raw table having two different

_airbyte_generation_id

values indicates that a refresh sync occurred on that connection at some point, producing two generations of data for the same Salesforce record. Regarding the official connection's raw table having zero records for that Id — the knowledge sources do not provide enough information to explain this definitively. This is the same anomaly you originally reported (records missing from the raw table, which is expected to be append-only), and it remains unexplained by the available documentation. What the knowledge sources can confirm is that if a Full Refresh - Overwrite (non-history-retaining) was triggered on the official connection, the raw table would have been wiped and re-written. If that refresh failed or was incomplete, records could be missing. But this is the extent of what can be inferred — the knowledge sources do not describe a specific failure mode that would cause records to disappear from the raw table without a deliberate refresh action. I would strongly recommend raising this with Airbyte support directly, as the raw table data loss is not explained by the available documentation and requires investigation of your specific connection's sync history and logs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.