@kapa.ai I am currency syncing data from ASA, however because of the schema change, Airbyte refetched all the data from the very beginning with new fields. The data record is too much, and couldn't sync successfully. What could I do? or could I do the stream or data chunking to update?

Airbyte’s documented behavior is that when it detects certain schema changes (like new columns/fields), it may re‑sync the entire stream so that the new columns are fully populated. This can indeed cause very large backfills and failed syncs if the dataset is big. [Schema change mgmt; Backfill new columns] From the docs, here are the levers you do have: 1. Control automatic backfill of new/renamed columns If you have “Backfill new or renamed columns” enabled, Airbyte will re‑sync the entire stream whenever a new column is detected via auto‑propagation. Disabling this avoids the full backfill; only rows updated in normal incremental syncs will get values in the new column. [Backfill new columns] ◦ This backfill only happens when Detect and propagate schema changes is set to: ▪︎

Propagate field changes only

, or ▪︎

Propagate all field and stream changes

, and the schema change is detected as part of a sync. [Backfill new columns] 2. Adjust “Detect and Propagate Schema Changes” behavior On the connection’s Settings tab, you can choose: [Detect & propagate] ◦ Propagate field changes only – only column changes are propagated; new/removed streams are ignored. ◦ Propagate all field and stream changes – all new streams and column changes are auto‑propagated. ◦ Approve all changes myself – schema changes are detected but not applied until you manually approve them; syncs continue with the old schema. ◦ Stop future syncs – connection pauses as soon as any schema change is detected. For very large datasets where a full re‑sync is failing, you can: ◦ Switch to “Approve all changes myself” so Airbyte doesn’t auto‑apply schema changes and trigger big backfills. ◦ Then, selectively approve schema changes when you’re ready, and optionally clear/refresh only when you can handle the load. 3. Understand what happens per schema change type For context, when a new column/field appears, Airbyte will create it in the destination and fill values for updated rows; a full backfill can be done either manually (clear/refresh) or via the Backfill new or renamed columns option. [Schema change mgmt] Other changes (like removal of columns/streams, data type changes, cursor/PK removal) have different behaviors, but none of the docs mention “chunking” a single stream’s backfill into smaller pieces beyond the normal sync behavior. [Schema change mgmt] ------------------------- About “stream or data chunking” The provided documentation does not describe a feature to automatically split a single large stream’s historical backfill into smaller chunks specifically to handle schema‑change backfills. It only describes: • How schema changes are detected and propagated. • How full‑stream backfills are triggered for new/renamed columns. • How to pause or manually approve schema changes. [Schema change mgmt; Backfill new columns] So, based on these docs, your practical options within Airbyte’s documented capabilities are: 1. Disable automatic backfill of new/renamed columns (if enabled). 2. Change “Detect and propagate schema changes” to “Approve all changes myself” to avoid surprise full re‑syncs. 3. Manually control when to clear/refresh streams so you only trigger a full backfill when your infra can handle it. If you share how your connection is currently configured (those two settings), I can help you choose the safest combination given your large ASA dataset, within what’s described in the docs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Sorry @Yenying Chen, I don't have permission to access files in this workspace. Please ask a workspace admin to reinstall the kapa.ai bot to grant the required permission.

@kapa.ai Detect and propagate schema changes is approve all changes myself now

Given your setting is already “Approve all changes myself”, Airbyte will: • Detect schema changes but not automatically propagate them; syncs continue with the last saved schema until you manually approve changes. [Detect & propagate] From the docs, here’s what you can do to avoid another huge backfill that fails: 1. Do not enable automatic backfill for new/renamed columns The “Backfill new or renamed columns” option is what causes a full re‑sync of the entire stream when a new column is detected via auto‑propagation. [Backfill new columns] ◦ Make sure this option is disabled on your connection. ◦ With it disabled, new columns will be created, but only rows updated in normal incremental syncs will get values; there is no automatic full‑stream backfill. [Backfill new columns] 2. Manually review and apply schema changes (without clearing/refreshing) With “Approve all changes myself”: ◦ Go to Connections → [your connection] → Schema. ◦ If changes are detected, click Review changes, then Save changes so the new fields are added going forward. [Detect & propagate] ◦ Do not clear/refresh the stream unless you explicitly want a full historical re‑sync; clearing/refreshing is what forces a full reload of all data. 3. Be aware of breaking changes If a cursor or primary key is removed, Airbyte will pause the connection and require manual review regardless of your setting. [Schema change mgmt; Detect & propagate] In that case you must fix the cursor/PK and then choose whether to refresh/clear (full backfill) or just continue. ------------------------- The documentation does not mention any built‑in “chunking” mechanism for splitting a single stream’s historical backfill into smaller pieces; it only covers the above schema‑change and backfill controls. [Schema change mgmt; Backfill new columns] So, to handle your large ASA dataset: • Keep “Approve all changes myself”. • Disable “Backfill new or renamed columns”. • Apply schema changes without clearing/refreshing streams, so you avoid another full re‑sync. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I didn't use “Backfill new or renamed columns”. as I mentioned, and I applied schema changes with refreshing streams. You said that without refreshing, there can be unexpected errors.

You’re right to call this out, and I need to clarify. From the documentation: • When “Approve all changes myself” is selected, Airbyte will detect schema changes but not propagate them until you manually approve and save; syncs continue with the last saved schema. [Detect & propagate] • For new columns/fields, the destination column is created and values are filled for updated rows; a full backfill only happens if you either: ◦ enable “Backfill new or renamed columns”, or ◦ manually refresh/clear the stream, which forces a full re‑sync. [Schema change mgmt; Backfill new columns] The docs do not say that not refreshing will cause “unexpected errors.” That was my own extrapolation and is not supported by the sources. What the docs do say is: • If you refresh or clear a stream after schema changes, the stream and its historical data are reloaded, which is what caused your huge backfill. [Schema change mgmt] • For breaking changes (cursor/PK removed), Airbyte pauses the connection and requires you to fix the cursor/PK and then choose whether to refresh/clear; it “highly recommends” refreshing or clearing to ensure future syncs are successful, but that’s a recommendation, not a guarantee of errors if you don’t. [Detect & propagate] Given your situation: • You did not use Backfill new or renamed columns. • You did apply schema changes with refreshing streams, which (per docs) triggers a full re‑sync and explains why all data was fetched from the beginning. [Backfill new columns] To avoid this next time with a very large dataset: 1. Keep “Approve all changes myself”. 2. When schema changes are detected, approve and save them without refreshing/clearing streams, so only new/updated rows are synced with the new fields. 3. Only use refresh/clear when you explicitly want and can handle a full historical reload. The docs do not describe any built‑in “chunking” mechanism for splitting that full backfill into smaller pieces. [Schema change mgmt] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai yes I already did Only use refresh/clear when you explicitly want and can handle a full historical reload, so now I couldn't do anything ?

You’re not completely stuck, but the options the docs describe are limited once you’ve already triggered a full refresh. From the documentation: • Refreshing/clearing a stream after a schema change forces Airbyte to reload the entire stream and its historical data. That’s why it started from the beginning and is now struggling with volume. [Schema change mgmt] • There is no documented feature to “chunk” that historical reload into smaller pieces or to partially refresh only a subset of the historical data. [Schema change mgmt] What you can still do, based on the docs: 1. Stop the failing full refresh and continue incrementally The docs don’t describe a “resume from middle of refresh” feature. Your practical path is: ◦ Let the current full refresh fail or stop it. ◦ Ensure the connection is set to “Approve all changes myself” (which you already have). [Detect & propagate] ◦ Apply the schema changes (if not already applied) without refreshing/clearing again, so future syncs only process new/updated rows with the new fields. This means you may have: ◦ Old historical rows without values in the new columns. ◦ New/updated rows correctly populated going forward. 2. Avoid further full reloads Going forward, to prevent this situation again: ◦ Keep “Approve all changes myself”. ◦ Keep “Backfill new or renamed columns” disabled. [Backfill new columns] ◦ Do not use refresh/clear on large streams unless you truly want a full historical reload and your infra can handle it. Because the documentation does not mention any way to: • Partially backfill only a slice of history, or • Automatically chunk a full backfill into smaller pieces, there isn’t a documented, built‑in mechanism in Airbyte to “salvage” the already‑started full refresh other than stopping it and continuing with incremental syncs as described. [Schema change mgmt] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I have finished the very first time fully refresh of the records. However the connection turned out to be fetch very few records. What's the issues ? Other connections with the same streams change work well.

The knowledge sources don’t give a specific root cause for your exact case, but they do describe several patterns that match “full refresh finishes, but far fewer records than expected are loaded.” Here are the most relevant, documented possibilities to check: 1. Pagination / source-side limits There are known cases where a stream appears to run successfully but only a subset of records is read because pagination is not working as expected or the connector stops early without an error. Examples: ◦ A paginated custom stream to Snowflake only loaded about half the expected records even though the job “succeeded.” [Paginated stream issue] ◦ A Salesforce stream (OpportunityContactRole) only extracted ~2,300 records while SOQL showed 8,821; logs showed “Read 2303 records … Marking stream … as STOPPED” with no error and no further pages fetched. [Salesforce pagination] What to do: ◦ Check the job logs for your connection and look for lines like

Read X records from

stream followed by

Marking stream

stream

as STOPPED

or similar, without errors. [Salesforce pagination] ◦ Compare the count Airbyte reads (from logs) with the count you get by querying the source directly (e.g., API, database query). If Airbyte consistently reads fewer records, it’s likely a connector/pagination configuration issue. 2. Connector-specific bugs / partial loads The docs and issues show that sometimes a connector can stop fetching new data after an internal error or timeout, while the overall sync still appears “successful”: ◦ In one case, a full replace sync replaced destination tables with empty temp tables after a source error, because the destination assumed all data had been fetched. [Paginated stream issue] ◦ For MongoDB CDC, a bug caused every incremental sync to load exactly 101 records due to Debezium timing out on subsequent batches; increasing

initial_wait_time

was required. [MongoDB 101 records] What to do: ◦ Inspect logs for warnings/timeouts around the time the stream stops reading records, even if the job status is “success.” [Paginated stream issue; MongoDB 101 records] ◦ If you see a consistent pattern (e.g., always ~N records, or always stops after a certain time), it may be a connector bug or timeout that needs a configuration change or a bug report. 3. Schema change handling is not the main suspect here Since you say: ◦ Other connections with the same schema changes work well. ◦ You already completed the first full refresh successfully. The schema-change settings (

Detect and propagate schema changes

,

Backfill new or renamed columns

) mainly control when full backfills happen and whether streams are paused, not selective under-loading after a completed full refresh. [Detect & propagate; Backfill new columns] So for this specific symptom (very few records after a full refresh), the docs and issues point more toward pagination / connector behavior than schema-change configuration. ------------------------- Because the knowledge sources don’t include your exact connector or logs, I can’t say definitively which of these is happening. Based on what is documented, the next concrete steps are: 1. Compare counts: ◦ Query the source directly for the stream (e.g., “how many records exist?”). ◦ Compare with the count in the destination table after the full refresh. 2. Read the job logs for that connection: ◦ Look for

Read

N

records from

stream and any

STOPPED

messages or warnings/timeouts around that stream. [Salesforce pagination; Paginated stream issue] 3. If you see early stopping or partial reads with no clear config mistake, it’s likely a connector-specific issue similar to the Salesforce or paginated-stream examples above, and you may need to open a ticket or forum post with: ◦ Connector name and version ◦ Sync mode ◦ Excerpt of logs showing where the stream stops reading If you can share which source connector you’re using and (roughly) what the logs say about records read for the problematic stream, I can map it more precisely to the patterns described in the docs and issues. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai what is airbyte generation id mean?

_airbyte_generation_id

is an Airbyte metadata column that tells you which “generation” of data a record belongs to. • Airbyte increments this generation ID every time you run a Refresh sync (of either type: Refresh & retain history, or Refresh & remove records). [Data generations; Metadata fields] • It is a monotonically increasing counter: later refreshes have higher generation IDs; Airbyte never decreases it. [Data generations; Refresh FAQ] • Records synced before this feature existed will have

_airbyte_generation_id = null

. [Data generations] You can use it to: • Distinguish data before vs. after a refresh (e.g., to analyze or reconcile changes from a flaky source). [Data generations] • Compute metrics per generation (e.g.,

SUM(total_sales)

by

_airbyte_generation_id

) to see how a refresh changed your dataset. [Data generations] On DV2 destinations, when you Clear or run a Refresh that removes records, all pre‑existing data is deleted from raw and final tables; new data will come in with a new

_airbyte_generation_id

. [Refresh FAQ] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.