@kapa.ai Hello. I have two connections C1 and C2 replicating a stream S1 full_refresh_overwrite. C1 is refreshing in the night C2 is refreshing at noon and in the evening For several months, everything was going ok, stream refreshed everytime with no duplicates. until several days, every morning the stream is doubled. why? Here are some tests I did today to check the behavior: 15h C1: gen_id 508 -> duplicates 15h15 C2: gen_id 509 -> ✅ 15h25 C1: gen_id 509 -> duplicates 15h37 C1: gen_id 268 -> ✅ • What should I do • Why do these connections behave so? • what are the best practices to handle streams in several connections?

New message text here

@kapa.ai thanks, I have more questions: • I was working for months before, why is it not working now? • I am using several connections because C1 is handlings streams during the night and C2 is handling fewer streams (but some are common) during the day. • What should I do?

I’m constrained to what’s in the docs and linked issues, so I’ll be explicit where they don’t say “why”. ------------------------- 1. “It worked for months, why is it not working now?” The docs don’t explain changes over time in your specific setup, but they do say: • Airbyte no longer supports multiple connections writing to the same destination stream; unique stream identifiers (name + namespace) are now mandatory. [Configuring schemas; 1.6 release note] • There is an open bug (for destination-postgres) where Full Refresh | Overwrite appends instead of replacing when the destination table is shared across multiple connections, described as a regression introduced with Direct Load. [GH issue 74283] Putting only that together: • Your pattern (C1 and C2 both writing Full Refresh | Overwrite to the same table) is now explicitly unsupported and risky. [Configuring schemas] • A regression in at least one destination can cause overwrite to behave like append when a table is shared. [GH issue 74283] The docs do not say exactly when this behavior changed or which version you’re on, so I can’t state precisely why it started failing “several days ago”. The most defensible explanation from the sources is: • You were relying on an unsupported configuration that happened to work for a while. • A change (e.g., Airbyte or destination upgrade / Direct Load behavior) exposed the underlying risk, and now some runs append instead of overwrite. ------------------------- 2. “I’m using several connections because C1 is night, C2 is day with overlapping streams” This scheduling pattern is understandable, but the key constraint from the docs is: > “Don’t configure more than one Airbyte connection to sync to the same destination stream. Doing so can cause job failures and isn’t permitted in Airbyte.” [Configuring schemas] So even if: • C1 runs at night, and • C2 runs during the day, and • Only some streams overlap, it’s still an unsupported pattern as soon as any stream from C1 and C2 targets the same destination table (same name + namespace). The docs recommend that if you want a single logical table, you should not achieve that by having multiple connections write to the same table; instead, you should combine data downstream. [Configuring schemas] ------------------------- 3. “What should I do?” Based strictly on the documentation and release notes: 1. Stop having C1 and C2 write to the same destination stream. For every overlapping stream S1: ◦ Give C1 and C2 different destination identifiers by: ▪︎ Using a stream prefix per connection (e.g.,

c1_

and

c2_

), or ▪︎ Using different namespaces per connection. [Configuring schemas; Configuring schemas] This is exactly what the docs prescribe to avoid clashes: [Configuring schemas] > “To resolve this, you can: > • Provide streams in each connection with a unique stream prefix > • Provide namespaces in each connection with a unique namespace format > • Disable conflicting streams” 2. Keep your scheduling logic (night vs. day) but on separate tables. ◦ C1 can still run at night, C2 during the day. ◦ They just must not target the same destination table. 3. Create a unified view/table downstream in your warehouse. ◦ Once C1 writes to, say,

c1_s1

and C2 to

c2_s1

, create: ▪︎ A view or transformation that `UNION ALL`s (or otherwise merges)

c1_s1

and

c2_s1

into a single logical table you query. ◦ This is the recommended pattern in the docs: if you want a single table, combine it downstream, after Airbyte has synced it. [Configuring schemas; 1.6 release note] 4. (Optional, if you’re on Postgres) If you are using the Postgres destination and still see Full Refresh | Overwrite appending even when each connection has its own table, that would align with the open bug and should be checked against your destination version. [GH issue 74283] ------------------------- In short: • It “worked for months” because you were using an unsupported pattern that happened to behave as you expected until a change/regression surfaced the risk. • The official, supported way to handle your night/day split with overlapping streams is: ◦ Unique tables (via prefix or namespace) per connection, and ◦ Downstream consolidation into a single logical table. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai • I am using Bigquery destination 3.0.17 • Is it the same issue with streams append_deduped? should I split them into different stream prefixes?

For your two questions, here’s what the knowledge sources support. 1. BigQuery destination 3.0.17 and your issue The only destination-specific issue in the provided sources is for Postgres, where Full Refresh | Overwrite can append instead of replace when the same table is used by multiple connections. [Postgres FR bug] There is no information in the knowledge sources about this exact bug affecting BigQuery 3.0.17. So I cannot say it’s the same implementation issue on BigQuery. However, independently of bugs, your setup is in an unsupported pattern: > “Don’t configure more than one Airbyte connection to sync to the same destination stream… Doing so can cause job failures and isn’t permitted in Airbyte.” [Configuring schemas; 1.6 release note] This rule applies regardless of sync mode or destination type. 2. Does this also apply to

append_deduped

streams? Yes, from the docs’ perspective you should treat all sync modes the same regarding stream uniqueness: • The “stream uniqueness” rule is stated at the connection/schema level, not tied to a specific sync mode. [Configuring schemas] • It explicitly covers the general case of “multiple connections to the same destination stream”, which includes

incremental append_deduped

or any other mode. [Configuring schemas] So, yes: for streams using

append_deduped

that are written by both C1 and C2 to the same destination table, the docs say you should split them into different destination identifiers: • Use different stream prefixes per connection (e.g.,

c1_

vs

c2_

), or • Use different namespaces per connection. [[Configuring schemas](https://docs.airbyte.com/platform/using-airbyte/configuring-schema#modify-stream-names; Configuring schemas] Then, if you need a single logical table, combine them downstream in BigQuery (view or SQL job), which is the recommended pattern. [Configuring schemas] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai what do you mean by different namespaces?

In Airbyte, a namespace is the place in the destination where data is written – for databases this is usually the schema (or dataset in BigQuery). [Core concepts – destination namespace] Examples from the docs: [Namespaces – what is] • BigQuery → namespace = dataset • Postgres → namespace = schema • Snowflake → namespace = schema • S3 → namespace = path prefix When we say “use different namespaces” for C1 and C2, it means: • Configure C1 to write into one dataset/schema (e.g.

raw_c1

) • Configure C2 to write into a different dataset/schema (e.g.

raw_c2

) So even if both connections have a stream called

S1

, they land in different places: • C1 →

raw_c1.S1

• C2 →

raw_c2.S1

This satisfies the requirement that stream identifiers (name + namespace) must be unique across connections to avoid clashes. [Configuring schemas – stream uniqueness; Namespaces] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.