In the same airbyte connector, is it possible to write to different destination schemas or custom table names?

Thinking completed Based on Airbyte's core features and connection configuration, here is how writing to different schemas and customizing table names works: Destination Schemas (Namespaces) Within the same connection, your ability to write to multiple schemas depends on how the namespace is configured: • Source-Defined: If your source provides namespace information (common with database sources like Postgres, Oracle, or MySQL), setting the connection's destination namespace to Source-Defined allows streams to be mirrored into their corresponding distinct schemas/namespaces in the destination [Namespaces; Namespaces]. • Custom or Destination-Defined: If you select Custom or Destination-defined, all streams in that connection are routed to a single namespace/schema [Namespaces]. • Connector-specific overrides: Certain destinations are introducing advanced per-stream layout overrides. For instance, the BigQuery destination introduced support for advanced per-stream configurations allowing different streams in one connection to target different datasets (schemas) [Pull Request]. Custom Table Names Airbyte does not generally support renaming individual destination tables or streams: • Stream Prefix: Airbyte only provides a connection-level Stream Prefix setting. This prepends text to all stream/table names in that connection (e.g.,

prefix_<table>

), rather than allowing arbitrary names per table [Configuring schemas]. • Individual Renaming: Airbyte does not allow arbitrary per-stream renaming for most connectors [Configuring schemas; Forum Post]. However, specific connectors with per-stream table layout overrides (such as Destination BigQuery) support overriding destination tables and table suffixes on a per-stream basis [Pull Request]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai The only thing is we're using cdc connector and there are some tables in postgres with the same names but in different schemas

Thinking completed When using the Postgres CDC source with tables that have the same name in different schemas, you can handle this using the Destination Namespace settings on your connection: 1. Separate Them by Schema (Source-Defined Namespace) If you want each table to go into its respective schema in the destination, set the connection's destination namespace to Source-Defined (Mirror source structure) [Namespaces; Namespaces]. • For example,

schema_a.my_table

will write to

schema_a.my_table

in your destination, and

schema_b.my_table

will write to

schema_b.my_table

[Namespaces]. • You can also use a Custom namespace format with the dynamic parameter

${SOURCE_NAMESPACE}

(e.g.,

prefix_${SOURCE_NAMESPACE}

) to keep them separated [Namespaces; Namespaces]. 2. Avoid Destination Collision If your connection is configured with a Destination-defined namespace or a static Custom namespace (where all streams go into a single destination schema), tables with the same name will collide. Because Airbyte does not allow renaming individual streams within a single connection [Configuring schemas], all identical table names would attempt to write to the same destination table. Potential Multi-Schema CDC Limitation In older configurations or community discussions, users have reported issues when configuring a single Postgres CDC source to read across multiple schemas at once [Forum Post]. If you encounter issues reading multiple schemas under one CDC source, a common workaround is to create two separate Postgres sources and connections (one for each schema), each with a unique Stream Prefix or unique destination namespace [Configuring schemas; Forum Post]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Is it a good idea for multiple airbyte connectors to point to the same replication slot?

Thinking completed No, it is not a good idea—in fact, it is explicitly unsupported and will cause failures. Airbyte requires that each replication slot is dedicated solely to a single source connector/connection: • Exclusive slot usage: Airbyte documentation explicitly states that it requires a replication slot configured only for its own use, and only one source should be configured to use that slot [Documentation; Forum Post]. • Data loss / Incomplete syncs: Postgres logical replication slots track progress using Log Sequence Numbers (LSN). Because each consumer advances the slot's LSN position as it reads and confirms data, sharing a slot causes consumers to compete: each destination/connector would only receive part of the data stream [Documentation]. • Connection and LSN conflicts: If multiple connectors or concurrent processes attempt to use the same slot simultaneously, it can trigger connection errors stating that the replication slot is already in use [Forum Post], or crash syncs due to unexpected slot advances [ReplicationSlotManager.kt]. If you need to set up multiple Airbyte connections or sources reading from the same Postgres instance, you should create separate replication slots for each connection [Forum Post]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai how does "Source-Defined Namespace" work with the default dataset ID?

Something went wrong

@kapa.ai @kapa.ai how does "Source-Defined Namespace" work with the default dataset ID?

Something went wrong

@kapa.ai how does "Source-Defined Namespace" work with the default dataset ID? I want to name the destination tables per schema, but when it's public i don't want the schema name as prefix. e.g. public.accounts --> destination_dataset_name.accounts, audits.accounts --> destination_dataset_name_audits.accounts. It's best if we can add the schema name as prefix to the table name where I want

Something went wrong