@kapa.ai Setting up a MySql->Snowflake connection, using CDC Incremental Append, and selecting a stream tells me "Primary key missing". The table is basically a key-to-key link table. ie, the table is two fields both foreign keys to the same table so we can say entity A is linked with entity B with A + B being unique id keys from another table. If we tell Airbyte via it's gui that both fields are the primary key, will it still be able to handle CDC appropriately?

Thinking completed No, defining the primary key in the Airbyte GUI will not be sufficient for CDC Incremental syncs. Here is why and what you should do instead: 1. CDC Incremental requires a source-defined primary key For database CDC replication in Airbyte (including MySQL), the connector automatically identifies the primary key directly from the source table's metadata [Set up the CDC connection with incremental dedupe synchronization; Source-Defined Primary key]. In Airbyte's CDC architecture: • A table must have at least one primary key defined in the source database to be eligible for CDC incremental syncing [Troubleshooting MySQL Sources; Limitations]. • Tables lacking a source-defined primary key are treated as ineligible for CDC incremental replication and can only be replicated in Full Refresh mode [fix(source-mysql): promote to Bulk CDK 1.1.10 (field decoration for full refresh CDC streams with no primary key); Troubleshooting MySQL Sources]. • User-defined primary keys selected in the Airbyte UI are intended for downstream deduplication [User-Defined Primary key], but they do not bypass the requirement that the underlying CDC connector requires a source-level primary key to read incrementally [Troubleshooting MySQL Sources; fix(source-mysql): promote to Bulk CDK 1.1.10 (field decoration for full refresh CDC streams with no primary key)]. 2. Can you use both fields as a composite primary key? Yes, if you define it at the database level. Airbyte supports composite primary keys for MySQL CDC [fix(source-mysql): include the lower bound of the first split partition for composite primary keys]. Because your link table has two columns (say,

entity_a_id

and

entity_b_id

) that together uniquely identify a record, you can add a composite primary key directly in MySQL:

ALTER TABLE your_link_table
ADD PRIMARY KEY (entity_a_id, entity_b_id);

Once the primary key is created in MySQL: 1. Go to your Airbyte connection and click Refresh source schema [Limitations]. 2. Airbyte will discover the source-defined composite primary key automatically [Set up the CDC connection with incremental dedupe synchronization; Source-Defined Primary key]. 3. You will then be able to enable CDC incremental replication for that stream without error [Troubleshooting MySQL Sources]. Alternative options If you are unable or unwilling to alter the table schema in MySQL: • Replicate the table using Full Refresh mode instead of Incremental CDC [Troubleshooting MySQL Sources; Limitations]. • Configure a separate, non-CDC MySQL source connection in Airbyte for this table and replicate it using standard cursor-based incremental sync (e.g., using an

updated_at

timestamp) [Limitations]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.