https://linen.dev logo
g

gunu

02/07/2022, 7:20 AM
CDC incremental dedupe question here
Copy code
2 connections:
- MySQL --> S3 (Incremental Append)
- S3 --> Snowflake (Incremental Dedupe)
Once the data is written to S3, it now contains the additional metadata
Copy code
{
  "_airbyte_ab_id": "b00c41e6-8a2f-4ed7-a10f-123",
  "_airbyte_emitted_at": 1644216617782,
  "_ab_cdc_log_pos": 123,
  "_ab_cdc_log_file": "mysql-bin-changelog.123",
  "_ab_cdc_updated_at": "2022-02-07T06:04:14Z"
}
when configuring the S3 --> Snowflake connection: the cursor field is source-defined but for the primary key, can I now use one of the metadata columns e.g.
_airbyte_ab_id
or do i still need to use the primary keys that were defined in the original MySQL table?
1
m

Marcos Marx (Airbyte)

02/09/2022, 10:36 PM
It’s better to choose the PK defined in original MySQL table
4 Views