Running airbyte 0.40.3
Source: postgres 1.0.5 conn...
# advice-data-ingestionr
Rocky Appiah
09/15/2022, 6:05 PMRunning airbyte 0.40.3
Source: postgres 1.0.5 connector
Destination: snowflake 0.4.34 connector
Running:
Copy code
>select sent, _airbyte_emitted_at from TABLE_SCD where uuid = 'myid' order by 2;
+-------------------------+-------------------------------+
| SENT | _AIRBYTE_EMITTED_AT |
|-------------------------+-------------------------------|
| 2022-09-05 22:30:19.000 | 2022-09-15 15:40:14.684 +0000 |
| NULL | 2022-09-15 16:56:56.121 +0000 |
+-------------------------+-------------------------------+sent Copy code
>select sent, _airbyte_emitted_at from TABLE where uuid = 'myid' order by 2;
+-------------------------+-------------------------------+
| SENT | _AIRBYTE_EMITTED_AT |
|-------------------------+-------------------------------|
| 2022-09-05 22:30:19.000 | 2022-09-15 15:40:14.684 +0000 |
+-------------------------+-------------------------------+Rocky Appiah
09/15/2022, 6:08 PMsource table is NULL, which matches the SCD table
Rocky Appiah
09/15/2022, 7:20 PMCreated issue here
Rocky Appiah
09/16/2022, 12:17 PMany insight @Marcos Marx (Airbyte)?
m
Marcos Marx (Airbyte)
09/16/2022, 7:10 PMI’ll follow the Github issue @Rocky Appiah but please check your source configuration
r
Rocky Appiah
09/16/2022, 7:15 PMWhat specifically in my source configuration do I need to check?
Rocky Appiah
09/16/2022, 7:16 PMDid you see my update? It looks like the CDC_UPDATED_AT is wrong.
I do have multiple replication slots, one for smaller tables and another for larger tables. Airbyte uses separate replication slot for each connection, but uses a single publication for all tables - is that OK?
m
Marcos Marx (Airbyte)
09/16/2022, 7:17 PMmaybe it’s a limitation and you need to separate the tables into two different connections
Marcos Marx (Airbyte)
09/16/2022, 7:17 PMI need to dig deeper to understand correctly the issue 😅
r
Rocky Appiah
09/16/2022, 7:19 PMok - your help is greatly appreciated, thank you.