Khai Quang Nguyen

01/13/2022, 5:36 PM
Hi, testing Airbyte right now to connect from Postgres to Snowflake. An issue that I have seen repeatedly is on incremental-dedupe settings, Airbyte fails on subsequent syncs with the following error.
Copy code
2022-01-13 04:07:36 source > 2022-01-13 04:07:36 ERROR i.d.r.TableSchemaBuilder(lambda$createValueGenerator$5):269 - Failed to properly convert data value for 'public.investment_listing.updated_at' of type timestamp for row [null, null, null, null, 213, null, null, null, null, null, null, null, null, null, null, 917bb142-db72-4ae4-8a5c-7398db43f527, null]:
2022-01-13 04:07:36 source > org.apache.kafka.connect.errors.DataException: Invalid value: null used for required field: "updated_at", schema type: STRING
this seems like an issue with postgres DELETE since WAL will just store the primary key for cases like this. A potential solution I have seen from a PR is to change the replication identity to FULL, but this has performance issues on very large table. Any workaround for this? I would imagine this should have been a very common use case.
hi! we encountered the exact same problem a while ago, and opened this ticket.
Hi @Khai Quang Nguyen, you can try to go through the workaround suggested by smartpierre on the issue above. Our team will look into this soon to provide a fix.