Hey, I'm using the *CDC replication mode* for sou...
# advice-data-ingestion
n
Hey, I'm using the CDC replication mode for source. and
sync mode : Incremental -> append
. But the can't find a way to define a cursor field. `Airbyte Version`: v0.40.0-alpha
source
: pgSQL v12 `destination`: pgSQL v12 Logs:
Copy code
2022-09-14 21:57:19 [44msource[0m > 2022-09-14 21:57:19 [32mINFO[m i.a.i.s.r.s.CursorManager(createCursorInfoForStream):151 - Found matching cursor in state. Stream: AirbyteStreamNameNamespacePair{name='test_1_default_pk', namespace='test_cdc'}. Cursor Field: null Value: null
On web UI: I can't find option either to update it / or it's locked. Can anyone help with this ?
a
@Nishant Soni with CDC, you don't need a cursor field. The equivalent behavior is controlled in your source database replication slot/publication. I'm probably abstracting this a little bit, but with CDC, the "cursor" is the WAL segment.
n
I have made all the WAL changes as per documentation, followed by service reload as well. But if i make update/ delete changes, and sync the connection. The changes are not reflected in the destination. And above logs says cursor field to be null. ( In addition even configuring CDC with pgSQL, somewhere in documentation it mentions supported cursors. and i have created same in source table
( updated_at ) with dtype : timestamp with zone
. 1. https://docs.airbyte.com/integrations/sources/postgres/#configuring-postgres-connector-with-change-data-capture-cdc 2. ( under same documentation ) -> https://docs.airbyte.com/integrations/sources/postgres/#supported-cursors If you need anymore details, that can help debug. I'm happy to share that.
a
That log is expected. We see it in all of our postgres CDC syncs. You want to look for logs like the following: • "Obtained valid replication slot ReplicationSlot" • "Found previous offset PostgresOffsetContext" • "Retrieved latest position from stored offset" • "WAL resume position <> discovered"