Good Morning - I have a usecase where we need to r...
# troubleshooting
n
Good Morning - I have a usecase where we need to read from Postgres CDC for a bunch of tables and write to respective iceberg tables. For Flink Postgres CDC connector, I see the source connector is reading from WAL files through the replication slot and publication option where you get to read the information for required tables but to reliable write to iceberg into respective table… any input on how anyone here is already doing it - to properly read per table specific data (from that single threaded pull from source) and flush it in reasonable/reliable to the iceberg sink.
p
Hello there! Sorry, I did not understand why reading from WAL would be an issue here. Would you like to fetch the data in another way?
n
@Pedro Mázala - thank you for getting back, sorry for the delay in response. No issue per say from reading but let’s say I have 200 tables for which I need to pull CDC data and write respective data into 200 target tables (in my case iceberg tables in Lakehouse for analytics)… if I am doing this through a replication slot / publication or even if I break that into 20 replication slots with 10 tables each in a publication, wouldn’t keeping a counter and trying to replay if there is an issue be a problem ? Also if there is one table which has lot of changes coming in to WAL would that not make the data for other tables making it a single pipe way of writing data than independent or parallel pipe for each table ? cc: @Venkata Raman T
p
That's a great question, @Nageen Gandikota On the throughput side of things, I'd say you'd be safe with a single map (Postgres <> Iceberg). In the other hand, if you have something tier 0 and you'd like to isolate it from others (more looking to the perspective of disaster control), perhaps it makes sense to have multiple instances each looking to their own replication. I the past I had a setup which was a single Postgres database with 5 different debezium instances: • 5
tier 0
tables from a specific domain • 3
tier 0
tables from another domain • general high throughput tables • low throughput tables • outbox pattern table So i'll give you the infamous answer that
it depends
and ties up directly with the way you will want to handle it 😄
h
@Nageen Gandikota did you find a solution because same thing I am trying . I am able to do with the Flink Pipeline but not with the DataStream or FlinkSQL. @Pedro Mázala Do you have any reference where I am looking Single Flink CDC instance mapper to 1 postgress instance having 8 tables and synced data to Iceberg.