Hello Team, I am using the Debezium connector to ...
# troubleshooting
k
Hello Team, I am using the Debezium connector to listen to different database tables, perform aggregation, and finally publish the data to a NoSQL-based database. I have a questions: 1. Whenever I restart the Flink job/task, it reads all the data from the tables and republishes it. However, I only want to process new records from where the last job was stopped. I tried different snapshot.mode but all returning same result.
s
Are you using checkpoints and save points with exactly one semantics? Also please checkout if flink supports EOS with debeziun or not
k
Yes I am using checkpoint with interval=60000 and Exactly_ONCE
s
This is happening because debezium works with atleast once guarantee , now if i m understanding correctly your debezium must be dumping data to kafka and then from kafka you are consuming it and aggregating in flink and finally dumping to sink , if yes then you should leverage the some primary key to remove the duplicates
k
We are not using kafka
We have multiple CDC (Debezium) as a source then we aggregating data, and after that we have a sink with MongoDB