Hello All, I am exploring Flink for ETL, to migrate data from a legacy database system to a modern one. Both DB share different schema designs.
It has to be a Historical data migration as well as changes have to be migrated on a real-time basis as both systems would be online. Further, I need to think of a way to do bi-directional sync as well.
Currently, I am capturing Leagcy DB CDC into Kafka. MSSQL -> Debezium -> Kafka -> Flink -> Postgres
I have also built a small POC, which uses Flink rocksdb, checkpoints, and Flink SQL tables with Kafka Connector as Source and Postgres as Sink connector, in between performing SQL-based joins, and things are working fine.
I have regarding stats that I don't need to maintain states in my use case as it will be event-based let's, So can I avoid using states in Flink?