Hello All, I am exploring Flink for ETL, to migrat...
# troubleshooting
m
Hello All, I am exploring Flink for ETL, to migrate data from a legacy database system to a modern one. Both DB share different schema designs. It has to be a Historical data migration as well as changes have to be migrated on a real-time basis as both systems would be online. Further, I need to think of a way to do bi-directional sync as well. Currently, I am capturing Leagcy DB CDC into Kafka. MSSQL -> Debezium -> Kafka -> Flink -> Postgres I have also built a small POC, which uses Flink rocksdb, checkpoints, and Flink SQL tables with Kafka Connector as Source and Postgres as Sink connector, in between performing SQL-based joins, and things are working fine. I have regarding stats that I don't need to maintain states in my use case as it will be event-based let's, So can I avoid using states in Flink?
m
If you're doing a join, then most likely there is a need to maintain state in Flink. See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#regular-joins