Hi Flink community, I need a design suggestion for Flink use case of data migration, Problem Statement: We have huge number of records in our DB, and we wanted to copy data from source database to destination DB. Description: We have Source DB where we have almost 50+ tables in Database, all we wanted to migrate data from source DB[TimescaleDB/PostgreSQL] to destination DB[TimescaleDB/PostgreSQL] using Flink and scala. If we insert all these records sequentially, would not be feasible for huge number of records. Having said, we need to make sure our Job should maintain integrity. [All or None records should transfer] Pseudo Code for Batch Job[1-HR Interval]: 1. Initialize stream and table execution env. 2. Connect to default db 3. Check if data migration request present if no stop the flow 4 Otherwise read source and destination db details 5. Connect to source DB and start reading tables data 6. Temporarily stores data in dataset 7. Connect to destination DB and start adding data from temporary storage/Dataset to respective destination table 8. Follow steps 5 to 7 for all tables 9. Update status all data copy has been completed 10. Stop the job So I need a suggestion on 1. Is there Flink plugin or module support for data migration? 2. What is the feasible way to migrate data from one DB to other DB using Flink? 3. Does sequential record copy recommended in this scenario where table count and record count is huge? 4. Please share link/resources if any for this use case.

This looks like a use case for Apache Spark! why not use that?

If you have a lot of data (billions of rows) I recommend looking at database-native tools for copying data, e.g. https://www.postgresql.org/docs/current/sql-copy.html Doing it with any streaming engine is going to be significantly slower. If it’s not the case, take a look at: • https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/jdbc/ • https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/jdbc/ • https://ververica.github.io/flink-cdc-connectors/