Hello everyone,
I am reaching because of a challenge we are facing with Apache Flink and the sqlserver-cdc connector. Our current setup involves pulling changes from tables in MSSQL, using the connector, and creating views from these tables. We then write these changes to a Kafka topic with the upsert-kafka connector.
However, we are encountering some difficulties due to the complexity of the views and the large size of the SQL tables (some tables are ranging from 9 to 25 million rows). Running the Flink job with a parallelism of 1 is not feasible in our case, as it significantly affects the performance. On the other hand, running it in parallel introduces ordering issues.
Considering these constraints, what is the best practice to handle this scenario?