Hey everyone!
I’m currently using Flink 1.20.0, flink-connector-mongodb-cdc 3.1.1, and flink-bigquery-connector 1.0.0 (by Google Cloud Dataproc).
The current architecture is:
MongoDB → Flink CDC → BigQuery.
When triggering a snapshot from MongoDB, I’m using the DataStream API format from the flink-bigquery-connector with the sink parallelism set to 1. However, the loading speed seems extremely slow—I’m only able to load around 30 million records in 4 hours.
Is there any way to speed this up or optimize the setup?