Hi. I'm checking possibilities of Apache Flink. I'...
# troubleshooting
p
Hi. I'm checking possibilities of Apache Flink. I'm trying to make some kind of "pipeline" which will be constantly syncing data from MongoDB to PostgreSQL when changes or new records in Mongo appears. How to achieve such behavior with Flink? All I have for now is a job in Java which is streaming data from Mongo to Postgres. But when I'm running the job, it works and finish. What should I setup (or can I setup) something to have the job working constantly and listening for changes in Mongo?
r
Have you looked at Debeizum? It's what the Flink CDC connectors are built on. https://debezium.io/documentation/reference/3.0/connectors/mongodb.html It also has a JDBC Sink that you could use to write to postgres https://debezium.io/documentation/reference/3.0/connectors/jdbc.html
p
Yep, I've heard once or twice but I was hoping that Flink will be some kind of "user friendly" overlay on it.
What about Flink CDC? Is it different from standard Flink?
What is the difference than CDC job is syncing constantly while the manually prepared job is starting and finishing?
r
CDC reads the binlog (or equivalent). A manual job has to poll for changes. Flink CDC is part of the Apache Flink project. It provides connectors to use from Flink. There is also a YAML-based option with pipeline connectors. Not all Flink CDC connectors have a pipeline connector tho; IIRC there's only mysql as a sink connector, for example.
p
So for YAML-based version I need the pipeline connector?
r
yes. I wrote this about it last year that might help: https://www.decodable.co/blog/exploring-flink-cdc
🙌 1
j
@rmoff Am I understanding that the standard
flink-connector-mongodb
source, only does an initial load from the source collection? I hadn't seen updates or new records flow after it did initial connection so I wasn't wondering if I had missed something. Is the cdc connector the only way to maintain state from mongo in Flink? My use case intends to use a Mongo collection for rule based routing, it doesn't change often but it does change. I'm fine with using cdc, but I was wondering how people are using the Mongo source if it didn't support updates. Single use config loads? Thank you for any feedback.
r
@Justin Mason I'm not familiar with the MongoDB source, sorry. You might also want to post your question outside of this thread as no-one will see it other than the three of us already on this thread 🙂
j
Thanks, will do