Hello all,
I have a question but first is this th...
# randomt
Theau Heral
02/20/2023, 2:29 PMHello all,
I have a question but first is this the right place to ask my question or should I email the dev mailing list?
My question is: There seems to good support in Flink for JDBC sinks but not as much for JDBC sources. I have looked at Debezium and ververica CDC Flink connectors but what I really need is a Kafka Connect style JDBC source with offsets/timestamp states based queries polling regularly for updates. (Why? DB admin refuses to turn on table capture mode). I would want to create and perhaps contribute it to Flink. Is it something helpful for the community, does it already exist or would the right pattern be to consume via Kafka hence it has not been implemented by choice?
For reference: https://github.com/confluentinc/kafka-connect-jdbc/tree/master/src/main/java/io/confluent/connect/jdbc/source
m
Martijn Visser
02/21/2023, 9:53 PMIt’s better to ask this on the Dev mailing list
t
Theau Heral
02/21/2023, 9:53 PMThank you, yes I subscribed and was wondering if I should send a message there instead
m
Martijn Visser
02/21/2023, 9:54 PMI would think that the pattern is kind of weird: it could put a large load on the database as a source
t
Theau Heral
02/21/2023, 9:54 PMTrue, but depends on your poll frequency.
m
Martijn Visser
02/21/2023, 9:55 PMYes and on the amount of records that you get with each pull
t
Theau Heral
02/21/2023, 9:56 PMThe thing I was more worried about was doing something too similar to Kafka Connect that it would be a problem with the license. I started working on it this morning though and I am doing a mix of Kafka, File source and Kafka connect.
m
Martijn Visser
02/21/2023, 9:56 PMParallelization is also a first thought: could you distribute this or would it effectively be a source with a parallelism of 1
Martijn Visser
02/21/2023, 9:57 PMYou can’t use code from Kafka Connect in any ASF project
t
Theau Heral
02/21/2023, 9:58 PMIf you have offsets/timestamps as a state there should not be that many rows per poll. Well depends on the user…
Theau Heral
02/21/2023, 9:58 PMRight I know I can’t use it so I am writing something from scratch
Theau Heral
02/21/2023, 10:00 PMOn parallelization, you can split it on partition or index keys. That’s how Kafka connect does it and hope it would also be done with a JDBC Flink source but yes most of the time you would probably want a single consumer. It also requires your table to be append only…
Theau Heral
02/21/2023, 10:00 PM*insert
m
Martijn Visser
02/21/2023, 10:01 PMThat’s a lot of constraints. It would be good to gauge interest in the Flink community but tbh I don’t see a lot of value to maintain that under the Flink umbrella
t
Theau Heral
02/21/2023, 10:01 PMCould be added to the connectors no?
m
Martijn Visser
02/21/2023, 10:02 PMIt’s not only the initial work, but also having multiple maintainers who want to keep it working in future changes of Flink, solve bug tickets etc
Martijn Visser
02/21/2023, 10:02 PMThe JDBC connector is already externalized, but adding a feature as this one (a source) would require a FLIP and a vote on it
t
Theau Heral
02/21/2023, 10:05 PMWhen looking at alternatives for CDC, I found Airbyte (entirely batch based so did not fit my use case) and it has so many connectors. I feel like it would be beneficial for Flink adoption to have more sources. It really makes it easier on the developer to have existing native sources instead of having to create your own. Increased JDBC source support in general would be better is my point I guess.
Theau Heral
02/21/2023, 10:06 PMAgreed that it’s added maintenance and would require a vote per bylaws. Read that too on confluence :)
Theau Heral
02/21/2023, 10:06 PMCDC is supported but can’t always be done (in my case)
Theau Heral
02/21/2023, 10:08 PMRight now the alternative for me is to use Kafka connect, create a Kafka topic per table and then consume it in Flink. To me it feels like the Kafka topic could be skipped with a JDBC source.
m
Martijn Visser
02/21/2023, 10:45 PMOh I agree that I would love to have more sources
❤️ 1
Martijn Visser
02/21/2023, 10:45 PMBut you do need maintainers for those
t
Theau Heral
02/21/2023, 11:24 PMTrue time and money as always. :)
Theau Heral
02/21/2023, 11:27 PMI’ll see how far I get with my JDBC source and come back on the dev mailing list to discuss it.
Theau Heral
02/21/2023, 11:28 PMThanks for the reply :)
j
Joao Boto
02/22/2023, 8:01 AMThere are a FLIP for a new source (and sink) on jdbc.. https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217386271
Joao Boto
02/22/2023, 8:03 AMWe have the same problem some time ago, we did a source based on a time field.. basically we search for modify rows after that time.. but we miss the deletions (that in our case rarely happen)
t
Theau Heral
02/22/2023, 8:24 AM@Joao Boto thank you! I will read this now. I couldn’t find this FLIP.
Yes agreed your table needs to be append only if you use this method or you take the entire table in each time…
Theau Heral
02/22/2023, 8:28 AM(Or a diff with unique ids..)
Theau Heral
02/27/2023, 12:10 PM@Martijn Visser @Joao Boto have you ever come across Chunjun? Any thoughts?
https://github.com/DTStack/chunjun
Theau Heral
02/27/2023, 12:11 PMIt’s a JDBC Flink based wrapper. Looks interesting for my use case.
2 Views