Hi folks. Could you recommend examples/presentations about implementing source (with new source api) which isn't FileSource-like (unlike flink-connector-hive for example). I want to implement source for specific case on top of MongoDB and try to ascertain what is the minimal set of interfaces/abstract classes to implement/extend without using any types from files connector, how to work with configuration, what should be in the configuration and what in the splits themselves etc. Currently my best guess is to look at pulsar/kafka connectors but if you have examples with smaller code surface it would be great.

Maybe you could refer to

org.apache.flink.api.connector.source.lib.NumberSequenceSource

, it's a data source that produces a sequence of numbers and simple enough.

I've read FLIP-27, it mentions that there're simplifications but not what they are. And NumberSequenceSource is trivial bounded source. My current questions touch things like: • how to handle mixed bounded and unbounded splits, • what should go into split/split state and what should go into enumerator state, • what combination of processed/in-flight/unassigned splits should enumerator state contain, • should enumerator limit number of emitted splits or should it assign all at once like

StaticFileSplitEnumerator

does, • etc So I'm looking for something more in-depth that trivial example and on par or a bit less complex that

PulsarSource

and

KafkaSource

especially with different enumeration (both generate splits by topic and partition) and split reading model (both have push based split readers).

Since you're specifically talking about doing this for Mongo, there's currently a FLIP being drafted to accept a MongoDB connector as Flink community. There is already a draft PR open too, see https://github.com/apache/flink/pull/20848