Hello team, i am looking at the existing connector framework code and it seems the reader is single threaded (Refs: https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/[…]ctor/base/source/reader/fetcher/SingleThreadFetcherManager.java, https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/[…]r/base/source/reader/SingleThreadMultiplexSourceReaderBase.java). In FLIP-27 i see there was a proposal to create Multi-split multi-threaded readers, but it seems its not implemented yet. For context, the DDB and Kinesis connectors follow a multi-split multiplexed approach where a single thread processes from multiple streams. I would rather want to move the DDB and Kinesis connector to a multi-split multi threaded model, is there anything blocking the connector framework to implement the same? Thanks in advance

It sounds reasonable to me what you are asking for. I suggest that you raise this on the dev list; to check there still support for this in the community. This might mean you may need to take over the Flip and its implementation, unless someone else volunteers. Some benchmarking by adding to https://github.com/apache/flink-benchmarks could be really useful to understand and justify this change by measuring the performance benefits.

sure, i can take over creation of the FLIP if community agrees

@Abhi Gupta These are the conversations to have on the dev list, so you know you are supported along the way. Maybe try to get some commitment from the community around binding votes which you need 3 of to get the Flip accepted.