I am trying to recreate: DataGeneratorSource in my test code: But Gradle cannot find these Flink files:

org.apache.flink.api.connector.source.lib.util.IteratorSourceReaderBase
org.apache.flink.api.connector.source.util.ratelimit.RateLimitedSourceReader

I can see them in the main Flink repo on GitHub, but I cannot find the right dependencies for my project. My Gradle dependencies:

flinkVersion = '1.16.0'
implementation "org.apache.flink:flink-streaming-java:${flinkVersion}"
implementation "org.apache.flink:flink-clients:${flinkVersion}"
implementation "org.apache.flink:flink-table-common:${flinkVersion}"
implementation "org.apache.flink:flink-table-api-java:${flinkVersion}"
implementation "org.apache.flink:flink-table-api-java-bridge:${flinkVersion}"
implementation "org.apache.flink:flink-table-planner_2.12:${flinkVersion}"
implementation "org.apache.flink:flink-connector-base:${flinkVersion}"
implementation "org.apache.flink:flink-connectors:${flinkVersion}"
implementation "org.apache.flink:flink-connector-test-utils:${flinkVersion}"
implementation "org.apache.flink:flink-core:${flinkVersion}"

RateLimitedSourceReader was only introduced in Flink 1.17 Maybe try using Flink 1.17?

Thanks @Hong Teoh! I wasted a long time on this. I am using PyFlink so I do not think that Flink 1.17 is an option?

Hm, it should still be an option, but 1.17 is not released yet 😅 Target release is some time March!

I am testing now. But I think that it is a little harder to use PyFlink. I am not sure how I would even do an install for PyFlink. I wanted to try to run with Flink 1.16.1-rc1 but could not figure what to do

Hello @Hong Teoh, I am having a similar needs, I need to set a rate limit on my KafkaSource operator, do you think I can use RateLimitedSourceReader to do this? and explain a little bit approach to do it? 🙏thanks

Hello, Kafka Source does not have this at the moment. You will need to implement a custom kafka source. However, you can probably just implement the KafkaSource and wrap the SourceReader with the RateLimitedSourceReader. https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka[…]n/java/org/apache/flink/connector/kafka/source/KafkaSource.java

ok, thanks!! I'll try it then