Hello, I was wondering what are the opinions of ad...
# troubleshooting
c
Hello, I was wondering what are the opinions of adding a rate limiter to Flink? I see there are some implementations for it like: https://nightlies.apache.org/flink/flink-docs-release-1.9/api/java/org/apache/flink/api/common/io/ratelimiting/FlinkConnectorRateLimiter.html Is this considered an anti pattern though? My understanding is streaming applications should be free flowing. I've heard that having Flink output it's work to a Kafka and having a worker pull from the Kafka where the worker is rate limiting isn't great either cause I'm just shifting the work and making the system more complicated
d
It’s not necessarily an anti-pattern. The objectives of a flink system are not just to be free flowing but also integrate well with downstream systems so there are reasons to use rate limiting sometimes.
For example if a database has limitations on number of rates per second this has to be considered.
So rate limiting does the limiting with Flink and that can be simpler than doing it at the Sync level or Kafka for example because you introduce some complexity.
You have to look at issues like back pressure, latency requirements etc and run some experiments. If downstream systems cannot process without some form of rate limiting then you either rate limit within Flink or you do it at the Sync via something like Kafka.
You could do rate limiting at a downstream component like Kafka. Thats an option.
The factors to consider are volatility of the rate limits, impact on end-to-end latency, backpressure on your system. So consider those in terms of whether rate limiting at 1) source, 2) within Flink or 3) downstream Kafka consumer or similar or some combination best meets the requirements of your system.
c
This makes sense. Thank you so much