Throttling a streaming Flink job: how can I do that? I want to be able to limit how fast a job runs to avoid starving other consumers on the kafka streams. How can I achieve this? I’m thinking by limiting the output so that the job build back-pressure and slows down its ingest, but even that I am not sure how to achieve. Any insight from the experts?

Why would you want to do that? If you're keeping up, the fetches will be as small as possible, more or less. If you're not, you need big fetches to catch up.

@Jeremy DeGroot i want to do that because the job is not the only consumer, and if it is catching up becausd it was stopped and restarted later, it might starve other consumers.

It's not Flink-specific, but limiting the amount of data your Kafka consumer pulls at once through Consumer property settings will probably give you the results you want. https://www.linkedin.com/pulse/throttling-kafka-consumption-dispelling-myths-ganesh-ghag/

I've seen this but the 'myth' is that you can throttle, which is not quite true. You can limit req size and nb of records, but that only means the consumer (if not throttled) will make more requests and eventually be limited, but it will still starve other consumers.

I mean, the goal is just not to be a noisy neighbor right? Letting it wait longer between fetches, and controlling how much it will fetch at once, limits the effect the fast consumer's fetches have on the broker.

Both nb of requests and bandwidth are limitations in my streaming service.

Is reducing parallelism an option?

I see... You mentioned wait between fetches but kafka consumer has a max wait, does it have a min wait too? I don't see that. Reducing parallelism to a point where it performs well in real time and throttles while in catch up mode might work, but i'm afraid it might still be an issue. I need to test.

https://kafka.apache.org/documentation/#consumerconfigs_fetch.min.bytes It has min bytes it'll wait for So if you set max bytes high enough, that could work

yes.

That is tricky, if you're worried about making sure they both keep up.

i’m merging CDC streams and some are like 1 event an hour while others may be 100k events per hour

Any pseudo-throttling you do is likely to block the slow stream, unless you can have two different consumers and join the streams.

and indeed it is tricky to keep those in sync but for the most problematic use case they don’t need to be.

https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/joining/ That's easy to do in Flink, fortunately

kinda 😉 easy if your entities are in the same window… a bit more complex when they are not.

Easier than it would be without the framework.

Throttling a streaming Flink job: how can I do that?

You can't. The only thing I could imagine is that you write your own sink which intentionally becomes a bottleneck and therefore causing backpressure in operators prior to that, but that would risk running into failing checkpoints and other unexpected behaviour.