I've got a Flink job where data is read from a Kafka cluster, Filtered and Mapped, Keyed and then it goes into a Windower with a ProcessWindowFunction before the Sink. During a spike in traffic yesterday, we noticed that a couple of the Windowers were running behind and putting backpressure on the KafkaSources. We observed significant backlog in one or two partitions (the partitions key is a component of the keyBy our Flink job uses).
We solved the backpressure by increasing the MAX_PARTITON_FETCH_BYTES
value on our KafkaSource consumer configs. My question is, why did that perform so much better than the smaller batches, when the source could always just grab more when there was room in the buffers, in either configuration? We'd already scaled horizontally, so each Source was handling only one or two partitions, and most of the partitions had small backlogs.