Hello guys, I am writing a kafka consumer flink job, our flink and kafka clusters are in different geo location and thats why network lag is slowing down our throughput, I am exploring the idea of concurrent kafka poll per partition per task manager a with different offset range to increase the throughput. I am planning to write custom RichParallelSourceFunction for kafks source and use multithreading in run(). I'd appreciate some suggestions or recommendations.

What kind of Kafka config changes have you tried? If your network hops are expensive, I'd try raising

fetch.min.bytes

,

fetch.max.bytes

,

max.poll.records

to start. It's also worth raising parallelism if there are enough Kafka Partitions for it.

I had tried increasing all three of them but somehow still not able to keep up with high volume and parallelism is same as number of topic partitions

Are you seeing back pressure?

I am not seeing much of back pressure on flink side, but kafka lag stays high because we are not able to poll data quick enough

Did you also try raising

max.partition.fetch.bytes

? Usually Kafka does really well pushing up your batch size when you run behind.

yes, I tried increasing that too but still not much of difference

I'd raise those max partition fetch bytes values both up absurdly high. If you're doing 1 consumer per partition, they should be the same. If you know how large your records are, calculate approximately how much memory you should need for 1 second of traffic, and set it to that.