Tudor Plugaru
04/23/2025, 2:42 PMmax.poll.records=100
fetch.min.bytes=1
fetch.max.wait.ms=5
fetch.max.bytes=8388608
max.partition.fetch.bytes=2097152
and on the producer side:
batch.size=16384
max.request.size=2097152
buffer.memory=33554432
linger.ms=0
Job parallelism is 4 and I already have 1 consumer per partition.
So far, this worked it I managed to drop the latency, but looking at the consumer lag, there are still records in the topic... My goal would be to have as few as possible 😅 Any advice is very much welcomed! ThanksPedro Mázala
04/23/2025, 2:54 PMdisableChaining if the backpressure is happening not in the reading.
Something else I'd also check is the amount of operators you have in your pipeline. Each operator will have to serialize and deserialize the record to forward it.Tudor Plugaru
04/23/2025, 3:13 PMWhich deserializer are you using? POJO or Kryo?Pojo
Does your parallelism applies to everything?Yes, I don't use
disableChainingPedro Mázala
04/23/2025, 4:24 PMTudor Plugaru
04/24/2025, 7:49 AMTudor Plugaru
04/24/2025, 7:49 AMPedro Mázala
04/24/2025, 10:28 AMTudor Plugaru
04/24/2025, 11:50 AMTudor Plugaru
04/25/2025, 7:03 AMsourceIdleTime metric, AVG is ~200ms 😕 so it's like it to have this one lower, I know for sure that there are records in the topicPedro Mázala
04/25/2025, 8:51 AMTudor Plugaru
05/05/2025, 12:21 PMPedro Mázala
05/05/2025, 1:12 PMTudor Plugaru
05/05/2025, 1:39 PMDo the e2e show where more time is spent?It's basically a metric per sink -> operator latency. On our end, we have 4.
Tudor Plugaru
05/05/2025, 1:39 PM