Zackary Naas
05/30/2025, 2:57 PMZackary Naas
06/02/2025, 3:05 PMbuilder.put("properties.max.partition.fetch.bytes", "20971520");
We are using Debezium to process binlogs from MySQL into Kafka and then doing some initial transformations to put those into a new topic, and in both of those upstream clusters/topics, we have the max message size turned up to 20MB to handle large binlog messages from some of the tables that can contain free-form text blobs or JSON data in one or more of their columns.
Without turning this up for the Upsert-Kafka Source Consumer to match, we found that the Flink job would reach an offset where it couldn't process the message and would get stuck.
I think this may be a bug, but I'm not sure. @rmetzger have y'all seen this before?
The Kafka docs seem to imply that even if the max.partition.fetch.bytes is not turned up to match the max message size, that it should not prevent the consumer from making progress, but in reality that is occurring for us in this particular case. Turning up max.partition.fetch.bytes to 20MB eliminates this problem.rmetzger
06/03/2025, 12:42 PMbut the source operator will show that it sent 170M records through to the next stepI'm not sure, but maybe watermarks are tracked as well. Are you using watermarks, how frequently are they emitted, and for how long was the source stuck?
rmetzger
06/03/2025, 12:52 PMZackary Naas
06/03/2025, 2:31 PMI'm not sure, but maybe watermarks are tracked as well. Are you using watermarks, how frequently are they emitted, and for how long was the source stuck?we aren't using watermarks for this particular application, but I think something inside of the consumer or the operator itself is incrementing even though no records are being consumed. I can verify in the consumer group within Kafka that the consumer is stuck in a certain offset in each of the two partitions. This may be a bug within MSK express - we just switched to using AWS MSK Express and have had some issues show up. If we get more details from AWS on this I'll circle back here to close the loop. Thanks Robert!
Zackary Naas
06/03/2025, 2:36 PM