Hi! we are using Flink 1.19.2, and we are seeing a...
# troubleshooting
z
Hi! we are using Flink 1.19.2, and we are seeing an interesting issue. Hoping someone may have experience with something similar: • Kafka sources, consuming data from multiple topics in the same upstream kafka cluster • joining those sources together in Flink using regular inner and left outer joins (not temporal or anything else - we need all state from all time for each upstream source) • Using 64 parallelism, 8 slots We are seeing that occasionally one of the consumer operators, which is reading from the kafka topic, seems to be having some sort of bug related to infinitely consuming and seemingly never actually progressing. The upstream kafka topic has a total of 16M records, but the source operator will show that it sent 170M records through to the next step. There's no logic in this operator, as it is reading from the source and selecting columns and that is it. We are using avro-debezium and upsert-kafka for this source. Most of our other jobs like this just work, but there are a handful that get stuck in this state, and they basically show a static value for pending records and for remaining kafka records via metrics reporting, meaning it looks like they are never making progress. Any ideas? Anyone faced this before?? Thanks y'all!
Wanted to follow up on this, in case anyone finds themselves in a similar situation. We were able to resolve this by adding an additional property to our Upsert-Kafka Sources:
builder.put("properties.max.partition.fetch.bytes", "20971520");
We are using Debezium to process binlogs from MySQL into Kafka and then doing some initial transformations to put those into a new topic, and in both of those upstream clusters/topics, we have the max message size turned up to 20MB to handle large binlog messages from some of the tables that can contain free-form text blobs or JSON data in one or more of their columns. Without turning this up for the Upsert-Kafka Source Consumer to match, we found that the Flink job would reach an offset where it couldn't process the message and would get stuck. I think this may be a bug, but I'm not sure. @rmetzger have y'all seen this before? The Kafka docs seem to imply that even if the
max.partition.fetch.bytes
is not turned up to match the max message size, that it should not prevent the consumer from making progress, but in reality that is occurring for us in this particular case. Turning up
max.partition.fetch.bytes
to 20MB eliminates this problem.
r
Thanks for posting the solution to the problem.
but the source operator will show that it sent 170M records through to the next step
I'm not sure, but maybe watermarks are tracked as well. Are you using watermarks, how frequently are they emitted, and for how long was the source stuck?
z
thanks for the link, reading now 👀
I'm not sure, but maybe watermarks are tracked as well. Are you using watermarks, how frequently are they emitted, and for how long was the source stuck?
we aren't using watermarks for this particular application, but I think something inside of the consumer or the operator itself is incrementing even though no records are being consumed. I can verify in the consumer group within Kafka that the consumer is stuck in a certain offset in each of the two partitions. This may be a bug within MSK express - we just switched to using AWS MSK Express and have had some issues show up. If we get more details from AWS on this I'll circle back here to close the loop. Thanks Robert!
Interesting link - this is indeed the behavior we seem to be experiencing, however this only became an actual issue upon the switch to MSK Express instead of standard MSK. Here is where Confluent/Kafka states that the consumer _should_ be able to progress even if this value is set to low. I think for now we can assume this is an AWS and/or Kafka-side issue (as we only saw this issue upon switching to MSK Express) and not try to file any sort of issue for Flink. Thanks again!