Flink 1.19.x Cluster and Job Issue: not consuming ...
# troubleshooting
s
Flink 1.19.x Cluster and Job Issue: not consuming from all Kafka Topic Partitions with Checkpoint/Savepoint We're experiencing an issue with our Flink jobs (versions 1.19.0 and 1.19.1) where, upon deployment with checkpointing and savepoints enabled, the jobs do not consume from all partitions of a Kafka topic. Despite the Flink documentation not listing the Kafka connector for these versions, we found the connector version 3.2.0-1.19 available on Maven. Has anyone encountered similar issues with Kafka topic partition consumption in Flink 1.19.x? Any insights or solutions would be greatly appreciated.
d
Check that Flink job’s Kafka consumer is configured to correctly handle offsets, especially if it’s a new group or if the topic’s partition count has changed. You can set the
Copy code
auto.offset.reset #latest or earliest depending on your needs.
Check for parallelism mismatch. there can be common issue is a mismatch between the number of Kafka topic partitions and the parallelism of your Flink source. Flink will assign each task to consume from a subset of the partitions. Make sure the number of parallel source tasks matches the number of Kafka topic partitions.
Check fetch.min.bytes, max.poll.records, and session.timeout.ms etc. Kafka consumer settings can affect this as well
Although you may have found a compatible connector don’t count on this. You need to carefully read release notes etc to be sure.
s
we have set, offset, parallelism and kafka related config accordingly. it's working fine with flink 1.18 but not working with flink 1.19
d
are you using the latest 1.19? what exact version are you using?
You should check issues and release notes for 1.19 version and see if there are any relevant issues.
Also the Kafka connect 3.2.0-1.19 you are using read the notes carefully
Enable debug for flink and kafka
If possible try to do minimal configuration that just reads from kafka to isolate the issue.