Hi all, we're seeing an inconsistent behavior in F...
# troubleshooting
g
Hi all, we're seeing an inconsistent behavior in Flink committing Kafka offsets. Flink job lag shows a different number than what is committed to Kafka. Below are two screenshots that show lag seen by Flink vs Kafka. Has anyone faced this issue before? Our assumption: This inconsistency might be due to the fact that partition leaders might have moved to different brokers on MSK and Flink now has stale information about them.
As you can see, Flink lag metrics are almost negligible whereas the Kafka lags are in millions.