Hey folks - hoping someone can shed light on a weird MSK lag behavior I'm seeing.
Setup: Managed Flink on AWS, KafkaSource, 256 partitions, parallelism 16, checkpoints every 1 min (~2-3s duration, no failures, ~700MB state).
The problem: Flink's records_lag_max is basically flat at 200-1000 β app is caught up and happy. The only spikes are brief jumps to ~20K when windows fire, which is expected. But MSK's SumOffsetLag and MaxOffsetLag show a sawtooth pattern: lag grows linearly for ~50 minutes (up to ~13M sum / ~60K max), then drops abruptly to near zero. Repeats endlessly.
This makes no sense to me. If offsets are committed on checkpoint (every minute), why would broker-side lag accumulate for 50 minutes before resetting? The 50-min cycle doesn't correspond to anything in my app config.
Has anyone seen this? Is there something about how the Flink Kafka connector commits offsets that could explain this? Or is MSK's lag metric calculation just weird/delayed?
Any pointers appreciated π