Apache Flink

Hi <#C03G7LJTS2G>, I am running a kafka to iceberg writer for events data and see a consistent consumer lag, but from querying kafka I can see that the data is actually there, so flink is actually delaying commiting offset to kafka. I can see a correlation between checkpoint interval and consumer lag. Is there a way to make it zero ?  or is this expected behaviour ?

Hey, this is an expected behaviour - kafka connector does not use offsets from consumer group during runtime, and only commits them on each checkpoint.

&gt; Kafka source commits the current consuming offset when checkpoints are *completed*, for ensuring the consistency between Flink’s checkpoint state and committed offsets on Kafka brokers.
&gt; 
&gt; Note that Kafka source does *NOT* rely on committed offsets for fault tolerance. Committing offset is only for exposing the progress of consumer and consuming group for monitoring.
<https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/connectors/datastream/kafka/#consumer-offset-committing|nightlies.apache.org/flink/…/kafka#…>

Hi <@U03JS3Q15TK>, I have also noticed some missing records in the same pipleine, 900 messages which came in around 2 seconds time period. I have no idea how to trouble shoot this. Kindly help :sweat_smile: