About the variable checkpoint interval introduced ...
# troubleshooting
j
About the variable checkpoint interval introduced in Flink 1.19 (https://cwiki.apache.org/confluence/display/FLINK/FLIP-309%3A+Support+using+larger+checkpointing+interval+when+source+is+processing+backlog) How can I verify that this is effectively working? I restarted my job from scratch, so it is reading from a large Kafka topic where the oldest records have timestamps from months ago. The job may take a few hours to catch up with the latest kafka offsets. I configured checkpointing in the SQL job:
Copy code
SET 'execution.checkpointing.interval' = '3m';
SET 'execution.checkpointing.interval-during-backlog' = '10m';
However, all checkpoints are taken every 3m. Is this the expected behavior? I'd thought it would checkpoint every 10m in the first phase... I'm on Flink 2.0
l
when config kafka source properties, check your commited offset. Maybe you are reading from earliest offset
j
On second reading of the FLIP: _A Flink job is started to read from a Kafka topic starting at an offset corresponding to 5 days ago. There is no need to process records from 5 days ago with low latency. Therefore, Kafka source can allow user to specify a threshold (e.g. 5 minutes), and set its isProcessingbacklog to true iff the current_system_time - watermark > threshold._ So it appears I need to define a watermark on my source (which I did not) and I need to configure a threshold. Does anyone know where this is documented?