I am trying to debug or find reasons for checkpoint latency, has anyone observed 1 min or above checkpoint latency ?

Of course, it depends on many things. For example, if a sink implements exactly-once delivery it typically needs to finalize a batch of data during a checkpoint. So Flink may need to wait for the data store to confirm something. Or maybe it takes a while for the object storage that you use for storing checkpoints to acknowledge the write. Or you may have a data skew and it takes a while for the busiest tasks to report their progress.

Or the object storage (e.g., S3) may be subject to rate limits.

@David Anderson Checkpoint is once in 5 min and hardly I see crossing the limits. Also saw checkpoint starting delay.