Hello team 👋 I am having trouble identifying the source of a problem on a flink app that is failing to complete a checkpoint. It's almost always the 3rd checkpoint that fails and I don't really know what could be wrong, any ideas? app is running on kuberentes and handled by the flink operator checkpoint data is on GCS

2023-08-11 18:31:41,642 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 1 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1691778701615 for job 432e70a1c5fc763750e484d1002f5a07.
2023-08-11 18:31:42,878 INFO  org.apache.flink.fs.gs.GSFileSystem                          [] - Creating GSRecoverableWriter with file-system options GSFileSystemOptions{writerTemporaryBucketName=Optional.empty, writerChunkSize=Optional.empty}
2023-08-11 18:31:43,471 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 1 for job 432e70a1c5fc763750e484d1002f5a07 (30329 bytes, checkpointDuration=1677 ms, finalizationTime=179 ms).
2023-08-11 18:31:43,474 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 1 as completed for source Source: flink-ticket-views-es-sink-source.
2023-08-11 18:31:46,493 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 2 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1691778706472 for job 432e70a1c5fc763750e484d1002f5a07.
2023-08-11 18:31:46,651 INFO  org.apache.flink.fs.gs.GSFileSystem                          [] - Creating GSRecoverableWriter with file-system options GSFileSystemOptions{writerTemporaryBucketName=Optional.empty, writerChunkSize=Optional.empty}
2023-08-11 18:31:47,224 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Completed checkpoint 2 for job 432e70a1c5fc763750e484d1002f5a07 (30329 bytes, checkpointDuration=464 ms, finalizationTime=288 ms).
2023-08-11 18:31:47,225 INFO  org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Marking checkpoint 2 as completed for source Source: flink-ticket-views-es-sink-source.
Aug 11, 2023 6:31:47 PM com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem repairImplicitDirectory
INFO: Successfully repaired '<gs://flink-ticket-views-us-east1-2607/checkpoints/432e70a1c5fc763750e484d1002f5a07/>' directory.
2023-08-11 18:31:47,906 INFO  org.apache.flink.metrics.datadog.DatadogHttpReporter         [] - The metric flink.jobmanager.job.lastCheckpointExternalPath will not be reported because only number types are supported by this reporter.
2023-08-11 18:31:50,242 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 3 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1691778710224 for job 432e70a1c5fc763750e484d1002f5a07.
2023-08-11 18:36:28,471 WARN  org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Could not fulfill resource requirements of job 432e70a1c5fc763750e484d1002f5a07. Free slots: 0
2023-08-11 18:36:39,332 INFO  org.apache.kafka.clients.NetworkClient                       [] - [AdminClient clientId=flink-ticket-views-es-sink-enumerator-admin-client] Node -1 disconnected.
2023-08-11 18:36:50,242 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Checkpoint 3 of job 432e70a1c5fc763750e484d1002f5a07 expired before completing.
2023-08-11 18:36:50,244 WARN  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to trigger or complete checkpoint 3 for job 432e70a1c5fc763750e484d1002f5a07. (0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2165) [flink-dist-1.16.1.jar:1.16.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
2023-08-11 18:36:50,271 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 4 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1691779010254 for job 432e70a1c5fc763750e484d1002f5a07.