I am using the ContinuousFileMonitoring function to stream files coming into GCS bucket. I set a monitoring interval of 30s. My job streams 7-10k files per hour, and I see a few files missing to be streamed for some hours. On adding some debug logs, I find that it skips the file here :https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/ContinuousFileMonitoringFunction.java#L362 It happens that the file is just added during the scan interval, and the timestamp of the file is lesser than the lastModificationTime in the next scan. Example log:

2023-08-29 23:08:16,934 INFO  org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction [] - Ignoring <gs://scribe-incoming-data/scribe-journal/endpoint=device/date_key=2023-08-29/hour=22/ZO56HQM_scribe-78c_events-2023-08-29-22-26.log.gz>, with mod time= 1693350431378 and global mod time= 1693350431409

Here it can be seen that the last Modification time and the file modification time has a diff of 31 ms, so it is likely happening during this edge case. Has anyone encountered the same, and any way to workaround this problem?