Hi. A simple question: if i activate continuous file monitoring from a s3 bucket and flink checkpoint, as the later keep trace of processed file, doess the state will keep growing and growing? Thx

The amount of state involved is small. IIRC, there’s one global timestamp plus an offset for each active split. However, performance on S3 is poor if the bucket being monitored has a lot of objects in it.

Thx David, yu mean it only store timestamp of Last processed file + files if multiples in same timestamp?

Whenever there are files to be read, the source splits these files into chunks called splits that are consumed in parallel. It keeps track of the progress of reading these splits in the state it checkpoints so it can avoid re-reading fully processed input during restarts.

Thx, i unserstand this, my question is what is been tracked? A timestamp for each Split? The list of already processed items? A kind of offset, in this case how is it working? Other?

The details seem to have changed with the recent rewrite, which introduced the CheckpointedPosition class. See https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/connector/file/src/util/CheckpointedPosition.html. I’m not totally clear on how this is used in practice.

Thx a lot, i will try to understand.