Hi All, am hoping someone can help me, with the flink filesystem connector (for the table api), how / can we customise reading the partitions? And if we read the files in batch mode, do we get any order guarantees? My use case, I need to read a bunch of S3 files, where the path is /YYYY/MM/DD/HH/[file], and a bunch of files in each path thereafter. I need to read these files in order and output them to a stream for reprocessing, in the original order. Can I customise the partition timestamp extractor to read my (slightly different) path structure, and sort by that? Or is there a more efficient better way to do the above?

As far as I am aware (but this is based on knowledge from DataStream) it won't guarantee an order of read files because

ai.dema.flink.connector.file.src.assigners.LocalityAwareSplitAssigner

is providing next splits using a

HashSet

. Maybe someone has better solution but if not you could implement your own

FileSplitAssigner

and provide it to the soruce using

setSplitAssigner

that would give splits in order. Also please be aware that this would work only with parallelism of one.

Thanks for the advice, I'll take a look