Is it possible to specify an order for Flink to re...
# troubleshootingt
Tony Wang
02/21/2023, 10:10 PMIs it possible to specify an order for Flink to read through a directory of Parquet files?
s
sap1ens
02/21/2023, 10:35 PMNot out of the box. I had to implement a custom
FileSplitAssignerFileSourcet
Tony Wang
02/21/2023, 10:36 PMSo this basically means it's impossible to use in the SQL API? But have you noticed a particular order? eg. does it always read files in alphabetical order
Tony Wang
02/21/2023, 10:37 PMAlso do you know the default behavior of how different workers go through a directory of files?
s
sap1ens
02/21/2023, 10:39 PMIt’s not impossible, it’s just unordered. Not all use-cases need ordering 🙂
t
Tony Wang
02/21/2023, 10:40 PMif you're running your flink job on historical data, to reproduce results you probably need ordering right
s
sap1ens
02/21/2023, 10:40 PMAgain, not in all use-cases.
✅ 1
sap1ens
02/21/2023, 10:40 PMThis is the logic for the default `FileSplitAssigner`: https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files[…]nk/connector/file/src/assigners/LocalityAwareSplitAssigner.java
t
Tony Wang
02/21/2023, 10:41 PMDo you know if this is the Assigner for files in S3/Hive too?
s
sap1ens
02/21/2023, 10:42 PMShould be for anything that uses FileSource
9 Views