Hello, I am looking for information on processing...
# troubleshooting
s
Hello, I am looking for information on processing order in a FileSource. Specifically, I'd like to know if a FileSource maintains the order in processing lines (and splits). Here's what I mean: I have an S3 FileSource, and let's say I have only one file in the source bucket. It is a TextInputFormat:
Copy code
Line1
Line2
Line3
...
If my job graph is structured with a parallelism of 1, can I assume that the stream of text records will be in the same order that they appear in the file?
Copy code
S3 FileSource (parallelism=1) ---> map (parallelism=1) ---> sink (parallelism=1)
In this case, can I assume that Line1 will be processed before Line2, which will be processed before Line3 ? Thank you.
d
Yes, when you set the parallelism to 1 for both the FileSource and subsequent operators in your Flink job, the processing order will match the order of lines in the file.
with parallelism at 1, you ensure that the entire file is read by a single task.
The only caveat to that is if there is failure and retries it’s theoretically possible to introduce some change in order of processing if the recovery is not setup properly
s
Thank you! I was trying to find the source/implementation of the S3 FileSource to confirm if the splits are processed in the expected order (i.e., split 0 is lines [0, n], split 1 is lines [n+1, m], etc., and that split 0 is processed before split 1), but couldn't find it. Anyway, thanks again.