Is it true that Flink does not support file chunking using InputSplit for CSV files. So a file is processed using only one reader. After chunking a file into smaller sizes, Flink was able to parallelize the reading step and the performance increased significantly. While processing a 7.2GB file with one reader takes over 2 minutes, chunking the file in advance and then processing them took 40 secs without fine grained tuning. Multiple reader processing is promising. We probably need to add support parallel file reading using InputSplits for CSV files. Any inputs how do we solve this?

Splittable enumerator is used by Flink if the file format supports splitting: https://github.com/apache/flink/blob/eff9b16799f162f31058be5acac567943a446dea/flin[…]c/main/java/org/apache/flink/connector/file/src/FileSource.java I guess it’s not the case for the default CSV format.

This would be a worthwhile improvement for Flink, but it needs someone who wants to volunteer to make a contribution on this 🙂