Yuliya Bogdan
06/02/2023, 12:32 PMFileSystem
connector to read avro files from a path which may contain non avro files that have to be filtered out (no partitioning), what I assume is a common case. Is there an easy way to do it?
I was looking for a way to pass custom file filter predicate to the default NonSplittingRecursiveEnumerator
, or specify FileEnumrator
provider to the FileSource
without the need to implement custom connector, but it seems to be tricky because FileSource.FileSourceBuilder
, which has a setter for the FileEnumerator, is initiated in private method in FileSystemTableSource.Yuliya Bogdan
06/02/2023, 10:33 PM