Hi All! I’ve seen a few anecdotes suggesting using the

ContinuousFileReaderOperator

is discouraged in favour of the new

FileSource

. I’m wondering if there is any risk of the

ContinuousFileReaderOperator

being deprecated in future versions. This is a critical operator for my team. We use this as a source in the middle of our pipeline because the true data source depends on other upstream events (user actions).

Yes, the ContinousFileReaderOperator will be deprecated since all functionality should already be available in the new FileSource

since all functionality should already be available in the new FileSource

Perhaps I need to look at this a little deeper. Our use case is rather unique (I assume). The files we read from S3 have unpredictable formats. Our streams true source is a Kafka message that contains an S3 path and metadata about the files format. We have custom file splits that carry this metadata to the reader so it can read and parse the file. (custom

InputFormat

too) If an equivalent operator in the new File Source exists then all is good.

That does not exist in the new FileSource, but it's also not something that's on the roadmap at this moment

So long as it’s not removed, then there is no concern. I might have to look into how we can do the equivalent in the new FileSource.

So long as it’s not removed, then there is no concern.

Given that we want to deprecate the old sources and sink interfaces, that could very well be removed at some point in the future

I might have to look into how we can do the equivalent in the new FileSource.

That would be great. I have seen other file for similar requests, but then it needs someone to drive the implementation for this. If you could help out, that would be quite neat.