Hello, has anyone figured out to use streaming file sink with batch mode execution. I am building on-demand pipeline processing but some scenarios could result in large amount of data to be processed. Running in streaming mode has challenges of cost.
m
Martijn Visser
03/22/2023, 2:52 PM
You can't
Martijn Visser
03/22/2023, 2:52 PM
Streaming File Sink only supports streaming
Martijn Visser
03/22/2023, 2:52 PM
You should use FileSink; that supports both batch and streaming