Hi Team! Has anyone ever tried making a File Sink ...
# troubleshooting
d
Hi Team! Has anyone ever tried making a File Sink that doesn’t need to wait until a checkpoint to release files to S3? Is there a particular reason the FileSink doesn’t actually honor
at_least_once
semantics?
For context, I am referring to this comment here.
IMPORTANT: Checkpointing needs to be enabled when using the
FileSink
in
STREAMING
mode. Part files can only be finalized on successful checkpoints. If checkpointing is disabled, part files will forever stay in the
in-progress
or the
pending
state, and cannot be safely read by downstream systems.
Local testing seems to confirm this.