Hello, What’s the most effective approach to imple...
# troubleshooting
a
Hello, What’s the most effective approach to implement micro batching (i.e., grouping a set of events into a time window) for writing to a single folder at the destination while enhancing fault tolerance? Should this be done using windowing and triggers, or through checkpointing process time?
d
The filesystem sink can do this based on checkpoint interval, look at the rolling policy https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/filesystem/#file-sink
But it really depends on the details of your micro-batches as to what is best.
a
In my case, I'm using the keyBy operator for grouping and the microbatch ID (window start time) when writing to the output location. Is it possible to schedule the checkpoint to occur at the end of the window? Specifically, can the window and checkpoint messages be arranged to happen sequentially, with the window message first and then the checkpoint message?
d
What does the window represent here? If it's just chunking by time, can the checkpoint interval be your "window"?
a
I'm using windows for microbatching (not Spark's microbatch buffering), which involves tagging all events in a window chunk with a specific timestamp. I can use the checkpoint interval as the window if the checkpoint time is accessible and can be attached to the events. I also use the microbatch timestamp for metrics, aggregations, and specific destination paths. Based on filesink implementation, i tried to write psuedo code for the both options: 1. Window + trigger implementation: https://github.com/adasari/mastering-flink/blob/main/app/src/main/java/org/example/customsink/App.java#L97 https://github.com/adasari/mastering-flink/blob/main/app/src/main/java/org/example/customsink/CustomStatefulSinkWriter.java#L109 2. Checkpointing process time: https://github.com/adasari/mastering-flink/blob/main/app/src/main/java/org/example/customsink/CustomStatefulSinkWriter.java#L173 (The implementation is still incomplete)
d
Ok, if you are doing other things it is best to keep it separate. I am not sure this will work, but you could look at implementing a custom BucketAssigner /RollingPolicy for the sink.
👍 1
Then rely on your windows for the batching
a
but you could look at implementing a custom BucketAssigner /RollingPolicy for the sink.
Thats the plan. just used map of bufferwritters in my pseudo implementation.
Is it possible to schedule the checkpoint to occur at the end of the window?
d
No
👍 1
a
last qq: is it possible to get window start/end events listener in all components of the pipeline post source like process time callback ? looks like it is not.
d
No, but you can include them in the output of the window. That is what I was envisioning when I suggested a custom BucketAssigner. You pick the file based on the record metadata then roll the file once complete.
a
visibility of window start time in the pipeline is useful to capture the window level execution time for pipelines like below. Source -> transform -> Two phase commit sink (sends additional window execution time metric along with data and other metrics)
@Danny Cranmer i tried to implement processing time callback similar to FileSink and it was never invoked. https://github.com/adasari/mastering-flink/blob/main/app/src/main/java/org/example/customsink/CustomStatefulSinkWriter.java#L173 Could you please assist me in finding the issue here ? Thanks in advance. In meantime, I will try to find the docs if any.
nvm. i was able to fix it. thanks.