Does Flink support any kind of micro-batching in D...
# troubleshooting
v
Does Flink support any kind of micro-batching in Datastream API? We’d like to reduce the number of times state is modified in a windowing operation (it updates state on every incoming message). Ideally we’d batch the incoming updates for each window (by count and time), and update the window in batches that way. However it doesn’t look like there is an easy way to do that without either updating state on every message anyway, or risking data loss. Has anyone tried implementing micro-batching in Datastream?
d
It sounds like you are looking for the equivalent to the Table API's mini-batch aggregation for the DataStream API. I'm not aware of any way to do that, but I suppose you could convert from datastreams to tables.
v
i’m wondering if there’s any way to aggregate in memory while keeping the consistency guarantees and fault tolerance
only way I could come up with is pre-aggregate by appending to list state and then using that, but that still requires modifying the state for every record
d
That would probably be worse.