What data types can I use in incremental window ag...
# random
k
What data types can I use in incremental window aggregation for safe checkpointing? Hi! I want to do incremental window aggregation using
.aggregate(new MyAggregateFunction, new MyProcessWindowFunction)
as described in documentation. I'm wondering what Java/Scala types I can safely use as part of the incremental state ("accumulator"). I would like to, for example, keep a set of seen IDs in Scala Set. Everything seems to work fine, but I'm wondering if also checkpoint and creating savepoints will work as expected. Will Flink know how to serialize such a data structure for checkpointing and savepointing? Thanks for any insights 🙏 EDIT: The comment by David Anderson in this SO question says that "_an AggregateFunction where the accumulator is a Set tends to be problematic_". Why is it problematic? If I use Java HashMap, is that problematic?