Jonas Birgander
06/05/2024, 1:50 PMAggregateFunction UDF that combines arrays of strings. The way it’s implemented is by the accumulator having a HashMap<String, Integer> to keep track of the strings. For accumulate, it increments the counter, in retract it decrements it, and if the counter reaches 0, removes the entry. For this to work it has to get every change exactly once.
I’m seeing an issue where the result (in Elasticsearch) fluctuates even nothing should change. The Elasticsearch sink does fail quite regularly, with “Connection reset by peer”.
It’s set up with checkpointing every 10 minutes, and it’s okay to reprocess 10 minutes worth of data - if the accumulator gets each update exactly once.
I’ve been trying to reproduce the problem locally with no success, but it keeps happening in production.
I have some questions that I hope someone here can help me with.
• Are `AggregateFunction`’s accumulators (with data) serialized in checkpoints?
• When restoring from a checkpoint, is there a guarantee that it will get every change exactly once?
• Does changing execution.checkpointing.mode from AT_LEAST_ONCE to EXACTLY_ONCE change that?
A related question, but more specific to Elasticsearch
• How does the Elasticsearch connector handle errors in bulk operations? Retries are enabled. If a bulk of 1000 operation has 2 failing operations, are only those to requeued, or the whole bulk?
Any guidance is very much appreciated!