Hello! I have a Flink job reading from Kafka topi...
# troubleshooting
j
Hello! I have a Flink job reading from Kafka topics, with some intermediate tables, and in the end it writes to a couple of Elasticsearch sinks. In the logic, there is a lot of joining of tables and aggregations. One of the aggregations has an
AggregateFunction
UDF that combines arrays of strings. The way it’s implemented is by the accumulator having a
HashMap<String, Integer>
to keep track of the strings. For
accumulate
, it increments the counter, in
retract
it decrements it, and if the counter reaches 0, removes the entry. For this to work it has to get every change exactly once. I’m seeing an issue where the result (in Elasticsearch) fluctuates even nothing should change. The Elasticsearch sink does fail quite regularly, with “Connection reset by peer”. It’s set up with checkpointing every 10 minutes, and it’s okay to reprocess 10 minutes worth of data - if the accumulator gets each update exactly once. I’ve been trying to reproduce the problem locally with no success, but it keeps happening in production. I have some questions that I hope someone here can help me with. • Are `AggregateFunction`’s accumulators (with data) serialized in checkpoints? • When restoring from a checkpoint, is there a guarantee that it will get every change exactly once? • Does changing
execution.checkpointing.mode
from
AT_LEAST_ONCE
to
EXACTLY_ONCE
change that? A related question, but more specific to Elasticsearch • How does the Elasticsearch connector handle errors in bulk operations? Retries are enabled. If a bulk of 1000 operation has 2 failing operations, are only those to requeued, or the whole bulk? Any guidance is very much appreciated!