Pratyush Sharma
05/20/2025, 4:29 PMpublic class AggregateFunction extends KeyedProcessFunction<String, Transaction, Result> {
private Map<String, Map<String, Double>> aggregatedValues = new HashMap<>();
private ValueState<Long> nextTimerState;
@Override
public void open(Configuration parameters) {
nextTimerState = getRuntimeContext().getState(
new ValueStateDescriptor<>("timer-state", Types.LONG));
}
@Override
public void processElement(Val x, Context ctx, Collector<Result> out) throws Exception {
// Add data to aggregation
String entity = x.getEntityId();
String type = x.getType();
aggregatedValues.computeIfAbsent(entity, k -> new HashMap<>())
.merge(type, x.getAmount(), Double::sum);
// Schedule hourly timer if not already scheduled
if (nextTimerState.value() == null) {
long nextHour = computeNextHourTimestamp();
ctx.timerService().registerProcessingTimeTimer(nextHour);
nextTimerState.update(nextHour);
LOG.info("Registered timer for {} at {}", ctx.getCurrentKey(), new Date(nextHour));
}
}
@Override
public void onTimer(long timestamp, OnTimerContext ctx, Collector<Result> out) throws Exception {
LOG.info("Timer fired for key {} at {}", ctx.getCurrentKey(), new Date(timestamp));
// This is where the NPE occurs - trying to merge data that doesn't exist
try {
String key = ctx.getCurrentKey();
Map<String, Double> entityData = aggregatedValues.get(key);
// NPE happens here when trying to access a missing key
Double pendingValue = entityData.get("pending"); // This can be null!
Double availableValue = entityData.get("available");
// This will NPE if pendingValue exists but availableValue doesn't
entityData.put("available", availableValue + pendingValue); // NPE HERE!
// Output result and schedule next timer
out.collect(new Result(key, entityData));
long nextTime = timestamp + 3600000; // Next hour
ctx.timerService().registerProcessingTimeTimer(nextTime);
nextTimerState.update(nextTime);
} catch (Exception e) {
LOG.error("Error in timer for {}: {}", ctx.getCurrentKey(), e.getMessage(), e);
throw e; // This should cause crashloopin, but timers stop working
}
}
}
We had a bug wherein if one timer failed, we shouldnt set the next one but my expectation is the timer failures should cause the application to crashloop.
Logs
Logs Showing the Issue
[14:54:00] INFO - Registered timer for entity123 at Fri Apr 21 14:55:00 UTC 2023
[14:54:05] INFO - Registered timer for entity456 at Fri Apr 21 14:55:00 UTC 2023
[14:55:00] INFO - Timer fired for key entity123 at Fri Apr 21 14:55:00 UTC 2023
[14:55:00] ERROR - Error in timer for entity123: null
java.lang.NullPointerException
at EntityAggregateFunction.onTimer(EntityAggregateFunction.java:44)
[...]
[14:55:01] INFO - Task failure, triggering recovery...
[14:55:05] INFO - Task restored from checkpoint
# No more timer logs appear after this, even though they should fire hourly
# Normal data processing continues for all keys
1
[14:54:00] INFO - Registered timer for entity123 at Fri Apr 21 14:55:00 UTC 2023
2
[14:54:05] INFO - Registered timer for entity456 at Fri Apr 21 14:55:00 UTC 2023
3
[14:55:00] INFO - Timer fired for key entity123 at Fri Apr 21 14:55:00 UTC 2023
4
[14:55:00] ERROR - org.apache.flink.runtime.taskmanager.AsynchronousException: Caught exception while processing timer.
at org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1568)
at org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1543)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1683)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: TimerException{java.lang.NullPointerException}
... 15 more
5
Caused by: java.lang.NullPointerException
6
at EntityAggregateFunction.onTimer(EntityAggregateFunction.java:44)
7
[...]
8
[14:55:01] INFO - Task failure, triggering recovery...
9
[14:55:05] INFO - Task restored from checkpoint
10
Questions for the Community
1. Has anyone encountered timers completely stopping without actually completing successful after an exception in the onTimer method?
2. Is there a known issue with timer state recovery after exceptions since we should have dropped a timer for a specific key but all timers for other keys were not fired.?
Our application was using Flink 1.16.2 with checkpointing enabled (every 300 seconds). Full recovery required a complete job restart.Pedro Mázala
05/26/2025, 8:22 AMPratyush Sharma
05/26/2025, 9:35 PM