Hey :wave: We saw a failure on flink 1.16 wherein ...
# troubleshooting
p
Hey 👋 We saw a failure on flink 1.16 wherein where a single NPE in one processing timer callback caused ALL timers across ALL keys to stop firing. Post one job restart timer wasn’t refired and application didn’t crashloop. Representative Code
Copy code
public class AggregateFunction extends KeyedProcessFunction<String, Transaction, Result> {
    
    private Map<String, Map<String, Double>> aggregatedValues = new HashMap<>();
    private ValueState<Long> nextTimerState;
    
    @Override
    public void open(Configuration parameters) {
        nextTimerState = getRuntimeContext().getState(
            new ValueStateDescriptor<>("timer-state", Types.LONG));
    }
    
    @Override
    public void processElement(Val x, Context ctx, Collector<Result> out) throws Exception {
        // Add data to aggregation
        String entity = x.getEntityId();
        String type = x.getType();
        aggregatedValues.computeIfAbsent(entity, k -> new HashMap<>())
                        .merge(type, x.getAmount(), Double::sum);
        
        // Schedule hourly timer if not already scheduled
        if (nextTimerState.value() == null) {
            long nextHour = computeNextHourTimestamp();
            ctx.timerService().registerProcessingTimeTimer(nextHour);
            nextTimerState.update(nextHour);
            LOG.info("Registered timer for {} at {}", ctx.getCurrentKey(), new Date(nextHour));
        }
    }
    
    @Override
    public void onTimer(long timestamp, OnTimerContext ctx, Collector<Result> out) throws Exception {
        LOG.info("Timer fired for key {} at {}", ctx.getCurrentKey(), new Date(timestamp));
        
        // This is where the NPE occurs - trying to merge data that doesn't exist
        try {
            String key = ctx.getCurrentKey();
            Map<String, Double> entityData = aggregatedValues.get(key);
            
            // NPE happens here when trying to access a missing key
            Double pendingValue = entityData.get("pending");  // This can be null!
            Double availableValue = entityData.get("available");
            
            // This will NPE if pendingValue exists but availableValue doesn't
            entityData.put("available", availableValue + pendingValue);  // NPE HERE!
            
            // Output result and schedule next timer
            out.collect(new Result(key, entityData));
            
            long nextTime = timestamp + 3600000; // Next hour
            ctx.timerService().registerProcessingTimeTimer(nextTime);
            nextTimerState.update(nextTime);
        } catch (Exception e) {
            LOG.error("Error in timer for {}: {}", ctx.getCurrentKey(), e.getMessage(), e);
            throw e; // This should cause crashloopin, but timers stop working
        }
    }
}
We had a bug wherein if one timer failed, we shouldnt set the next one but my expectation is the timer failures should cause the application to crashloop. Logs
Copy code
Logs Showing the Issue
[14:54:00] INFO - Registered timer for entity123 at Fri Apr 21 14:55:00 UTC 2023
[14:54:05] INFO - Registered timer for entity456 at Fri Apr 21 14:55:00 UTC 2023
[14:55:00] INFO - Timer fired for key entity123 at Fri Apr 21 14:55:00 UTC 2023
[14:55:00] ERROR - Error in timer for entity123: null
java.lang.NullPointerException
	at EntityAggregateFunction.onTimer(EntityAggregateFunction.java:44)
[...]
[14:55:01] INFO - Task failure, triggering recovery...
[14:55:05] INFO - Task restored from checkpoint

# No more timer logs appear after this, even though they should fire hourly
# Normal data processing continues for all keys

1
[14:54:00] INFO - Registered timer for entity123 at Fri Apr 21 14:55:00 UTC 2023
2
[14:54:05] INFO - Registered timer for entity456 at Fri Apr 21 14:55:00 UTC 2023
3
[14:55:00] INFO - Timer fired for key entity123 at Fri Apr 21 14:55:00 UTC 2023
4
[14:55:00] ERROR - org.apache.flink.runtime.taskmanager.AsynchronousException: Caught exception while processing timer.
	at org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1568)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1543)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1683)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1672)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
	at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352)
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: TimerException{java.lang.NullPointerException}
	... 15 more
5
Caused by: java.lang.NullPointerException
6
    at EntityAggregateFunction.onTimer(EntityAggregateFunction.java:44)
7
[...]
8
[14:55:01] INFO - Task failure, triggering recovery...
9
[14:55:05] INFO - Task restored from checkpoint
10
Questions for the Community 1. Has anyone encountered timers completely stopping without actually completing successful after an exception in the onTimer method? 2. Is there a known issue with timer state recovery after exceptions since we should have dropped a timer for a specific key but all timers for other keys were not fired.? Our application was using Flink 1.16.2 with checkpointing enabled (every 300 seconds). Full recovery required a complete job restart.
p
Hello there! Not specific to your issue. But I've encountered issues when Flink had too little CPU and it caused timers to stop. How is your CPU usage for this job?
p
It’s not really high.