Max Kolchinsky
05/15/2025, 7:24 AMKeyedProcessFunction .
in my pipeline I define a keyBy followed by process with the function implementation and then attach sinks.
The function is pretty simple. I have a processElement that sends a timer via ctx.timerService().registerProcessingTimeTimer(timerTimestamp) and onTimer which collects a result.
Doing it naively like that results in a problem - I noticed that I keep crashing in CopyingChainingOperator.java when trying to run recordProcessor.accept(copy); in pushToOperator because the processed record no longer has a timestamp, as it was removed by KeyedProcessOperator here:
@Override
public void onProcessingTime(InternalTimerK, VoidNamespace timer) throws Exception {
collector.eraseTimestamp();
invokeUserFunction(TimeDomain.PROCESSING_TIME, timer);
}
And the exception I get is:
Cannot invoke "java.lang.Long.longValue()" because "p2" is null
in ExceptionInChainedOperatorException.java due to the timestamp on the processed record being erased.
Which I really don't understand - why are you removing the timestamp when I will need it later in chaining?
Solution:
I had to solve the problem myself, which I have done in the onTimer override by doing this hack
override fun onTimer(
timestamp: Long,
ctx: OnTimerContext,
out: Collector<AggregatedMessageJsonNode>
) {
(out as TimestampedCollector).setAbsoluteTimestamp(lastUpdated.value())
Unfortunately, after doing that, although I am no longer getting an ExceptionInChainedOperatorException , the pipeline still gets cancelled without any logs indicating a problem and the sinks are not executed, and both reading the logs and debugging is leading me nowhere.
Hoping someone here can stir me in the right direction.