Hey everyone. I have really strange problems tryin...
# troubleshooting
m
Hey everyone. I have really strange problems trying to build an operator with sinks using
KeyedProcessFunction
. in my pipeline I define a
keyBy
followed by
process
with the function implementation and then attach sinks. The function is pretty simple. I have a
processElement
that sends a timer via
ctx.timerService().registerProcessingTimeTimer(timerTimestamp)
and
onTimer
which collects a result. Doing it naively like that results in a problem - I noticed that I keep crashing in
CopyingChainingOperator.java
when trying to run
recordProcessor.accept(copy);
in
pushToOperator
because the processed record no longer has a timestamp, as it was removed by
KeyedProcessOperator
here:
Copy code
@Override
    public void onProcessingTime(InternalTimerK, VoidNamespace timer) throws Exception {
        collector.eraseTimestamp();
        invokeUserFunction(TimeDomain.PROCESSING_TIME, timer);
    }
And the exception I get is:
Copy code
Cannot invoke "java.lang.Long.longValue()" because "p2" is null
in
ExceptionInChainedOperatorException.java
due to the timestamp on the processed record being erased. Which I really don't understand - why are you removing the timestamp when I will need it later in chaining? Solution: I had to solve the problem myself, which I have done in the
onTimer
override by doing this hack
Copy code
override fun onTimer(
        timestamp: Long,
        ctx: OnTimerContext,
        out: Collector<AggregatedMessageJsonNode>
    ) {
        (out as TimestampedCollector).setAbsoluteTimestamp(lastUpdated.value())
Unfortunately, after doing that, although I am no longer getting an
ExceptionInChainedOperatorException
, the pipeline still gets cancelled without any logs indicating a problem and the sinks are not executed, and both reading the logs and debugging is leading me nowhere. Hoping someone here can stir me in the right direction.