Hi folks. I'm playing with Flink and having some t...
# troubleshooting
y
Hi folks. I'm playing with Flink and having some trouble -- I have my job workflow listening to a DDB stream, processing the CDC events, and writing some stats to another DDB table. My workflow looks like this:
Copy code
final var watermarkStrategy = WatermarkStrategy.<CDCEvent>forMonotonousTimestamps()
                .withIdleness(Duration.ofSeconds(1))
                .withTimestampAssigner((event, timestamp) -> event.getTimestamp());
        final var lateTag = new OutputTag<Metric>("late") {
        };

        final var workflow = env.fromSource(dynamoDbStreamsSource, watermarkStrategy, "DynamoDB Streams source")
                .returns(TypeInformation.of(CDCEvent.class))
                .uid("custom-uid")
                .filter(...)
                .map(...compute deltas based on changes...)
                .filter(...)
                .flatMap(...map deltas to 1 or more "Metrics"...)
                .keyBy(...key by the metric key...)
                .window(TumblingEventTimeWindows.of(Duration.ofSeconds(10)))
                .sideOutputLateData(lateTag)
                .aggregate(new MetricAggregator()); // this is an AggregateFunction<> that merges the delta values across metric instances in the window

        final var lateMetrics = workflow.getSideOutput(lateTag)
                .sinkTo(new PrintSink<>(true));

        workflow.sinkTo(...dynamo sink...);
As you can see, nothing fancy, pretty straightforward... My problem is: when I set the
InitialPosition
of the stream to
TRIM_HORIZON
, I see in the debug output that it's receiving, processing, and sinking all the historical events in the last 24 hours... but once it reaches the end, it is completely stuck. I make requests to the DDB table that result in new CDC events, I see in the output that it receives the event immediately, does all the stateless processing (all the filters and maps before the windowing), and then it just disappears and nothing happens. I have logging in the
MetricAggregator
and my custom sink, and neither logs get triggered for these new events. When I set the
InitialPosition
to
LATEST
and feed new events, it flows through fine! (I still have to submit a new change before the previous change gets fully processed, despite the idleness setting in the watermark strategy, but at least I can get it to process.) My custom sink is modeled closely after the existing DDB connector, and has these properties:
Copy code
maxBatchSize.       = 10
maxInFlightRequests = 20
maxBufferedRequests = 1000
maxTimeInBufferMS.  = 5000
Just adding this in case it's relevant but I don't think it should matter. Based on what I've read on watermarking, I assume this has to do with the strategy and specifically the idleness, which I set arbitrarily low, but it doesn't seem to matter/do anything (with either
TRIM_HORIZON
or
LATEST
). I've tried both built-in watermarking strategies and have the same problem. Any ideas? Thanks.
I understand now that idleness wouldn't affect my job at all because there's only one source. It doesn't answer my question about TRIM_HORIZON vs LATEST but at least I know now that idleness has nothing to do with this problem.
Figured it out -- I had a bug in my sink implementation that was not jiving well with default rate limiting strategy. Sink operator was in a deadlock.