Yoshi Nagasaki
08/27/2025, 3:41 PMfinal var watermarkStrategy = WatermarkStrategy.<CDCEvent>forMonotonousTimestamps()
.withIdleness(Duration.ofSeconds(1))
.withTimestampAssigner((event, timestamp) -> event.getTimestamp());
final var lateTag = new OutputTag<Metric>("late") {
};
final var workflow = env.fromSource(dynamoDbStreamsSource, watermarkStrategy, "DynamoDB Streams source")
.returns(TypeInformation.of(CDCEvent.class))
.uid("custom-uid")
.filter(...)
.map(...compute deltas based on changes...)
.filter(...)
.flatMap(...map deltas to 1 or more "Metrics"...)
.keyBy(...key by the metric key...)
.window(TumblingEventTimeWindows.of(Duration.ofSeconds(10)))
.sideOutputLateData(lateTag)
.aggregate(new MetricAggregator()); // this is an AggregateFunction<> that merges the delta values across metric instances in the window
final var lateMetrics = workflow.getSideOutput(lateTag)
.sinkTo(new PrintSink<>(true));
workflow.sinkTo(...dynamo sink...);
As you can see, nothing fancy, pretty straightforward...
My problem is: when I set the InitialPosition of the stream to TRIM_HORIZON, I see in the debug output that it's receiving, processing, and sinking all the historical events in the last 24 hours... but once it reaches the end, it is completely stuck. I make requests to the DDB table that result in new CDC events, I see in the output that it receives the event immediately, does all the stateless processing (all the filters and maps before the windowing), and then it just disappears and nothing happens. I have logging in the MetricAggregator and my custom sink, and neither logs get triggered for these new events. When I set the InitialPosition to LATEST and feed new events, it flows through fine! (I still have to submit a new change before the previous change gets fully processed, despite the idleness setting in the watermark strategy, but at least I can get it to process.)
My custom sink is modeled closely after the existing DDB connector, and has these properties:
maxBatchSize. = 10
maxInFlightRequests = 20
maxBufferedRequests = 1000
maxTimeInBufferMS. = 5000
Just adding this in case it's relevant but I don't think it should matter.
Based on what I've read on watermarking, I assume this has to do with the strategy and specifically the idleness, which I set arbitrarily low, but it doesn't seem to matter/do anything (with either TRIM_HORIZON or LATEST). I've tried both built-in watermarking strategies and have the same problem.
Any ideas? Thanks.Yoshi Nagasaki
08/27/2025, 5:55 PMYoshi Nagasaki
08/27/2025, 7:58 PM