Hello everyone, I have a question regarding the wa...
# troubleshooting
c
Hello everyone, I have a question regarding the watermark "awareness" of sinks. I have a source that generates elements with monotonous event-time and therefore I use the built-in
WatermarkStrategy*.*forMonotonousTimestamps*()*
for watermark generation. Right after the sourceStream, the elements are sinked into Kafka. My expectation was that the sink would not emit the elements with the latest timestamp because the current watermark is
latestTimestamp - 1
, but the latest elements are indeed sinked to kafka before the watermark is emitted. Are sinks not aware of watermarks? Do I need to write a dummy ProcessFunction that holds these elements back? Thanks! Sorry, I could not find anything explaining this in the docs..
m
Watermarks don't have an effect on map steps and when you are passing data through as in your case. That's because the data is not held but processed and passed to the next operator. Watermarks are used to close time windows and synchronizing between multiple data streams.
You could try adding a 1 millisecond tumbling window that simply emits its elements to get your desired behavior. If you write your own process function be mindful that you need to manage the state while you hold the elements.