Does anyone have suggestions on a pattern for dedu...
# troubleshooting
i
Does anyone have suggestions on a pattern for deduplicating messages using process time? We have an event stream that can sometimes become backed up. When it is we want to dedupe those events so we're only processing the latest ones. We were hoping to use Window Duplication but it currently does not support process time dedupe: https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/dev/table/sql/queries/window-deduplication/#limitation-on-time-attribute-of-order-key
p
Hello there, Ian! Do you want to dedup on key+processed time?
i
Yes! There's an entity id in the change event, and we just care to process the latest change event for that entity id.