Join Slack
Powered by
Does anyone have suggestions on a pattern for dedu...
# troubleshooting
i
Ian Stewart
08/22/2025, 6:43 PM
Does anyone have suggestions on a pattern for deduplicating messages using process time? We have an event stream that can sometimes become backed up. When it is we want to dedupe those events so we're only processing the latest ones. We were hoping to use Window Duplication but it currently does not support process time dedupe:
https://nightlies.apache.org/flink/flink-docs-release-2.1/docs/dev/table/sql/queries/window-deduplication/#limitation-on-time-attribute-of-order-key
p
Pedro Mázala
09/02/2025, 9:24 AM
Hello there, Ian! Do you want to dedup on key+processed time?
i
Ian Stewart
09/11/2025, 11:34 PM
Yes! There's an entity id in the change event, and we just care to process the latest change event for that entity id.
Open in Slack
Previous
Next