:wave: I want to dedupe my stream and keep records...
# troubleshooting
m
πŸ‘‹ I want to dedupe my stream and keep records with maximum processedCount. Would the following be correct to use if I need to do event time processing afterwards?
Copy code
SELECT *
FROM (
  SELECT *,
    ROW_NUMBER() OVER (PARTITION BY id ORDER BY processedCount DESC) AS row_num
  FROM event)
WHERE row_num = 1
note
ORDER BY processedCount
. I know it does not translate to Deduplicate optimized plan because I am not using time attribute as the Flink doc suggests. My question is that whether the result preserve the watermark?
Originally, I was using a regular self-join but then came across this stackoverflow question
It is not possible for a regular join to have time attributes in its results, because the time attribute could not be well-defined.
I am wondering if my dedupe query above will have some kind of similar limitation.