Hello there! I've been studying about Flink for a ...
# troubleshooting
a
Hello there! I've been studying about Flink for a while, especially about watermarks. I think I've read all the docs, but I still have some questions. Can you help me? 1. I declare "WATERMARK FOR column AS column - 5 sec", where "column" stands for the column name with time semantics, and "column - 5 sec" stands for a watermark strategy. Why do we need to write the column name (after "FOR") separately? How will it be used and checked in the future? 2. Will it be checked if I write a streaming operation over a column without a watermark? 3. Are there any theoretical limitations to the watermark strategy (after "AS")? I've just found out that the strategy's output type should be a timestamp. Can I use something more complex here, e.g., "column + interval" or "column1 + column2" or "min(column, now())"? I know that those expressions are meaningless, but nevertheless
l
1. In Flink SQL,
FOR column
declares which column is the event-time attribute
. Flink will internally track watermarks against it. 2. Once declared, Flink uses the watermark for all event-time–based operations. If you try to use event-time operations on a column without a watermark, Flink will throw an error 3. The expression after
AS
must evaluate to a timestamp. Flink not allowed: arbitrary expressions like
column1 + column2
,
min(column, now())
, or anything that might break monotonicity.
r
@L P V answered it above, here is my attempt at the same in different words :)
Will it be checked if I write a streaming operation over a column without a watermark?
it'll be used as the table's watermark - so anything you do on the table that requires time semantics (time-based windowing, temporal joins, etc etc) will use it In theory you can define the watermark as a fixed value, illustrated here
a
Thank you for your answers! And special thanks to @rmoff for your blog, it helped me to understand watermarks better
🙌 1
flink 1
r
that's awesome, glad it helped 🙂