Hello all. I could really use your help understanding the

scan.watermark.idle-timeout

. I’m using the SQL Gateway to join two tables, backed by Kafka topics. One is a fact table that receives a constant stream of data. The other one is a compact topic populated by Debezium that contains dimension data and I’m joining the two to enrich the facts. I want to sink the results to an append-only Kafka table sink which is why I have made the dimension table versioned and use a Temporal join to get the snapshot of the dimensions at the specified fact timestamp. Because the dimension stream rarely gets any new data, the watermark will usually be hours if not days behind the fact stream watermark and this will stop the output watermark from progressing. I have used

scan.watermark.idle-timeout

on the dimension table to let the watermark generator know to ignore the dimension table watermark but I still see the watermark on the TemporalJoin operator as the minimum of the two. What am i I doing wrong? Sorry if my grasp of these concepts are somewhat shaky. I’d really appreciate the help. Thank you. Here’s some simplified code to demonstrate the setup:

That setup looks like it should work; perhaps the issue relates to how you are testing it. Is the fact_table being fed a steady stream of events? If so, I would expect the watermark at the join to start progressing once the dimension table has been idle for a minute. BTW, is it true that the fact table can be as much as 5 minutes out of order? That's an unusually large watermark delay. That's not a problem, except that it can make testing more difficult.

Thank you @David Anderson. I tested a bit more and you’re absolutely correct. It works well. The confusing thing for me was that the watermark on the join operator still shows the minimum of the two (ie: watermark of the dimension) but the watermark for the sink is correct and reflects the watermark of the facts. Everything is working beautifully. I have lowered the 5 min watermark to 2. As you pointed out, it was excessive.