Hello community, is there a way how to effectively perform look up join for two streaming queries without watermarking? I am using Table API which tries to enrich data of main stream with value from another stream. Issue I have is that records on the right/ build side might be years old so I can’t really use event time temporal joins with watermarking, processing time temporal join are not currently supported so look up joins are of topic too and regular joins doesn’t seem to fit, as it joins all previous and future records. Any ideas for some workaround?

Hey, have you checked https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/joins/#lookup-join?

I did check, but it uses processing time temporal join, which is not supported if i understood correctly from jira issue (besides error thrown by flink) https://issues.apache.org/jira/browse/FLINK-19830 I don’t like much non-temporal join as there are records coming to both sides and each one triggers join with all future and past records, it just doesn’t make sense to use, but i have nothing better

You might consider doing an ingestion-time temporal join, where you supply a watermark strategy that uses ingestion-time-based timestamps, and then do an event-time temporal join.

Could you please elaborate a bit more on that 2 join setup? My sources are kafka topics on which i originally tried an event-time temporal join on kafka rowtime (ingestion time) for both topics, like this

source_time TIMESTAMP(3) METADATA FROM 'timestamp',
            WATERMARK FOR source_time AS source_time - INTERVAL '10' SECOND

but one of them might be a year old, so finally it didn’t work out as records come independently to both of them.