Hi everyone, i have an scenario that i want in flink but i dont sure flink have any solution for it i want consume a topic that contains rules, and join it with our products, and change price based on that rules and return product. but i have an issue that when we run flink for first time, we not sure that we get all rule before get any product, and maybe some product come faster than they rules and we dont know how manage it is there any way to sure we get all message that in rules topic then start processing product stream and join it with rules

Hey there @MohammadReza Shahmorady! A way to sort this one could be either using a lookup; or a postpone until you have the right info. Timers may help you. LMK if you need help understanding it.

Thank you @Pedro Mázala I tried a few ways to handle it. For example, while running, I would get the last offset of the topic and check in the consumer whether I had reached that offset or not. If I did, I would set a flag to true. Then, in another stream, I would check if the flag was false and dump the messages into a list. But you know, these methods didn’t seem right to me, and I was wondering if Flink itself has a built-in solution for this. I think this is a common problem, and I’m not sure how to write this scenario in a clean and principled way. Some ideas, like loading all roles in the

open

method from a database, also came to mind, but it still doesn’t seem like the right approach. I’m looking for a proper solution.

if Flink itself has a built-in solution for this.

A temporal/lookup join would sort this IMO. Instead of the offset approach, I'd try something related to watermark and timers to try again.

I think this is a common problem

For sure. I've seen quite some people around this topic.

@Pedro Mázala can you explain about temporal/lookup join more? I cant understand Or any document for that

I think this session may be able to help you https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#interval-joins

@Pedro Mázala thanks a lot