I have SQL table contains houses (each row contain...
# troubleshooting
u
I have SQL table contains houses (each row contain house and his location). I need to do a geography join on that table - join each house with the near houses around him by thier locaiton. I want to do it in Flink batch, but I dont know how to do it in datastream api, becuase I need all the table for the join. Can someone help me with that? thanks!
p
If you key them by some sort of location key, all the data would be available for you. Is this what you want?
u
Yes I can keyby them by geography key, but for the join i need the entire data, and i am working with datastream api, there is a way that i can know if my data is finished (the stream is bounded)? Maybe a “close” function?
The challenge that I am facing is how to know when the stream is finished in datastream api (flink batch and bounded source). I thought maybe use the 'close' or 'cancel' function in process function, or maybe set event time timer for LONG.MAX_VALUE when the watermarks will finish. Thanks!
p
Ah I had understood it wrongly first then. So you want to: • read all available data in a bounded context • Process all the data at once on the end of the bounded data Is it correct? IIRC, Flink produces a high watermark value (
LONG.MAX_VALUE
) when it gets to the end of a bounded context. So I think you wouldnt need to do that manually. Would setting a timer to get trigged at this watermark make the trick for you?