I'm joining two streams using the Datastream API a...
# troubleshooting
p
I'm joining two streams using the Datastream API and using .connect() and .flatMap() right now. Both streams are connected to a Kafka Topic. One Stream is actually a continous stream of data. The other is more of a lookup table. Is there a best practice on how to make sure, that lookup data is read completely before it starts joining data? I was using Flink SQL before with the Upsert-Kafka Connector, which kind of solved the problem for me, however I'm not sure, how to solve it with the Datastream API
p
Hey there! There are some techniques that you can do. Such as postpone events based on your state. Or even some lookups. I'm wondering how this problem was sorted by using the
upsert-kafka
connector.
p
Yea, that's pretty much what I'm doing right now - using a ValueState for the meta data and a ListState for the data when it arrives before metadata is available. This seems to work very well, I was just suprised, there isn't a ready-made construct for this
p
This is the feeling most people face 😄
😀 1