Hi!
I have a potential use case for broadcast state in this flink app. And i'm trying to understand some of the nuances on how you handle initial start up of a flink app and non-determinism of when the main data stream vs broadcast stream reaches the broadcast process function.
The main keyed data stream has messages that contain an id (pointing to the entity this represents), the current state of the entity, and a data provider field. We key the stream on id, and the stream itself contains messages for a single id that might come from different data providers.
We have a config stream that contains the a config item like: id, provider to use fields. and we have a processor that will accept the data stream and config stream (as broadcast state) and depending on the config for that message's id, we'll either discard or publish the message through.
one piece i'm struggling to figure out is how to handle the scenario where in the broadcast processor, how do i handle the scenario where the config stream takes significantly longer to start consuming and publishing to the broadcast processor, what do i do with the data stream elements? I don't want to drop messages, and I also don't want to just buffer messages indefinitely in a list. Ideally, i'd love it if i was able to introduce some type of semantics like:
• apply backpressure to the data stream / don't process or consume the data stream source until the broadcast stream reaches the processor
I can't find resources online for this, so i'm wondering if i'm just thinking about this problem incorrectly