Hi! I have a potential use case for broadcast sta...
# troubleshooting
r
Hi! I have a potential use case for broadcast state in this flink app. And i'm trying to understand some of the nuances on how you handle initial start up of a flink app and non-determinism of when the main data stream vs broadcast stream reaches the broadcast process function. The main keyed data stream has messages that contain an id (pointing to the entity this represents), the current state of the entity, and a data provider field. We key the stream on id, and the stream itself contains messages for a single id that might come from different data providers. We have a config stream that contains the a config item like: id, provider to use fields. and we have a processor that will accept the data stream and config stream (as broadcast state) and depending on the config for that message's id, we'll either discard or publish the message through. one piece i'm struggling to figure out is how to handle the scenario where in the broadcast processor, how do i handle the scenario where the config stream takes significantly longer to start consuming and publishing to the broadcast processor, what do i do with the data stream elements? I don't want to drop messages, and I also don't want to just buffer messages indefinitely in a list. Ideally, i'd love it if i was able to introduce some type of semantics like: • apply backpressure to the data stream / don't process or consume the data stream source until the broadcast stream reaches the processor I can't find resources online for this, so i'm wondering if i'm just thinking about this problem incorrectly
i'm running flink 2.0
for those reading we're currently solving this in the short term by introducing a buffer + dropping oldest if we see buffer overfill by the time broadcast state comes in. long term, I think we're going to evaluate managing initialization of state via our operator's open callback which is a bit more work, but does get around the issue
ok, i think i found out how to solve my problem without buffering. i have two options that keep everything within flink primitives: 1. use the state migration utility that flink supports, it's not amazing, but def would solve the problem with preloading broadcast state for my operator before the app starts up. 2. support "transition" deployments, that do state migration but within the context of the running flink app, not as a separate migration job. when testing, if my operator uid is the same and the state descriptors are the same, flink doesn't have any issue with accessing that state even if my operator changes shape