Hi team
We are using two streams - Non-keyed data...
# troubleshootingp
Preeti Nayak
10/19/2023, 7:29 PMHi team
We are using two streams - Non-keyed datastream for events and BroadcastState for rules. The events will be evaluated against the rules.
What happens when the pipeline starts and nothing has happened on the broadcast rule stream yet? Does the datastream halt till the rules reach the operator? Is there any bootstrap mechanism to set up some initial rules in the broadcast state?
On googling I came across - https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/libs/state_processor_api/#broadcast-state
This fits our requirement. But couldn't find pyflink support for it.
Any other suggestions?
a
Alex Nitavsky
10/19/2023, 8:19 PM1. For broadcast state you can use this API: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/fault-tolerance/broadcast_state/#broadcastprocess[…]stprocessfunction
And than you will be able to implement the logic, saying: do not output the data (probably buffer them) before you have broadcast state delivered.
If it is keyed state - do not forget to clean the state for a key for timer.
Alex Nitavsky
10/19/2023, 8:20 PM2. If broadcast state is not available you can: read it in driver function and than send it via closure (it should be serializable). Than those data should be persisted in the state. Also you can make a simple source, which would periodically read the data and broadcast them.