I'm looking for some job design insight. I have tw...
# troubleshooting
n
I'm looking for some job design insight. I have two process functions in my system. The first operates event data directly from devices (keyed by a device ID). That process function generates side outputs that are keyed by a rule identifier for further processing and feed into the other process function. I would like for the second process function to maintain an understanding of the latest event time seen for each device, but the current layout doesn't achieve that. Some ideas I had • Make the first process function generate a side output of something simple like
{device: <id>, time: <event time>}
, broadcast that side output stream and connect it to the stream that feeds the second process function. • Generate multiple events out of the first process function like
{rule: <id>, device: <id>, time: <event time>}
where all that changes is rule ID so that this side output can be connected to the other side output that is keyed by the same namespace. • Something else? The former seems easier but perhaps not the proper intent of broadcast state. The latter seems like a lot of extra traffic/noise in the system.
My thinking is that it will be best to go with broadcast state pattern for now since many partitions of the second process function will need to be informed of event times for a particular device. If the relationship were more unique, then I think it would make sense to go with the second option.