👋 I have a long shot question for y’all. As an alternative to stateful functions, I was thinking of using SideOutputs in the Data Stream API as a means to dynamically dispatch “messages” to/from different Process Functions. Are there any drawbacks to this approach? (eg. does it impact checkpointing?)

I wouldn’t see any impact to checkpointing — why SideOutputs and not filter messages you want to send to different processfunctions?

Hrmm my thought was to have 1 side output per function which is unioned with the “main” stream. The main steam is 5 process functions chained together. With the side outputs we can dynamically dispatch messages to each to a specific process function from a process function. Something along the lines of:

var alertFunction =
			eventRouterFunction
			.union(enrichedEventStream.getSideOutput(alertInput))
			.keyBy(alertInput -> alertInput.alertId)
			.process(new AlertFunction())
			.name("AlertFunction");

Either works, the filter would effectively do the same thing, just allow you to create one stream for this alert, one for your 2nd process function, etc, etc.

Make sense! Another issue that comes to mind if we don’t go the stateful functions route is hydrating state from an external data source. I was planning on leveraging Flink State more like a cache (with a long ttl) since the data we need to action on could be around forever. If the function state were empty, it would attempt to fetch it from an external DB (DynamoDB) before applying the rest of the logic. I’m aware that doing external calls in a

map

in the DataStream API would lock up the TaskSlot resulting in slowing down the entire job, and the AsyncIO API is intended to address this. Though I can’t really see how I could apply the AsyncIO API for this “caching” kind of use case since it doesn’t have any state. It’s almost like I’d like to have a

AsyncKeyedProcessFunction

that works off of Futures but that doesn’t appear to be a thing (and I get why that’d be challenging with how checkpointing is implemented). Are you aware of any patterns/libraries (other than stateful functions) that allow for this kind of “Flink State as a Cache” pattern (or keyed async io)?

you can certainly pre-hydrate your state using the State Processor API and restoring from a savepoint that already has state in it--this is not possible on Kinesis Data Analytics for Apache Flink unfortunately, but you’d basically download your existing state (or create new state) using the State Processor API and restore or start your application using that state. Additionally, check out this blog showcasing enrichment patterns in Flink--think it’ll resonate with you. https://aws.amazon.com/blogs/big-data/common-streaming-data-enrichment-patterns-in-amazon-kinesis-data-analytics-for-apache-flink/

Whoa, that’s a great resource! Surprisingly didn’t show up while googling the problem! There’s alot here to digest, going to need some time to see if we can make one of these fit. Thanks for sharing!