Is there a way to expose flink application state v...
# troubleshooting
d
Is there a way to expose flink application state via an HTTP endpoint? follow-up: would that be crazy?
j
There is this https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/fault-tolerance/queryable_state/ But it was removed. You could periodically write savepoints or maybe externalized checkpoints and then you could parse them using this https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/libs/state_processor_api/
🤔 1
☝️ 1
d
hmm... interesting options. Not sure those hit the mark for what I'm hoping though
thanks for the ideas!
j
I wish also that there was a better way to do this. Flink state is effectively a distributed key value store. I wouldn’t even need other databases for some of our services if we could just query it directly.
r
I am also interested in what options exist for exposing Flink state (also perhaps modify the state?). My colleague's current idea is for us to feed an incoming event to our stream processing that when seen will log what the state is for the key we're interested in. Parsing checkpoints is an interesting option, thanks for the link!
p
There is the State Processor API, which could help you to get there. There's also some other tools that achieve something similar. Do you want to serve this state to other applications? Or is this something as sanity check?
d
Do you want to serve this state to other applications? Or is this something as sanity check?
"Serve to other applications" Just thinking it would be nice to expose state directly from flink, instead of populating some other service/cache via a kafka topic (or similar) What I've seen of the state processor API and queryable state, neither meets my needs without some backflips - so I'll probably just stand up a separate service and hydrate the state via kafka.
p
I also thought about it once in one of the applications I build. But later on I thought a push data model would be better than a pull model. So then I served it somewhere else to be fetch with a side output.
so I'll probably just stand up a separate service and hydrate the state via kafka.
I think this is the path indeed. Another con for me later was exposing flink internals to an service that would not need to know such thing.
❤️ 1