I wanted to ask a bit about state in flink. A bit ...
# troubleshooting
k
I wanted to ask a bit about state in flink. A bit of context first, I'm building a flink SQL pipeline to streams data from kafka topics to Apache doris. I have table level topics in kafka and in flink I'm performing joins using these topics, and then streaming those finally to my sink tables. I have set state backend type to rocks db. My understanding is that flink will be maintaining the current table state to be whatever events I have received from kafka on a table basis. If my task manager fails, do i lose that state? is there a way to persist my state indefinitely?
p
Hello there, @Karthik Raj! If you have HA configured on Flink and a persistent storage for your savepoints/checkpoints, your state will be persisted indefinitely.
k
@Pedro Mázala Thanks for your reply! I do have my savepoints and checkpoints pointed into an azure storage container. I have a question though - in the event that my jobmanager crashes, will the next taskmanagers to be spun up also pick up the state from my persisted storage?
p
If you're using HA (either zookepper or k8s) it will. Flink always store a "link" to the latest checkpoint/savepoint. If you delete the job manually tho, you will have to specify the path for it to start.