Hey guys, I was wondering if anyone has some advice on creating a more backend stateful Flink application. Our team has a kinesis stream that has metrics that our Flink application processes to see if there are any issues with devices. We need to add more context however. To get this information it will likely be read from an SQS consumer. However, I'm not sure how we should be saving this data in our Flink application. I was thinking of saving it in a dynamoDB table and/or cache however, I worry we process too much data for it. We get these metrics very frequently but only want to add the context to specific metric types. Do you guys have any suggestions on what is the best way to store this information. It would be great to not have to initialise this data manually every time we start the Flink application.

Flink is a stateful stream processor. It stores managed state in its state backend (most apps use the embedded rocksdb state backend), which is made durable via checkpoints and savepoints. The docs are here: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/fault-tolerance/state/ If you use SQL or the Table API to join your incoming metrics with this context info, Flink will use this managed state for you. If you use the DataStream API, you'll need to explicitly handle the state yourself.