Hi all, Is there any formal way we could externali...
# troubleshooting
a
Hi all, Is there any formal way we could externalize flink state instead of using in-memory state storage? I watched the YT video which was posted on Flink Forward:

https://www.youtube.com/watch?v=ZWq_TzsXssM▾

p
Have you checked Flink HA and externalized checkpoints? You can use S3, GCS, file, or any other external file system to do so.
a
Yes. I've enabled S3 checkpointing, I'm dealing with massive data ingestion where windowing isn't feasible. To prevent state explosion due to the real-time changes in our substantial stateful data, I'm considering storing the state in an external source rather than in memory. This approach could potentially facilitate easier scaling of our solution. I also have identified lookup joins but since data is expected in less than 30 seconds. I am not able to use this as well. Could you please help me with any other suggestions to deal with this situation apart from scaling up the memory