Dennis Sosnoski
07/24/2025, 9:22 AMListState on checkpoints (with parallelism always 1, to avoid any repartitioning of the data). This works fine as long as the number of records is small, but when we grow the number of records the checkpoints seem to work but we're unable to restore from them - the job gets stuck in INITIALIZING state forever.
We're using Amazon Managed Flink 1.20, which is hard-wired to use the RocksDB state backend. The record count where we start getting the problem is around 1200, with the checkpointed data size about 1 MB. There's no noticeable delay restoring state up until we have the problem, then the job just gets stuck. The FlameGraph shows solid network activity when we're stuck, reading from S3.
Has anyone else experienced this kind of issue with Amazon Managed Flick, or have any feedback on this in general?Pedro Mázala
07/24/2025, 9:32 AMs3 s3a or something else?Dennis Sosnoski
07/24/2025, 11:50 PMinitializeState method in our operator has not been called.
Note that the CPU graph is mostly showing data from previous runs. The time of the restart was right around 2330.Dennis Sosnoski
07/25/2025, 3:55 AMinitializeState method. It took 1:45 (hours/minutes). Maybe I should have signed up for Prime?Dennis Sosnoski
07/28/2025, 8:19 PMPedro Mázala
07/29/2025, 9:08 AMDennis Sosnoski
07/30/2025, 9:35 PMVictor Babenko
08/14/2025, 8:06 AMDennis Sosnoski
08/27/2025, 9:02 AM