We have a job where we buffer a number of records ...
# troubleshooting
d
We have a job where we buffer a number of records in an operator to re-order output based on the record data. The buffered records are pojos, and we save them as
ListState
on checkpoints (with parallelism always 1, to avoid any repartitioning of the data). This works fine as long as the number of records is small, but when we grow the number of records the checkpoints seem to work but we're unable to restore from them - the job gets stuck in INITIALIZING state forever. We're using Amazon Managed Flink 1.20, which is hard-wired to use the RocksDB state backend. The record count where we start getting the problem is around 1200, with the checkpointed data size about 1 MB. There's no noticeable delay restoring state up until we have the problem, then the job just gets stuck. The FlameGraph shows solid network activity when we're stuck, reading from S3. Has anyone else experienced this kind of issue with Amazon Managed Flick, or have any feedback on this in general?
p
How is your CPU usage on start time? I had a similar issue once. 1200 should not be a big amount for flink TBH. Are you sure it's stuck on the state restoring? Perhaps some debug logs may help understanding. Humm, what s3 protocol are you using?
s3
s3a
or something else?
d
Hi Pedro, thanks for responding. Yes, 1200 records with <1 MB checkpoint size seems very small. Here's some info from a run today with a larger buffer of up to 10K records - looks like it's using s3a, but that's all Amazon code we have no control over. CPU is low, busy time high. Logging shows that the
initializeState
method in our operator has not been called. Note that the CPU graph is mostly showing data from previous runs. The time of the restart was right around 2330.
I left this running and it did eventually restore the state and call the operator
initializeState
method. It took 1:45 (hours/minutes). Maybe I should have signed up for Prime?
@Pedro Mázala any suggestions?
p
Hum.. Is prime a AWS offering? TBH I think this is probably something related to how AWS is restoring the state. Have you tried contacting their support? It really looks like an infrastructure issue. Happy to discuss it further to help you
d
yes, we have an issue open with Amazon support. I agree that it seems likely to be something with their code. the "Prime" reference was meant as a joke, since Amazon Prime offers fast delivery for Amazon purchases. 😄
😅 1
v
@Dennis Sosnoski wondering if you heard anything back from AWS on this, is this an MSF bug?
d
AWS confirmed the bug in their own testing. They recommended using keyed state with a constant key value to try to get around this. I'll try the kludge soon to see if it works.