Hi all,
I have a question regarding MapState. We a...
# troubleshootingd
Daniel Suh
10/17/2023, 11:53 PMHi all,
I have a question regarding MapState. We are seeing ever growing checkpoint size in a flink application running in AWS KDA. While looking through the code, I had a hypothesis that MapState.remove() does not also actually clear the state, but rather just deallocates the space used by the MapState. This would mean that the checkpoint size would continue to grow, as the MapState would never get cleared.
Can anybody confirm that this is the case? Does a MapState need to have .clear() run on it specifically to allow the garbage collector to free up that space?
Daniel Suh
10/18/2023, 12:22 AMFor more context, we have a MapState which looks like key: timestamp, value: Listobject.
We create the state upon open(). Then find the closest last 5 min interval for incoming objects. Then we will update the state so it looks like
state[timestamp] = [object1, object2, object3]
We set a timer to go off 5 minutes later, which will count all objects collected for that timestamp.
Then we state.remove(timestamp)
Could this increase the checkpoint size over time, since the state never gets .clear()?
m
Martijn Visser
10/18/2023, 7:33 AMWhich state backend are you using?
d
Danny Cranmer
10/18/2023, 7:44 AMDefault would be RocksDB with incremental checkpoints enabled.
m
Martijn Visser
10/18/2023, 7:45 AMSo I think then the cleanup during RocksDB compaction is applicable, right? https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/fault-tolerance/state/#cleanup-during-rocksdb-compaction
d
Danny Cranmer
10/18/2023, 8:14 AMYes AFAIK. @Daniel Suh can you confirm if triggering a savepoint results in the checkpoint size dropping? Which Flink version are you using?
d
Daniel Suh
10/18/2023, 2:00 PM@Martijn Visser I’m not seeing a backend specified, so it looks like according to htis: https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/ it should be HashMapStateBackend
Daniel Suh
10/18/2023, 2:22 PM@Danny Cranmer we are using 1.8.3. Let me look into the savepoint dropping the checkpoint size
Daniel Suh
10/18/2023, 4:32 PMwhoops, according to this, yeah AWS defaults to RocksDB https://docs.aws.amazon.com/managed-flink/latest/java/reference-flink-settings.title.html#reference-defaults-state-backend