I am using Flink 2.1 but had the same problem with 1.20. We have fairly large snapshots 40-50 GB. Is it possible that savepoint at some point need to fit in TM memory?
During savepoint pod memory usage jumps by the around size of the snapshot (half of the size for each of 2 TMs). No Flink memory metric notices the memory usage, only Kubernetes pod memory usage metrics.