Hello everyone, Following our recent upgrade from Flink 1.19 to 2.2, we have noticed a memory leak that seems to originate from FLIP-513 (Split-level Watermark Metrics). When consuming from Paimon/S3 (which frequently creates new splits, ~9,000 splits/hour in our case), Flink instantiates metric groups per split (

SourceSplitMetricGroup

), but these metric groups are never released after the split finishes. Over a few days, this fills up the heap, triggers heavy GC saturation, causes source starvation, and increases consumer lag until we restart the job. Our heap dump shows that 99.9% of the leaked metric groups contain the

'split'

key. Is there already a JIRA issue or a PR tracking this lifecycle cleanup bug for FLIP-513? If not, we can create a JIRA issue with our heap dump findings. Also, is there a recommended configuration workaround to completely disable/exclude these split-level watermark metrics in Flink 2.2 to prevent registration in the meantime? Thanks!

Hey ALX, I think the issue you are referring to was fixed on 2.3 github.com/apache/flink/pull/26388

Ok ! Thanks @Efrat Levitan!

Hi @Efrat Levitan, I'm the team member! Confirming your fix

onSplitFinished()

is at L195 of release-2.2.0, so we do have it. That's what makes this a

separate gap

rather than a missing backport: we run 2.2, that code is active, and we still retain

~301k

of these groups. The fix closes

splitWatermarkMetricGroup

— the inner

watermark

child — and that part works. What nothing does is detach either group from the parent: • the

<splitId>

group from

addGroup(SPLIT, splitId)

github.com/Efrat19/flink/blob/…/InternalSourceSplitMetricGroup.java#… is never closed at all. •

AbstractMetricGroup.close()

clears a group's own groups/metrics maps but never removes it from its

parent's

map — github.com/Efrat19/flink/blob/…/InternalSourceSplitMetricGroup.java#… Heap dump of one TaskManager, 1.4 days uptime, on 2.2:

GenericValueMetricGroup (<splitId>)  301,792  closed=false   ← never closed
    GenericMetricGroup ("watermark")     301,709  closed=true    ← closed, still attached
    → ~983 MB of a 1.67 GB live heap

Both still reachable via

<operator>.groups["split"].groups[<splitId>]

. And #26388 clearly does its job on the metrics: 301,591 of those watermark groups have

metrics.size == 0

, and only 208 TimerGauge are retained instead of ~600k. The metrics are deregistered;

the group objects aren't released

. So it looks like a follow-up to FLINK-37596 rather than a duplicate.

Hey, thanks, yeah I see the issue There's just no API today to deregister a metric group added through

addGroup

on the parent I'm looking into into possible solutions

Hi @Efrat Levitan, any new about it?, there are some teams that are doing rollbacks due this issue, and we would like to know if exist a quick fix. I think @Aakash Sandhyan has a pr ready to fix it.

Hey Alex, I'm discussing possible fixes with my team and plan to create one by EOD Alternatively happy to checkout your suggested fix

Thanks @Efrat Levitan!

Hi @Efrat Levitan, Please find this PR: github.com/apache/flink/pull/29089 Wanted to test this change out before sending it out.

Hey based on internal discussion I created github.com/apache/flink/pull/29083 , let me know if it aligns with your expectations

Yes, aligns.

Hi @Efrat Levitan, we assume that this is to create a new minor release version?, when do you think that it will be ready to use?

Yes, the fix will be backported to 2.3.x and 2.2.x

When can we expect this?

Hello, The backports are merged bp2.3 bp2.2