Hi all Our Flink Session cluster job manager is seeing a spi Apache Flink #troubleshooting

Hi all, Our Flink Session cluster job manager is ...

Guruguha Marur Sreenivasa

07/27/2023, 3:36 PM

Hi all, Our Flink Session cluster job manager is seeing a spike in heap memory usage for the past two weeks. I tried to update the memory limit to 3GB using the property:

jobmanager.memory.heap.size

I started getting this error when I try to apply the above memory config:

Copy code

IllegalConfigurationException: Derived JVM Overhead size (77.000mb (80740352 bytes)) is not in configured JVM Overhead range [1024.000mb (1073741824 bytes), 1024.000mb (1073741824 bytes)].

Anyone knows why the job manager could be using more memory or how to solve this?

Guruguha Marur Sreenivasa

07/27/2023, 3:56 PM

We have 20 jobs running in our cluster.

Puneet Duggal

07/27/2023, 6:06 PM

Can you add more details like flink version, state backend used and what kind of flink job have you deployed on flink cluster

Guruguha Marur Sreenivasa

07/27/2023, 6:06 PM

Flink version: 1.14.3 State backend: HashMapStateBackend Cluster type: session cluster Job type: Streaming pipeline processing events from Kafka

Guruguha Marur Sreenivasa

07/27/2023, 6:08 PM

I was able get around setting the job manager memory setting. But would like to understand in what scenarios the job manager would start using more and more memory. Like: 1. GC not triggering 2. Some stuck thread 3. Some blocking call

Guruguha Marur Sreenivasa

07/27/2023, 11:14 PM

Here's the heap memory usage we saw:

Open in Slack

Previous Next