Reaching out here to ask if anyone knows about the ins-and-outs of running gradle inside a docker container in build CI. It's a topic that's been covered a dozen times around stack overflow and the forums but never seems to have a satisfying answer (either no responses, or "turn off parallel builds" and similar non-solutions, or the threads are super out of date). It seems that something weird happens with memory management when running in a docker environment. On local developer machines (macbook with 18gb memory) a gradle build on a large project with

-Xmx4g

setting on the gradle daemon tends to only use about 5gb max between the main daemon process and all child processes. Looking at a build scan it appears to rarely reach that amount and spends most of the build using around 2-4gb of memory. Running htop on the macbook while the build runs shows that the total used memory on the machine (including all other apps, browser, slack, etc) hovers stably around 5-7gb. Meanwhile if I run exactly the same build, with exactly the same gradle properties (specifying the same number of max workers) inside a docker image on the same macbook (docker configured with 10gb max memory), all of a sudden that build continues to eat more and more memory, eventually blowing past 8-9gb and eventually using all 10gb available resulting the daemon being killed in the background. I'm not sure what to make of it. It implies there's something going on either in the OS (ubuntu vs macOS) or in docker, or the JVM, or in gradle itself. I've tried dozens of different combinations of JVM options and flags, I'm running Java 21 for the build toolchain on top of a Java 23 installation so flags like

UseContainerSupport

should be on by default. I've toyed with ensuring that

MaxMetaspace

is set when overriding

org.gradle.jvmargs

, I've set Java options globally for all java instances to try to constrain memory of child processes, etc. None of it has seemed to have any effect.

I have seen this kind of behaviour in (Gitlab) CI with large sets of tests, and the answer was to configure

forkEvery

to a low value when running in CI.

I'll give that a go to see if it helps, but I'm also more concerned with the root cause. It makes no sense to me why Gradle would use so much more memory across a bare metal macbook and a docker image with the same parameters, nor why the main gradle build daemon seems to actively ignore its JVM args and consumes more than twice the maximum memory it has been configured with

You could use a tool such as

jconsole

to check what the actual memory usage and limits are from the JVM point of view. This will also tell you how close to the limit your build gets. Note that you have 3 different JVMs running, each of them with a different place to set memory limits: • the

JAVA_OPTS

environment variable for the launcher, but you should not need to set any limit there • the

org.gradle.jvmargs

property for the Gradle daemon • the

kotlin.daemon.jvmargs

property for the Kotlin daemon. I would recommend settings limits with

-Xmx

and similar rather than RAM %. The Kotlin daemon inherits some arguments from the Gradle daemon, but won't take into account the RAM already used by Gradle. If other JVM processes are launched by the build (e.g. to run tests) they may have their own arguments defined elsewhere in build scripts. Finally you have to leave some room (up to 50% in constrained environments such as yours) for OS/filesystem/other processes overhead, which will also be accounted as RAM usage by the container.

I've tried both Xmx and ram percentage, both appear to be ignored. The Kotlin Daemon is supposed to inherit its settings from the gradle jvm args, but I've also tried setting a dedicated value on it independently. I've also set global

JAVA_TOOL_OPTIONS

memory limits. All of the above appear to be straight up ignored in the case of the gradle daemon. There's definitely more to JVM memory footprint than just the heap, but with the heap constrained to 2gb and metaspace constrained to 1gb, I certainly wouldn't expect it to be using a total of 6gb of memory, insinuating that if the heap and metaspace are properly constrained then there's 3gb of other stuff being allocated

Depending on how large the build is I would not be surprised by a +100% (i.e. 50% of total RAM) overhead. I also note that on your screenshots processes are run through

rosetta

and that probably adds some significant overhead as well. Your try with 3g+1g means both JVMs could allocate up to 8 GiB together, so one of them ending up OOM-killed in such a situation is not really surprising either. Did you try

jconsole

to check what it looks like from within the JVM, and see if you could further restrict memory allocation?

I haven't been able to connect jconsole to the gradle daemon instance specifically. As for the combined usage, what I'm actually seeing is that the Kotlin Daemon and other instances are using pretty limited memory and the Gradle Daemon itself is the one that is growing to 200% of its limit