Building precompiled script plugins in a multi project setup uses a lot of memory We have a dedicated project for building convention plugins. This project has multiple subprojects (roughly 20). Some plugins are compiled with

groovy-gradle-plugin

and some are compiled with the

kotlin-dsl

plugin. The problem is that both these plugins together seem to use ~200 MB on average on each subproject. This memory seems not to be freed when the subproject is done. On a simple hello world project (with 20 subproject) I built https://github.com/Meijuh/precompiled-mem-issue, one can see that close to 4GB is necessary. The easiest way I found to measure the used memory is by running the build in a docker container (e.g.,

docker run -it --memory=12g --cpus="8" -v .:/workspace -v eclipse-temurin:21.0.8_9-jdk-jammy bash

) and then running

docker stats

. Running

gradle-profiler

won't immediately work here because of all the process forking that is going on. Note that 4GB of memory on a hello world example is problematic, because our real world case attaches external dependencies to the plugins etc, causing 12GB to be used there, which causes scheduling issues on our Jenkins cluster. I have also created a single project variant https://github.com/Meijuh/precompiled-mem-issue/tree/single-project, which does not cause the memory to blow up. And this looks like the intended approach as is hinted by the plural name of https://docs.gradle.org/current/javadoc/org/gradle/plugin/devel/tasks/ValidatePlugins.html. However, moving all the plugins to a single project won't work for our real world case because: • Some convention plugins depend on other convention plugins, making it hard to generate a dependency graph of plugins (which is easy in a multi-project setup). • Most convention plugins depend on external dependencies, such as Spotless. This means every plugin would require the same (unused) external dependencies, cluttering the classpath in the build environment of projects that apply the convention plugins. • The code just gets messy. I've tried disabling the

ValidatePlugins

task (as it forks uncontrollably), limiting the number of Gradle workers, compiling the KTS scripts "in-process", but none seems to really help. Any ideas how to continue? I can imagine I am missing a helpful Gradle config, or that I have reached a Gradle limitation and I could create a feature request to improve the memory usage for this use case?

Try not using precompiled script plugins?

I considered that, but I think that is more of a workaround. Using the

java-gradle-plugin

would cause quite a bit of code duplication that otherwise the

groovy-gradle-plugin

would take care of. Also if I read the documentation correctly, I would still need to run the

ValidatePlugins

task, which seems to cause memory problems as well.

You can use

org.jetbrains.kotlin.jvm

instead of

kotlin-dsl

and remove all duplication (but that requires writing all your plugins in Kotlin)

Yeah, I guess it is fair to say some further experimentation is in order.

There's an issue open about enabling

ValidatePlugins

by default: https://github.com/gradle/gradle/issues/22600 If you find out this is too resource intensive, please leave comment there

That issue is not about enabling

ValidatePlugins

by default, that is the case. It is about setting

enableStricterValidation

of the added

ValidatePlugins

task to

true

by default. 🙂

Well, in general it is hard for me to analyze what part causes the memory usage to spike and thus also I don't know how to solve it. Which means (@Martin) I can't know for sure if

ValidatePlugins

plays a role here, so I can't place that comment yet. I would like to have some more insight into why so much memory is used, but I do not know how. Any help in that area would be appreciated. @Vampire grouping my plugins on some criteria was also something I was thinking, and I might look into that as well. I'm afraid though not much grouping can be done. I'm glad you agree splitting up plugins into subprojects is a good idea. However, I don't think much is wrong by using

subprojects {...}

the way I do. Gradle projects form a tree structure. So if I organize my projects in such a way they form a hierarchy and that they inherit properties from their ancestors (such as which Gradle plugins to apply) then I don't think that is bad design. I'd be happy to learn why this would be bad design though 🙂. I'd like my solution to be DRY.

For DRY have another included build or

buildSrc

with the convention plugins for building your plugins and apply that convention plugin in your plugin build projects.

subprojects { ... }

,

allprojects { ... }

,

project(...) { ... }

and any other form of cross-project configuration is highly discouraged bad practice. Even just reaching into another project model to get non-immuntable information is almost as bad. All immediately introduce project coupling, which work against some more sophisticated Gradle features and optimizations. You for example will never be able to use the upcoming Isolated Projects, which will allow the projects to be configured in parallel as they are guaranteed to not be coupled, also making IDE syncs significantly faster, ... Also applying plugins using

apply

is discouraged and legacy and it is recommended to always use the

plugins { ... }

block or a convention plugin. I would also not use the

buildscript { ... }

block to add a plugin to the buildscript classpath, but the

plugins { ... }

block with

apply false

. If you would use Kotlin DSL in your build scripts - which I highly recommend - you would also missing the typsafe accessors by not using the

plugins { ... }

block to apply plugins.

Well, I can see that an included build could work to keep it DRY and might give it a try. But I don't see why I couldn't exploit the tree structure of projects in Gradle. If

subprojects

or

allprojects

is bad practice, then might as well remove the tree structure as a whole from Gradle and just have a sequence of projects without any hierarchy. Note that I agree reaching into another branch of a tree is bad practice. Reaching into subprojects should still allow for parallelism, technically there is nothing that would stop you. I realize that the

apply plugin

is legacy, and I think that is unnecessarily annoying.

It's legacy, not deprecated. I don't think it will go anywhere, as you for example use it also in non-buildscripts. But in buildscripts you should always prefer

plugins { ... }

. And well, no how can you process A and B in parallel if B is a child of A and A reaches into B. It is simply impossible to do.

That would be good news if it is not getting deprecated. What do you mean exactly with "process" then? As Gradle has three separate phases, initialization, configuration and execution. The

subprojects

and

allprojects

the way I use them would still allow for parallel execution, which is what I meant with "process". Initialization and configuration is quick anyway, so I don't see a need to parallelize that.

I already said "configured in parallel", not executed. Do whatever you like in your builds. I'm just telling you that it is a very bad idea and highly discouraged bad practice. :-)

Sorry, I missed that you already said that (it was not my intention to mute this discussion). If you don't mind, I really like to understand why this is considered bad practice (maybe I am missing some strong argumentation). The first thing I would like to understand is how included builds would be a DRY solution. In my hello-world example I would need to apply the precompiled plugin 20 times, right? Second, from what I can find online,

subprojects

and

allprojects

is considered bad practice because of two main reasons: 1. build.gradle(.kts) should be self contained for readability. 2. It prevents for instance parallel configuration. I'd like to counter this: 1. This same line of reasoning can be used to say Java is bad, because the entire source code needs to be split up in classes in separate files. Also, tree structures are pervasive in software engineering; a well designed build hierarchy should be easily understandable. Does Gradle discourage hierarchical build setups in general? 2. I do not need parallel configuration, but in case you want it for large code bases; I understand that build.gradle(.kts) files can have multiple instances of

subprojects

and

allprojects

per project, which makes it hard to parallelize configuration. What if you add

subprojectspar

and

allprojectspar

that can occur only once per project and can be used to configure children in parallel, either by parallel BFS, or parallel nested DFS, depending on the traversal order required. This would be more flexible than forcing users to have "isolated projects". If it would help, I can give an example hierarchical build setup, which I think makes sense for a practical case too.

it was not my intention to mute this discussion

Don't worry, the "do what you like" was not meant like that, but just what it said. I'm just a user like you, sharing my experience and recommendations, but what you take away is up to you, you have to live with the consequences. 🙂

If you don't mind, I really like to understand why this is considered bad practice

As I said, it introduces project coupling which works against more sophisticated Gradle features and optimizations like the upcoming isolated projects. It also is the Gradle folks who highly discourage cross-project configuration. It also is not only the project coupling, it also generally makes builds harder to maintain and understand. You might think it is easier to write it in that way, but code is much more read than written. You could also have empty build scripts or even delete the build scripts (they are optional) and do every configuration in the root buildscript through

subprojects { ... }

,

project(...) { ... ]

and so on. But imagine someone that does not know the build is looking at the build. To understand how a project is configured, he would need to read each and every piece of build logic as cross-project configuration might be done somewhere. Whereas if you have the configuration in the build script where it belongs, you only have to look at the build script and understand the configuration. To centralize build logic, you use convention plugins that you then use in the projects. Of course you could argue that you then have to look there for the configuration that is done, but at least it is crystal clear that you have to look there. You look at the build script, see the 5 convention plugins that are applied, and if their name is not meaningful enough / you don't know what they do, you know exactly where to look to understand the configuration of the project.

The first thing I would like to understand is how included builds would be a DRY solution. In my hello-world example I would need to apply the precompiled plugin 20 times, right?

Ok, yes, that one line needs to be repeated. But that is a price you should usually pay to have better maintainable and understandable builds. 🙂

I'd like to counter this:

You are getting a bit polemic now. And you are comparing apples to pears.

What if you add

I add nothing except if I fells so and contribute, I'm not affiliated to Gradle in any way, I'm just a user like you.

It prevents for instance parallel configuration.

Parallel configuration is not the only thing prevented by project coupling, though a very important one. But I cannot name from the top of my head what other things it disturbs. Again, I'm just sharing my 15 years Gradle experience and also what the Gradle folks and the Gradle community considers current good and bad practices. What you take away from that and make out of that is your decision and you have to live with it. 🙂