This message was deleted.

Hi, to make it clear from the start, this is a user community. I'm just a user like you, not affiliated with Gradle in any way. Just to prevent confusion as you wrote things like "your docs".

Should I be using ‘include’ or ‘includeBuild’, or both?

I'd say that depends on your use-case and intents. You can use both to have more conceptual structure. If performance is your main point,

include

might be faster, but this is just a gut feeling and you would need to do measurements for your specific case to see whether that is true and significant enough. The original use-case of an included build was, that you can substitute a binary dependency by an included build, so you for example have project X and library A in complete separate projects and while adding a feature to A you want to right away test it within X so you use a composite build where you include the build of A in the build of X. But it evolved quite a bit and you can also use it as a way to structure your build and write local build logic.

Why can I not declare that I do NOT want this behaviour?

Because you just don't know how to do it. 😉

include(":app")
project(":app").projectDir = file("modules/applications/app")

what is the absolute fastest way of structuring my build files

I don't think anyone can give any meaningful answer to such a question. "Fastest" is very flexible. "Fastest" in what regards? build script compilation? build script evaluation? task execution? whatever? And even if you could answer that, I don't think there can be a good answer given. What comes to mind is the rules I learned in university: 1. make it work - optimally with also having test that verify it 2. make it nice by refactoring - here the tests come in handy 3. optimize cpu / ram usage - but only where you have a measurable problem

but I’m not quite sure how to ensure that I do not inherit dependencies when I don’t want to

I'm not sure what you are asking in this paragraph, can you maybe ask a bit more concretely?

Why is buildSrc promoted on the gradle website as the absolute go to for extracting build logic?

Where is is promoted as "the absolute go to"? I cannot remember this being stated somewhere and as you linked to yourself, there are also other parts in the docs which show an included build instead which objects "the absolute go to". That

buildSrc

is promoted in most places probably has two reasons, but this is just guessing by me. 1. it is what was there first with composite builds coming later. 2. in 98.7 % of projects (actual number totally made up) there is not much difference neither in functionality nor in performance whether you use

buildSrc

or an included build, and using

buildSrc

might be a µ easier maybe for the average user. If you for example only have things in

buildSrc

that all projects in the build use, there is not too much difference to using an included build, it will be run before each build just the same. But if you have many build logic and only some projects in the build use some parts and other projects other parts and you also separate these parts into different projects in the included build, then you might get a significant difference maybe in some situations. I personally prefer to use an included build too though instead of

buildSrc

unless I need the specialities of

buildSrc

, for example ot monkey-patch a 3rd-party plugin class.

Am I doing anything obviously wrong which is causing slowness?

Sorry, but I'm not going to analyze your build for a generic slowness. Not without you paying a horrendous amount of money. 😄

thanks for the reply, my apologies about not realising that this was a community space. totally my bad. 1. the use case for include(Build) is that I primarily wanted three separate nested folders: plugins, platform(s), and modules. the only suggested way from the docs that I could find to have precompiledScript plugins, that are able to be built completely in parallel. I wanted the latest changes from my plugin applied to all projects they were relevant to, along with any other changes to plugins since I last compiled the plugins. My observations were that buildSrc would recompile everything regardless, but that using an includedBuild, alongside composition of plugins, would give me this. Are there alternative ways of doing this instead? 2. Correct me if I'm wrong here, but that is NOT functionally equivalent. I want the project accessible on path 'modulesapplications:app', but don't want to incur the cost of observed dependency resolution slowness when this is the case (it's noticeably slower in an arguably small project). if this means that I can do include(modulesapplications:app) followed by the project dir, this was something I originally tried and had noticed large regressions in overall build script speed (task execution, compilation, dependency resolution, basically anything related to running, and compiling a gradle task) 3. I mainly wanted the fastest in terms of overall task execution, build script compilation, build script evaluation, dependency resolution/downloads 4. Lmao absolutely no worries about not wanting to comment on the suggestions for improvements. Figured I'd put it there in case anyone felt generous ;) 5. edit: to clarify on the paragraph I somehow managed to mince my words on: how can I ensure that when composing plugins to create larger plugins, that I 1) have an option of inheriting the dependencies of the plugins, and 2) the inheritance of these dependencies is off by default, and 3) if they're off by default already, what am I doing that is seemingly including them in the way I'm doing it (unintended)?

My observations were that buildSrc would recompile everything regardless

That's totally not the case.

buildSrc

as well does the same up-to-date checks. Regarding building it, it is a normal build. Just the integration in the "parent" build is a bit different, in that it is always built even if nothing uses things in it and prepended to all build script classpaths by living in a parent classloader. But the up-to-date checks are done the same.

Correct me if I'm wrong here, but that is NOT functionally equivalent.

It is functionally equivalent, it just is not structurally equivalent.

I want the project accessible on path 'modulesapplications:app',

Imagine project paths like file system paths with

:

being the path separator. If you want directory to have the directory

/modules/applications/app

, you have to have the parent directories

/modules/applications

,

/modules

, and

/

. The same with project paths. The project itself is

app

which is a subproject of

applications

which is a subproject of

modules

which is a subproject of the root project. If you just want the logical strings in the name when you refer to it, you can for example use

modules-applications-app

. But if you want to have the exact path

:modules:applications:app

for whatever reason, there is no way besides also having the projects

:modules:applications

,

:modules

, and

:

.

but don't want to incur the cost of observed dependency resolution slowness when this is the case (it's noticeably slower in an arguably small project).

I'm not aware of having those empty mid-projects making anything significantly slower, especially not dependency resolution. Maybe if you follow bad practices like doing cross-project configuration using

subprojects { ... }

,

allprojects { ... }

,

project(...) { ... ]

, or alike and thus putting logic to those projects when they actually should have none. But without that, I'd be curious how you measured a difference and how big a difference you measured.

have an option of inheriting the dependencies of the plugins

Depending from one plugin on another plugin is like depending from one library to another library. You get exactly the dependencies library you depend on needs too or the other library cannot work. If this is more a react-style plugin à la

pluginManager.withPlugin("asdfasdf") { ... }

then the dependency should be

compileOnly

. If that is not your use-case but something different, I think I still did not fully get what you ask for.

Ahhhhh that's really interesting regarding buildSrc. I had completely misunderstood what the documentation meant. The comment "..in that it is always built even if nothing uses things in it and pretended to all build script classpaths by living in a parent class loader" just to clarify my understanding, what specifically do you mean by that sentence? it's actually loaded with a lot of information when thinking about it. the assumptions I've made reading it are: 1. by adding everything in buildSrc to the classpath of the parent class loader, all plugins, binary or otherwise, are able to be used inside of a build script without declaring a dependency implementation like you would a composite build 2. when you say that it always builds, are you talking about compilation? or are you also talking about task execution? or both? for instance if I have a bunch of plugins that are run as a part of check (using tasks.check { dependsOn (pluginTask) }, and I run .gradlew check, will this plugin task run if if there's nothing using it? 3. i'm no doubt doing some bad practises and am wanting to learn, which is why i'm here. consider the following scenario: we have three sub-projects. each of these sub-projects uses a base-plugin (to configure java version), we then have a base-kotlin plugin (that applies kotlin specific things), and we then have a kotlin-application and kotlin-library plugin respectively. are you saying that if any of the sub-projects build scripts use even something like project() - under dependencies like: { project(":some-sub-project")}, that for the above scenario (where two sub-projects are a kotlin-library and one is a kotlin-application) - that it effectively means any of the logic inside of a kotlin-library is used for the kotlin-application? even though the logic appears separated? sub-project dependency tree would be: sub-project-one (kotlin-application) build.gradle.kts<---- no dependencies on anything except kotlin-application sub-project-two (kotlin-library) build.gradle.kts <----- this declares a depenency on sub-project-one using dependencies { project(":sub-project-one") }, sub-project-three (kotlin-library) build.gradle.kts <---- no dependencies on anything except kotlin-library how would you suggest modelling or structuring projects like above? 4. consider the following scenario: • the end result i am after is a plugin that adds junit test libraries (where the versions used are declared via a dependency constraint, in a platform plugin), and also adds a task named "generateClasspath" which generates the classpath at runtime • because i don't want to have to declare dependencies every single for each of the sub-projects i use, i want to be able to do plugins { id("my-kotlin-application-plugin") } <---- which will apply the above • this will mean that instead of having to manually declare the junit dependencies and task ie:

task.register("generateClasspath") {
 // something here
}

dependencies {
    // something here
}

i can instead just use:

plugins {
   id("my-kotlin-application-plugin")
 }

dependencies {
 // only extra deps go here
}

// when i run :project:generateClasspath , the task should be found

what's the 'best practise way' of doing this?

by adding everything in buildSrc to the classpath of the parent class loader, all plugins, binary or otherwise, are able to be used inside of a build script without declaring a dependency

yes

like you would a composite build

Depending on what you mean, you would not. If you apply a plugin in the

plugins { ... }

block like you always should, the included build building it is automatically found, built, and added to the classpath. But unlike with

buildSrc

only to those build scripts where it actually is used and only if it is used. As

buildSrc

is always put there regardless, you could also just declare a dependency in

buildSrc

and have it usable in all build scripts, which is of course not the case with and included build.

when you say that it always builds, are you talking about compilation?

what is "it"?

buildSrc

is always built, whether something from it used or not. an included build is only built if something from it is used, but then also always. "built" does not mean it does work, it has the same up-to-date logic or build cache and so on like any other build.

or are you also talking about task execution?

task execution of what? tasks in

buildSrc

/ the included build? see above task execution of tasks contributed by those plugins? that's irrelevant, or better said exactly the same. Like with almost anything in Gradle, things that need to run are run, things that can be avoided are avoided.

for instance if I have a bunch of plugins that are run as a part of check (using tasks.check { dependsOn (pluginTask) }, and I run .gradlew check, will this plugin task run if if there's nothing using it?

What do you mean by nothing using it? You are using it. You depend on it from the

check

task. And before you probably applied the plugin that added that task you depend on, so even without the

dependsOn

, you are using the plugin. Whether any task of the plugin are run depends on what the plugin does, where it hooks the tasks, or where you hook the tasks in your build script.

i'm no doubt doing some bad practises and am wanting to learn, which is why i'm here.

👌

are you saying that if any of the sub-projects build scripts use even something like project() - under dependencies like: { project(":some-sub-project")}, that for the above scenario (where two sub-projects are a kotlin-library and one is a kotlin-application) - that it effectively means any of the logic inside of a kotlin-library is used for the kotlin-application? even though the logic appears separated?

No, I'm not saying anything even similar, that would be total non-sense and render Gradle totally unfit to do any serious build work. You confuse "being in the classpath" with "being applied". The former just means it is available to be used, the latter means you said "I want to use it here". But if something is in the classpath and changes, this changes the classpath of everything, not only the tasks coming from that plugin and thus may cause things to be re-executed needlessly as the classspath changed and Gradle cannot know whether any task would behave differently in a different classpath.

this declares a depenency on sub-project-one using dependencies { project(":sub-project-one") }

This is a production dependency. Do not confuse production with build-logic. Just by adding a dependency on another project does not change anything in any build script classpath or in any project configuration, besides that there is this dependency.

what's the 'best practise way' of doing this?

Exactly what you said. A convention plugin that adds those dependencies and registers that task, that you then apply to the projects where you want this convention to be in effect.

what is "it"?

buildSrc

an included build is only built if something from it is used, but then also always.

sorry i really don't understand the last part of this. it's only built if something from it is used, but then also always?

"built" does not mean it does work

so what does built mean in the context you're using it? because in basically any other software developer context, build/built implies that it's doing work (usually a loose reference to either task execution, or code compilation)

What do you mean by nothing using it? You are using it.

sorry this was a terminology mis-match. as stated by you, what i'm referring to here is that the plugin is not being actively applied across any of the sub-projects

But if something is in the classpath and changes, this changes the classpath of everything, not only the tasks coming from that plugin and thus may cause things to be re-executed needlessly as the classspath changed and Gradle cannot know whether any task would behave differently in a different classpat

holy shit this was a huge revelation. it makes so much sense, but is also a subtle nuance. thank you so much for that statement/moment of clarity so is this the correct way to do the above scenario?

---------- platform/build.gradle.kts  ----------
    
plugins {
  id("java-platform") 
}

group = "com.nophasenokill.platform"
dependencies {
    constraints {
        api("org.jetbrains.kotlin.jvm:org.jetbrains.kotlin.jvm.gradle.plugin:1.9.20")
        api("org.jetbrains.kotlin:kotlin-bom:1.9.20") {
            because("It matches the version of 3.2.0 for spring")
        }

        api("org.jetbrains.kotlin:kotlin-stdlib:1.9.20")

        api("org.junit:junit-bom:5.10.1")
 ---------- platform/settings.gradle.kts  ----------
pluginManagement {
    includeBuild("../plugins")
}

dependencyResolutionManagement {

    repositories {
        gradlePluginPortal()
    }
}

rootProject.name = "platform"
 ---------- plugins/my-kotlin-plugin/settings.gradle.kts  ----------


dependencyResolutionManagement {
    // explicitly gradle plugin portal because we only want to search for our convention plugins,
    // where the convention plugins delegate the dependency retrieval to the platform
    repositories.gradlePluginPortal()
    includeBuild("../platform")
}

------- plugins/my-kotlin-plugin/build.gradle.kts ----------

plugins {
    `kotlin-dsl`
}

dependencies {
    implementation(project(":base-plugin"))
}


 ---------- plugins/my-kotlin-plugin/src/main/kotlin/my-kotlin-plugin.gradle.kts  ----------

plugins {
    id("base-plugin")
    id("org.jetbrains.kotlin.jvm")
}

dependencies {
    // enforces that versions from each of the boms are used
    implementation(enforcedPlatform("com.nophasenokill.platform:platform"))
    implementation(enforcedPlatform("org.jetbrains.kotlin:kotlin-bom"))

    // applies junit deps to projects that apply the plugin
    testImplementation(enforcedPlatform("org.junit:junit-bom"))
    testImplementation("org.junit.jupiter:junit-jupiter")
    testImplementation("org.junit.jupiter:junit-jupiter-api")
    testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}

tasks.test {
    useJUnitPlatform()
    maxParallelForks = (Runtime.getRuntime().availableProcessors() / 2).coerceAtLeast(1)
}

it's only built if something from it is used, but then also always?

if nothing from it is used, it is never built. if something is used, it is built everytime you run a build. it might everything be up-to-date, but build is executed everytime you run a build.

so what does built mean in the context you're using it? because in basically any other software developer context, build/built implies that it's doing work (usually a loose reference to either task execution, or code compilation)

As I said Gradle is great in avoiding unnecessary work. It being built means it being built, that is the build executed. All tasks might be up-to-date though if nothing changed and thus no work done - besides the up-to-date checks of course. Exactly the same as with your "main" build. You tell it "build my project" and if nothing changed, it checks, sees everything is up-to-date, and is just done.

what i'm referring to here is that the plugin is not being actively applied across any of the sub-projects

If it is not applied, how do you do

tasks.check { dependsOn (pluginTask) }

? Assuming

pluginTask

is a task added by your plugin, you can hardly depend on the task without applying the plugin.

so is this the correct way to do the above scenario?

Unlikely. • For example doing

includeBuild

within

dependencyResolutionManagement { ... }

is just visual clutter as there is no such method in that scope, it just calls the top-level

includeBuild

. So either call

includeBuild

within

pluginManagement { ... }

if you are including a build with build logic, or call it top-level if you are including a build with production dependencies. • And including

platform

from

plugins

and

plugins

from

platform

is a bit circular, unless you really depend from something in platform on something in plugins and from something plugins on something platform that does not build a cyclic dependency. But even then it would at least be bad architecture. From what you have shown you do not use anything from plugins within platform, so there is probably no actual circularity, but you should then also remove that unnecessary

includeBuild

. •

id("base-plugin")

this is not a good idea, custom plugins should always have a namespace, so at least one dot in the id somewhere to prevent current or future name-clashes with built-in plugins. •

maxParallelForks

I personally would prefer using parallelity within the test engine (Spock and Jupiter for example provide this) if possible as it can better distribute the work. Of course the tests have to be written in a way that they can be executed in parallel within one JVM for this. •

enforcedPlatform

strongly reconsider whether you really want to do this. In most situations

platform

is more approprate.

enforcedPlatform

is more for edge-cases where you really have not much other choice.

schweet, that clarifies a lot of my remaining questions. 1. yeah i forgot to remove this from my example, but there is a dependency on both (currently figuring out other ways to do what i need to do, just haven't migrated it over yet). basically i have a meta-plugin which checks the build scripts ordering, and alphabetises them. this is currently defined in plugins, and platform relies on it. but you're right, it's awful and needs to be changed. 2. interesting about the naming for custom plugins, hadn't thought of that at all 3. is there a reason for this preference? have you observed better optimizations in the test engines themselves? while i chose the above, i actually don't have a strong preference either way. i was under the assumption that you needed this to enable the paralleleity for the underlying test engine, but if not, i'm all for removing it 4. there were conflicts between the kotlin-bom and the dependency-analysis-plugin (which i've left out of the above), which made using the enforcedPlatform appropriate as it rules out the multiple transitives issue. if you have any better suggestions for how i'd do this, i'm open to ideas/other thoughts on solving this. what are your concerns with using enforcedPlatform? if i want the versions declared inside of my project to be consistent, transitively, this really seems like the only way to do it nicely

is there a reason for this preference?

As I said, the test engine which you tell "execute all tests" can better leverage your CPUs properly if the tests are properly written parallel-safe. If you for example have 9 test classes and set

maxParallelForks

to 3, Gradle will fork 3 test workers and give each 2 test classes (or something like that) but cannot know whether maybe 2 of the workers are finished in 5 seconds while the last has a monster test class that needs an hour. With parallelity in the test engine all tests that can run in parallel can run in parallel properly.

i was under the assumption that you needed this to enable the paralleleity for the underlying test engine

No, that is different layers of parallelity. With that you tell Gradle to kick off multiple test workers, giving each some test classes to execute. Within each of these test workers the tests given could then of course also run in parallel if you have set it up, which might even decrease performance then if you for example have 8 processors and tell Gradle to spawn 6 test workers and the test engine is also configured for parallelity and spawns 6 threads (or whatever) to run tests in parallel and you are suddenly trying to execute 36 tests in parallel with only 8 processors. These parallelities are totally independent layers.

if you have any better suggestions for how i'd do this

Well, if you have good reasons to use it to fix some problem, go on with it, as long as you are not using it in some library you are publishing.

what are your concerns with using enforcedPlatform?

That the creator of this system personally told me not to use it unless really really necessary. 🙂 If you for example have an enforced platform dictating some version and then having some other dependency that also has a strict version for the same dependency, you are f***ed as a conflict between two different strict versions cannot be resolved, unless you then manually do some deep dark things like using a resolution strategy to force a specific version which overrides even strict versions. (Basically what the Spring dependency management plugin is doing and one of the many reasons one should never use it 🙂)

awesome - thanks for the overview, really appreciate it :)

Example build scan is: https://scans.gradle.com/s/3dgkbsftjhajk/performance/configuration

Three

compileJava

tasks were created at configuration time and 3 other tasks, not shown. So yes, I'd say that is not good. When using included builds afair it might be expected, but afaiu in your build there is none. If you set the system property

org.gradle.internal.tasks.stats

to an empty value you should get in the output which tasks were created, hopefully also the three that are missing in the build scan. And if you set the same property to a filename, you even get stacktraces for each of those created tasks in that file and can check why they were created hopefully.

Do you have any idea what the detachedConfigurations are from

Hard to guess, why?

My build scripts in each sub-project now mainly consist of the following repitition:

Do you really get siginificant enough time improvement by disabling those tasks explictily instead of just having them skipped due to "NO SOURCE"?

when trying to move to these into convention plugins/precompiled-scripts, I've found it very difficult to get the exact same amount of tasks/exact replica of behaviour.

Why? If you do the same and apply it to the same projects where you did it manually, you should get the identical result.

Realistically I'd have thought I'd be able to have as close to the exact same number of tasks

I'd say so too. Well, if you create a new build for those convention plugins and not have one yet, you will of course add the tasks for that build. But if that is what you are seeing and are concerned about, then I'd say don't sacrifice clearness and deduplication for a quantum of speed, unless you have an actual and measurable problem.

ie: convention plugin number one may share 2 tasks with convention plugin number two (compileJava/compileKotlin etc). So the total expected tasks should be: convention plugin one tasks + convention two tasks - 2

I'm not sure whether this is the best way to describe it, as I didn't really understand what you try to say. 😄 But I'd tend to agree.

didn't even realise i could use that command to see information like that. is there any documentation on this? i can't seem to find much. as for measurable performance, the short answer is no, but i'm just playing around learning different things. HAHAHA as I typed the text "best way to describe it" - i went yep..... who am i kidding. thanks for your patience while i feel my way through it :D

"that command" meaning?

sorry yes the system property

Well, the

internal

in the name should tell you something 😄

schweet, i've figured out what the issue is anyway through debugging with intellij. were there any other system properties or things that were hidden away that i might find useful when debugging this type of thing?

None I'm aware of right now

for completeness, the configuration avoidance problems I was having are due to bugs in gradle java plugins. this has been confirmed here https://github.com/gradle/gradle/issues/28331 thanks for the help once again good sir, really appreciate it :)