Hey folks. I need to make a task which creates a fake local Maven repository (long story, involves Minecraft) which the

compileJava

task will depend on. The problem is that the

compileClaspath

configuration is resolved before task execution. So, by the time my task, which produces artifacts in a fake Maven repository that's added to the repositories, finishes execution, it will be too late and

compileJava

will fail with an error saying it can't find the (package from) dependency it's looking for. Any thoughts on how to better tackle this? I've already tried doing this fake Maven generation in configuration time itself, but that completely breaks Configuration Cache since it involves running

ExecOperations#javaExec

.

I think you can do something like this:

val files = files("build/localMaven")
files.builtBy("publishFooPulicationToBarRepository")

dependencies {
  implementation(files)
}

That would probably work, but I want it to be an actual Maven repository that can read modules as needed. So basically,

maven { url = '<file://build/localMaven|file://build/localMaven>' }

(pseudo-code).

I think the core question is why

compileClasspath

has already been resolved by the time your task executes.

The problem is that I don't know. Is there some piece of Gradle documentation that dictates when the compile classpath is resolved? I only assume it's at the end of configuration based on my interactions with it so far.

You could add a

compileClasspath.incoming.beforeResolve()

action to try to track when resolution happens (either set a breakpoint, or throw an exception and look at its stack trace in the logs)

Roger that

I want it to be an actual Maven repository that can read modules as needed

Looks like what you really want is to use coordinates instead of a directory as dependencies?

dependencies {
  implementation("com.example:foo:0.0.1")
}

?

Exactly what @Martin said. Even if you do not eagerly resolve a configuration, with configuration cache enabled the normal dependency resolution happens at the end of the configuration phase before the configuration cache entry is written and the result is put to the CC entry, so that you also do not have to do dependency resolution at execution time. So building up a repository at execution phase will most probably not work. Probably even if you create the structure at configuration time it will not be enough, as resolution would still need the metadata files to work properly, so you might at least need to create those too at configuration time. That all sounds to me like you really shouldn't do that, but maybe instead have an included build that provides the necessary dependencies at the given coordinates or something like that. 🤷‍♂️ But if "that completely breaks Configuration Cache since it involves running ExecOperations#javaExec." is your only problem, you can probably use a

ValueSource

to run the external process, there it is ok. But it will then be done on each run even if the the CC entry is reused and if it is not reused then it will even be run twice.

Using a

ValueSource

is not a problem because my tool does its own caching. But I would prefer to defer the invocation of the tool to a task. If I must, I can migrate it to use a

ValueSource

instead. I just need it to run consistently and since value sources have inherent caching, I don't know how to make it run consistently if it depends on the status of the output.

What do you mean with "value sources have inherent caching"?

If the output of a value source missed the cached value, it triggers configuration

So?

Unit

should be fine for my use case.

Wanting to jump in here as I'm trying to understand this. I am finding it hard to find any examples or actual explanations of ValueSource beyond the the javadocs/docs/github repo. Which seems to indicate that basically, a ValueSource is a Supplier. The consumer is supposed to wrap it in a Provider via

providers.of()

and invoke get() as needed. So is the intended solution just to invoke get() in an afterEvaluate block and hope no build script logic causes dependency resolution during evaluation? Do you have any actual concrete examples, ideally of javaexec? Currently we have a system like such (names/values are pseudo):

plugins {
  id 'minecraft.plugin'
}

repositories {
  maven minecraft.maven // effectively {url = "file://./gradle/fake_repo/" }
}

minecraft {
  mappings(channel: 'official', version: '1.0')
}

And for adding a dependency on Minecraft. Which is completely fake, no metadata is available on any existing maven. You would do:

implementation minecraft.dependency('net.minecrft:client:1.2.3')

Which returns a ExternalModuleDependency with our 'mappings' attribute set to 'official:1.0' It also adds

net.minecraft:client:1.2.3{mappings: 'official:1.0'}

to the list of artifacts our tool needs to create. We also need to support the ability to rename 3rd part dependencies, including sources, to the currently used name. These artifacts exist on an external repo, and ideally would be published with the gradle module attribute showing the mappings we want. Due to a long standing Gradle issues, Attributes are not properly taken into account for IDEs. There are also cases where the dependency is published using one set of mappings and we need it in another. Due to another long standing Gradle issue, Artifact Transformers are not a functional option to solve this. So it's back to our tool. (Im stating this not to be annoying, but to short circuit already hashed out avenues) So we would have something like this:

plugins {
  id 'minecraft.plugin'
}

repositories {
  maven minecraft.maven // Must be the first repo to override the existing artifacts
  maven { url = '<https://maven.example.com>' }
}

dependencies {
  implementation minecraft.dependency('com.example:dependency:1.0') {
    mappings(channel: 'official', version: '1.0')
  }
}

Currently all dependencies are aggregated together and created using a single execution of our external tool during our

syncMavenizer

task. As I'm writing this it occurs to me. In theory we should be able to just invoke our tool via

ValueSource.get()

right before we return from

minecraft.dependency

which which means it'll be invoked multiple times, and eagerly generate the artifacts. As well as mess up if anyone defines repositories after their dependencies block. But I'm sure we can throw a sane error message when that happens telling people to reorder their buildscript. Is this an intended approach, is there anything majorly wrong with this? Can toolchains be resolved at this time to be used for javaexec in ValueSource? Is there any reason to use ValueSource over just invoking java directly using normal Java Process API? Would using a detached configuration to resolve the classpath necessary to download our tool cause any other configurations to be eagerly resolved? Again, some concrete examples, and more reading (besides the official docs, and source code) on ValueSource's would be appreciated.

So is the intended solution just to invoke get() in an afterEvaluate block

As the phrase contains "in an afterEvaluate block" the answer is no, no matter what the rest of the sentence is. The main benefit you gain from using

afterEvaluate

is timing problems, ordering problems, and race conditions. Using

afterEvaluate

to fix a build problem is like using

Platform.runLater

or

SwingUtililties.invokeLater

to "fix" a GUI problem. You usually do only symptom treatment and delay the problem to a future-you for which it will be even harder to reproduce and fix the actual problem or even understand what you did. There is almost no use-case where you should use

afterEvaluate

with only very rare exceptions like using a misbehaving plugin that itself uses

afterEvaluate

and where you must do something after that action and don't have any other choice than to also use

afterEvaluate

.

Which seems to indicate that basically, a ValueSource is a Supplier

Pretty much a supplier where you are allowed to do things you are otherwise not allowed, like calling external processes at configuration time when using configuration cache, or to only have the result of some code or read files as part of the configuration cache fingerprint without having the actual files or whatever else as CC fingerprint intputs themselves.

and invoke get() as needed.

And as needed means, avoid calling

.get()

at configuration time if you can under almost all circumstances. If you do not need the value of a

Provider

- no matter whether from a

ValueSource

or however the provider was generate - at configuration time, then don't

.get()

it, or you for example introduce the same problems as from using

afterEvaluate

that this API tries to provide a way around. For a

ValueSource

-based provider `.get()`ing it at configuration time makes its value a configuration cache input and it will always be evaluated to determine whether configuration cache can be reused and if not then even a second time in the same build run. So whenever you can, do not

.get()

a

Provider

but wire it to some

Property

that is then only read at execution phase.

which means it'll be invoked multiple times, and eagerly generate the artifacts.

That sounds horribly wrong.

But I'm sure we can throw a sane error message when that happens telling people to reorder their buildscript.

That also sounds horribly wrong. Even if one plugin requires that another plugin is applied first, this is a bug in that plugin.

Can toolchains be resolved at this time to be used for javaexec in ValueSource?

I think so.

Is there any reason to use ValueSource over just invoking java directly using normal Java Process API?

At configuration time? If you ever intend to use configuration cache, yes, because you must not call unguarded external process with CC enabled at configuration time.

Would using a detached configuration to resolve the classpath necessary to download our tool cause any other configurations to be eagerly resolved?

Detached configurations should not cause other configurations to be resolved, unless you make it so like depending on the project or something like that.

Do you have any actual concrete examples, ideally of javaexec?

Actually if all your

ValueSource

is doing is calling

exec

or

javaexec

, you can also simply use

providers.exec

or

providers.javaexec

. Those are internally using a

ValueSource

. For example

dependencies {
    implementation(
        providers
            .exec {
                commandLine("bash", "-c", "echo -n commons-io:commons-io:+")
            }
            .standardOutput
            .asText
    )
}

will properly lazily add a dependency on commons-io. If you need to do more or for some other reason want to use a full value source, just inject an

ExecOperations

and use it in

obtain

like

abstract class MyValueSource : ValueSource<String, ValueSourceParameters.None> {
    @get:Inject
    abstract val execOperations: ExecOperations

    override fun obtain() = listOf(
        ByteArrayOutputStream().use {
            execOperations.exec {
                commandLine("bash", "-c", "echo -n commons-io")
                standardOutput = it
            }
            it
        },
        ByteArrayOutputStream().use {
            execOperations.exec {
                commandLine("bash", "-c", "echo -n commons-io")
                standardOutput = it
            }
            it
        },
        "+"
    ).joinToString(":") { it.toString() }
}

dependencies {
    implementation(providers.of(MyValueSource::class) {})
}

> Even if one plugin requires that another plugin is applied first, this is a bug in that plugin. This is another somewhat related question, that i'll ask later in a different thread, but we're trying to figure out a good way to add extra config options to dependencies, Basically cleaning up this code to one line that doesn't use

it

But okay, I've kept trying to avoid using afterEvaluate. The main thought was trying to avoid invoking our java tool multiple times by aggregating everything and invoking it once. > That sounds horribly wrong. What I say every 5 mins trying to wrap my head about gradle 😛 But what I meant is that it would need to invoke the tool multiple times. Once per dependency to create. Which seems like that is indeed what will end up happening as there doesn't seem to be a way to aggregate the list of artifacts to generate. Not a big deal, we do heavy caching, on hits it runs in 0.2s > If you ever intend to use configuration cache We have to, from what I heard its being forced in the future. So this entire effort is to do things correctly so that gradle is happy in the future. The provider would be doing more then just calling the executable. It has to return a valid dependency with the proper attributes. While also executing the task. But since DependecyHandler supports Providers. Could we do something like this? Non functional, just to illustrate the idea, creating and executing a ValueSource in a Provider's obtain.

FileCollection TOOL = project.configurations.detachedConfiguration(project.dependencies.create('myorg:mytool:1.0'));
Provider<RegularFile> JRE = project.getExtension('javaToolchains')
    .launcherFor(spec -> spec.languageVersion = JavaLanguageVersion.of(25))
    .map(JavaLauncher::getExecutablePath);

public Provider<ExternalModuleDependency> dependency(Object value, Closure<?> closure) {
  return providers.provider(() -> {
    var dep = project.dependencies.create(value, module -> {
        Closures.invoke(closure, module);
        module.attributes.attribute("mapping", "official:1.0");
        return module;
    });
    
    var runTool = project.providers.javaexec(spec -> {
      spec.classpath = TOOL
      spec.executable = JRE.get().getAsFile()
      spec.args = list.of(
        "--artifact", dep.group + ':' + dep.name + ':' + dep.version,
        "--mappings", "official:1.0"
      )
    })
    // This part doesn't feel right. But we're inside a provider being called already, so should be fine to call?
    runTool.getResult().get();
    
    return dep;
  };
}

But okay, I've kept trying to avoid using afterEvaluate. The main thought was trying to avoid invoking our java tool multiple times by aggregating everything and invoking it once.

Maybe you can leverage

configuration.withDependencies { ... }

that is a last-minute hook that is executed right before a configuration takes part in dependency resolution the first time. It is for example intended to add last-minute dependencies based on other dependencies that were declared and similar. But maybe you can for example use it to check the declared dependencies, check which need handling and then call your tool for all those of one configuration once last-minute.

We have to, from what I heard its being forced in the future. So this entire effort is to do things correctly so that gradle is happy in the future.

Well, that is the plan. But it was also the plan to replace the old configuration with the software model. Then plans changed and the software model got deprecated and removed again. Also, what is best-practice way to go today can be discouraged bad-practice tomorrow. Happened multiple times with Gradle already as it is a quite evolving project still. So you can imho just optimize for the current state mostly. I'd also expect CC being the only way in the future and also IP (isolated projects) to come. Both even if optional should be strived for as they bring big performance gains.

creating and executing a ValueSource in a Provider's obtain.

A Provider does not have an

obtain

, only a

ValueSource

has. Also keep in mind, that the

Provider

chain including supplier,

map

action,

flatMap

action, and so on is executed each time the resulting

Provider

is queried, unless you do something like

finalizeValue

or

finalizeValueOnRead

. See also https://github.com/gradle/gradle/issues/25550. A `ValueSource`'s

obtain

on the other hand is only executed once in the build if CC entry does not exist or is reused, and twice if the CC entry is existing but not reused. You can also

zip

two providers to lazily combine their values into a new

Provider

.

Closure<?> closure

Never use

Closure

anywhere in your API, this is highly Groovy-specific and will for example not work nicely with Kotlin DSL or any other JVM language. Better always use an

Action

instead which can be uniformly used nicely.

But we're inside a provider being called already, so should be fine to call?

Like in one of my examples, when currently evaluating the value of a

Provider

calling

get()

on another

Provider

should be fine, though if possible, `zip`ing two providers would be preferable. Especially when one of the `Provider`s might carry an implicit task dependency. A

zip

would preserve that task dependency.

Using configuration.withDependencies doesn't seem like it'd work because we don't know what configuration the dependency is being added to. Could in theory add a callback to scan ALL configurations but that get deep into the DomainObjectCollection.all stuff which seems like a hack. Okay, so maybe something more like this would be viable:

FileCollection TOOL = detached('mytool')
Provider<RegularFile> JRE = toolchain.laucherFor(25)

class ToolValueSource implements ValueSource<Boolean, ToolValueSource.Params> {
  static interface Params extends ValueSourceparameters {
    FileCollection tool();
    RegularFileProperty jre();
    ListProperty<String> args();
  }

  private final ExecOperations execOperations;

  @Inject
  public ToolValueSource(ExecOperations execOperations) {
    this.execOperations = execOperations;
  }
  
  public Boolean obtain() {
    this.execOperations.javaexe(spec -> {
      spec.classpath = this.parameters.tool()
      spec.executable = this.parameters.jre()
      spec.args = this.parameters.args()
    })
    return true; // Return true because this shouldn't invalidate the CC, it just needs to run the tool
  }
}

Provider<ExternalModuleDependency> dep(Object value, Action<? extends ExternalModuleDependency> config) {
  var module = (ExtenalModuleDependency)project.dependencies.create(value, dep -> {
    if (!(dep instanceof ExternalModuleDepenency external)) throw Unsupported()
    config.apply(external);
    external.attributes.attribute("mapping", "official:1.0");
    return external;
  });

  // this will create the ValueSource at configuration time and thus make the CC know it exists.
  // This does have the issue, that is repositories may not be defined yet depending on buildscript layout
  var runTool = project.providers.of(spec -> {
    var args = new ArrayList<String>()
    args.addAll(
      "--artifact", module.group + ':' + module.name + ':' + module.version,
      "--mappings", "official:1.0"
    )
    project.repositories.withType(MavenArtifactRepository).each(repo -> args.addAll(["--repo", repo.url])

    spec.tool = TOOL
    spec.jre = JRE.get().getAsFile()
    spec.ags = args
  });
  
  return () -> {
    // Should be safe because we're in a provider
    // And ValueSources are memonized so it shouldn't matter to blidly call get
    // Tho it would be nice to somehow tell Gradle that the ValueSource is an input of this dependency and have it automatically call obtain
    runTool.get(); 
    return module;
  };
}

I'll see if i can something functional today.

Also, what is best-practice way to go today can be discouraged bad-practice tomorrow.

Happened multiple times with Gradle already as it is a quite evolving project still.

We have been chasing Gradle's ever evolving landscape for 10 years, and honestly i'm just burnt out. This is the main reason we have decided to completely sidestep Gradle and build our own tool for doing all the heavy lifting. Its just a matter of trying to make the glue between gradle and our tool both light, and correct as possible. For additional context, and the arguments I need to gather, you can see our existing SyncMavenizer task.

Using configuration.withDependencies doesn't seem like it'd work because we don't know what configuration the dependency is being added to.

So? Just add the hook to all configurations using

configureEach

? That's why I said it will be one call to your tool per configuration that has according dependencies.

that get deep into the DomainObjectCollection.all stuff which seems like a hack.

all

is evil, especially with collections that are already treated lazily be Gradle like tasks and configurations. Didn't check that snippet in detail, but 3 quick points from a cursory look:

this.execOperations.javaexe(spec -> {

As all that this value source is doing is calling

javaexec

(assuming

javaexe

is a typo), just use

providers.javaexec

unless it is important you see the output of the tool during run.

return true; // Return true because this shouldn't invalidate the CC, it just needs to run the tool

Just use

Void

and return

null

?

// this will create the ValueSource at configuration time and thus make the CC know it exists.

When you create a value source is quite irrelevant for CC. Relevant is only when someone `get()`s it.

Alright, so I was able to get something that appears to be functioning and calling things at the correct time. The crux is makeing

minecraft.dependency

return

() -> { runTool.get(); return dependency; }

Which means that before gradle knows the dependency exists, the ValueSource will be invoked and thus the dependecy created. Which is our entire issue. The configure for the ValueSource parameters seems to be called twice. Both after the full evaluation {but before afterEvauate} happens. So the ordering of repository blocks in the buildscript won't be an issue.