https://gradle.com/ logo
Join Slack
Powered by
:wave: Hello, team! I am developing a Gradle plug...
# community-support
x
👋 Hello, team! I am developing a Gradle plugin designed to resolve dependencies using system artifacts. The plugin is initiated from an init script. I have implemented dependency resolution for plugins in the 
beforeSettings
phase and for regular dependencies in the 
projectsLoaded
phase. However, I have been struggling for a while to determine in which hook I should add transitive dependencies for plugins. For example, the plugin (
com.gradleup.shadow
) has the following dependencies: - 
org.vafer:jdependency
- 
org.apache.logging.log4j:log4j-core
- 
org.codehaus.plexus:plexus-xml
- 
org.codehaus.plexus:plexus-utils
- 
commons-io:commons-io
- 
org.jdom:jdom2
- 
org.apache.ant:ant
All of these dependencies are present in my system, including their artifacts and POM files. Currently, my plugin is unable to resolve transitive dependencies for plugins because I am unsure about the correct hook to use for adding them to the configuration and the precise method to do so. As a result, the build process fails with an error. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':shadowJar'.
org/vafer/jdependency/Clazzpath
* Try:
Run with --stacktrace option to get the stack trace.
Run with --debug option to get more log output.
Run with --scan to get full insights.
Get more help at https://help.gradle.org.
BUILD FAILED in 1s 5 actionable tasks: 1 executed, 4 up-to-date and if I add this to build.gradle
Copy code
buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.apache.ant:ant:1.10.14")
        classpath("org.ow2.asm:asm-commons:9.8")
        classpath("org.ow2.asm:asm:9.8")
        classpath("commons-io:commons-io:2.18.0")
        classpath("org.vafer:jdependency:2.13")
    }
}
then I get BUILD SUCCESSFUL in 1s 6 actionable tasks: 6 up-to-date
Copy code
Moreover, both attempts were made in an environment without Internet access and with a cleared cache.

Please tell me how to correctly add such dependencies to the configuration using the API so that they fall into the plugin classpath, I will be very grateful
Here is an example of a method for resolving plugins
Copy code
private void configurePluginResolution(Settings settings) {
    settings.getPluginManagement().getResolutionStrategy().eachPlugin(requested -> {
        String pluginId = requested.getRequested().getId().getId();
        if (pluginId.startsWith("org.gradle.") || !pluginId.contains(".")) return;

        MavenCoordinate coord = versionScanner.findPluginArtifact(pluginId, logger);
        if (coord != null && coord.isValid()) {
            String module = coord.getGroupId() + ":" + coord.getArtifactId() + ":" + coord.getVersion();
            requested.useModule(module);
            requested.useVersion(coord.getVersion());
            resolvedPlugins.put(pluginId, coord);
            logger.lifecycle("Resolved plugin: {} → {}", pluginId, module);
            collectPluginDependencies(coord);
        } else {
            logger.warn("Plugin not resolved: {}", pluginId);
        }
    });
}
it is used in such a hook
Copy code
PluginsDependenciesHandler pluginsHandler = new PluginsDependenciesHandler();
gradle.beforeSettings(pluginsHandler::handle);
there is also a method for direct dependencies
Copy code
ProjectDependenciesHandler projectHandler = new ProjectDependenciesHandler();
gradle.projectsLoaded(projectHandler::addRepository);
gradle.projectsEvaluated(projectHandler::handleAfterConfiguration);
But I don't understand in which hook the dependencies that plugins require should be processed and how exactly they should be added to the configuration
v
I guess the best would be if you not try to manipulate the dependencies, but simply change the repositories so that you add a repository that provides the dependencies just how they are declared. Btw. be aware that "POM"s might not be sufficient. For example there can be different variants of a plugin depending on which Gradle version requests them and those can only be resolved properly when having the Gradle Module Metadata available, which declares the feature variants and makes them properly usable, ...
x
Copy code
The thing is that I am a Linux distribution maintainer and we have jar files and pom files separated into different directories in the system, and in this case gradle will not take into account transitive dependencies as far as I understand. It often happens that the number of dependencies in a project can be downloaded insanely, but not all of them are actually required, so I need to have full control over dependency resolution
v
and in this case gradle will not take into account transitive dependencies as far as I understand
I think if you set up an Ivy repository with according patterns it should work just fine. But again, if you only have POMs but not GMMs, you potentially change what is resolved and might break builds and this can be an obvious break or also some silent one that you do not right away notice!
x
I have two directories /usr/share/java with jar files and the /usr/share/maven-poms directory with pom files so, I'm currently installing the flatDir repository on a folder with jar files. I take a dependency, use patterns to search for a pom File for it, parse all transitives using the maven model, and use artifactId to search for a jar file from the pom file because maintainers like to rename everything as they want. moreover, there is a different approach to naming plugins in the repository. So I need to find out anyway.
v
To be honest, all this really sounds to me like you should not do it. 😄
And again, even if you correctly parse and process all POMs for the whole dependency tree, you will still most likely break projects as they rely on the better dependency resolution capabilities provided by Gradle Module Metadata.
x
GMM, this is a problem for me, not the current one, right now I'm focused on building simple projects, so I still need to know this, even if I'm doing crazy stuff.
v
Ok, good luck with it. 🙂 I guess you would need to do the crazy stuff you do for normal dependencies on the buildscript dependencies to achieve the same for plugins or something like that, but no idea, never tried anything similar to that.
x
And who can I find out from? Do you know anyone who has been or is currently developing the Dependency Resolution mechanism?
Thank you in advance
v
I'm just a user like you, this is mainly a user-help-users community
You can probably find out who develops dependency resolution by looking at the Git history, but whether you then can reach those persons I don't know.
j
@Xeno It seems that you are trying to add buildscript dependencies programmatically in the same build being run, but that's not possible as the dependencies are evaluated in a separate early pass by Gradle before running the script. Once it starts running, it's too late to change them. You might be able to achieve something from your init script by using your 
projectsLoaded
hook (link to doc, see the example there, it looks very much like what you're trying to do) but it's also possible that you get some error before your hook is even called. Anyway I think that adding transitive dependencies as direct dependencies is the wrong approach here and is likely to cause hard to solve issues in more complex projects. FI I'm currently working on upgrading the Gradle package in Debian. In this distribution and its derivatives there is a "system Maven repository" under 
/usr/share/maven-repo
that can be used directly by Maven and Gradle:
Copy code
$ ls -l /usr/share/maven-repo/log4j/log4j/1.2.17/
total 4
lrwxrwxrwx 1 root root   37 Jan 31  2022 log4j-1.2.17.jar -> ../../../../java/log4j-1.2-1.2.17.jar
-rw-r--r-- 1 root root 3395 Jan 31  2022 log4j-1.2.17.pom
My own init scripts are (among other things) replacing external repositories with this system repository, and no specific handling is necessary for transitive dependencies, they are automatically looked up and found there.
v
Does the Debian system repo have the proper GMM files too, or also only POMs?
j
Currently gradle metadata isn't automatically installed there, and so far no package needed it anyway, but that's indeed some future work to do on the tooling.
v
Well, you might not immediately notice that it is necessary. It can lead to different result in dependency resolution which can cause different artifacts to be packaged or different dependencies being packaged and this might cause a build error, but might also just cause some runtime error or changed behaviour and maybe only if a certain code path is hit.
j
Normally there is a special comment added to the generated POM file that should redirect Gradle to the module metadata file, does it show a warning or error when it finds the comment but not the GMM?
v
No, it then just uses the POM information and ignores the missing GMM iirc. But yes, if you only have POMs without that comment in your system repo, it should indeed be fine.
That's for example also one of the problems when using the broken-by-design-in-Maven-already 
mavenLocal()
. If you have library X which provides GMM and a Gradle build that requires the GMM to work properly and have unfiltered 
mavenLocal()
in that build and before the real repository, it works fine if the dependency is fully present in 
mavenLocal()
because you for example did 
publishToMavenLocal
for X. And it also works fine if it is not present in 
mavenLocal()
at all. But if you then execute any Maven build that also uses X, Maven uses the hermaphrodite 
mavenLocal()
as download-cache and you only find those files that the Maven build needed, which of course does not include GMM files or also some classified artrifacts and so on. If you then run said Gradle build again, it sees the POM in 
mavenLocal()
, expects the dependency to be completely present there and lives with what it finds, failing (or misbehaving) as the GMM file is not found and thus resolution suddenly changed.
j
It might be worth a bug report or feature request against Gradle then (didn't check if there is already one): silently ignoring 
.module
files that should have been there and then having random issues further down is not too helpful. Also I don't know if repository managers / mirrors / proxies / caches are automatically pulling these files nowadays, but I suspect this could be, or has been an issue for folks depending on internal mirrors.
v
They should usually pull them automatically (I know Nexus does). Especially in Maven no proxy / mirror can know which files are actually there as it could always be additional files like classified artifacts or artifacts with other extensions and so on and Maven POMs usually know nothing about those explicitly. That the missing GMM file is ignored is intentional afair, but feel free to report a bug and see whether that get changed.
Ah, I did already - kind of 😄 github.com/gradle/gradle/issues/15893
👀 1
The suggested strategy would help for the mL case, but not for the bad mirror case, which can probably not be done, see the comments in there, there are libs out there that have the marker but don't publish the GMM file alongside and those would fail and break builds.
j
there are libs out there that have the marker but don't publish the GMM file alongside
🤦 that had to happen, obviously ...
v
It did not have to, but it did. I'd say that is a publishing bug, but in simple cases the GMM does not have more information than the POM anyway, so consuming just the POM still works fine and the same.
x
@Julien Plissonneau Duquène The thing is that I am the maintainer of the ALT Linux distribution and since there can only be 1 version of the library in the system, the dependency graph actually becomes flat and version conflicts are not a plugin problem, but a repository problem. Thus, I think that adding transitive dependencies as usual will not cause anything strange. Moreover, the structure of the system directories here does not correspond to the structure of the maven or ivy repository, and pom and jar files are usually named without the version in the name ls -l /usr/share/java/assertj-core/ total 1108 -rw-r--r-- 1 root root 1131359 june 9 2022 assertj-core.jar ls -l /usr/share/maven-poms/assertj-core/ total 12 -rw-r--r-- 1 root root 10457 june 9 2022 assertj-core.pom Thus, installing a maven or ivy repository will not do the trick, and flatDir cannot process pom files to obtain transitive dependencies.
v
Moreover, the structure of the system directories here does not correspond to the structure of the maven or ivy repository,
Ivy repos might have a default layout, the patterns to retrieve artifacts and metadata files are configurable, so you should be able to configure it.
j
there can only be 1 version of the library in the system
That's the same in Debian (though there are sometimes exceptions for major versions), but I still think that adding the dependency explicitly is going to be more brittle in the long term than getting the transitive dependency resolution to work. Anyway you are the maintainer 😉 so it's your call. In Debian we install the JARs with a version and add a symlink without the version, but you could also do the reverse.
Copy code
$ ls -l /usr/share/java/assertj-core*
-rw-r--r-- 1 root root 1373614 Oct 29  2024 /usr/share/java/assertj-core-3.26.3.jar
lrwxrwxrwx 1 root root      23 Oct 29  2024 /usr/share/java/assertj-core.jar -> assertj-core-3.26.3.jar
the patterns to retrieve artifacts and metadata files are configurable
I haven't tried this, but I would expect that not to work as Gradle won't find the Ivy metadata that isn't provided. But a 
flatDir
should work according to the doc:
To resolve a dependency, this resolver looks for one of the following files. It will return the first match it finds:
• …
• [artifact].[ext]
• …
... though with no metadata thus no transitive dependencies, which is not great.
v
I haven't tried this, but I would expect that not to work as Gradle won't find the Ivy metadata that isn't provided.
I didn't actually try that yet, but you can configure for a repository which kind of metadata it provides, so I would hope you can confiugre that it contains POM even for an Ivy repository.
Hm, no 😞
What you can do is to have an Ivy repository with only 
artifact()
metadata source. That would work similar to 
flatDir
then without the requirement that all artifacts are in the same directory. But still you would miss the metadata then of course. 😞
If the paths would individually follow Maven path conventions, you could also define a 
maven
repository that points to the directory where the POMs lie and add an artifact URL that points to the directory where the JARs lie, then it would work. But as far as I understood that is not the case either.
j
A third option (that I actually use in some cases) would be to have the packaging scripts generate a temporary Maven repository for the build, with symlinks to the system artifacts and metadata, and make the Gradle build use that repository.
👌 1
9 Views
Open in Slack
PreviousNext
Powered by