https://gradle.com/ logo
Join Slack
Powered by
This message was deleted.
# community-support
s
This message was deleted.
c
the symlinks would be transparent to Gradle. Could this be modelled as standard inputs (files, directories, configuration that, when changed, triggers the task) and outputs (directories, files that, when different than expected, triggers the task)?
n
yes, but the problem I've come across is that since Gradle inspects the actual files, Gradle takes a long time to check each node_modules. Our build contains several different node_modules that each can take 20 seconds to check
I haven't tested with symlinked directories (the way pnpm does it) and only have a small test project where I modified a file linked to by a symlink
and as soon as I change the linked file, Gradle considers the task no longer up to date
c
ok. two options come to mind: a) task has a FileCollection of @OutputFiles, where each entry is one of the symlinks; b) 
outputs.upToDateWhen {}
that returns true when all expected symlinks exist.
n
I'm unsure if I understand 
a
. The 
pnpm install
task will be the one creating all the files, so I don't know which files will be created by the 
pnpm install
task. I don't understand how I can avoid having 
outputs.dir("node_modules")
?
Will having them as 
output.files()
change the behaviour of the symlink logic in Gradle's up to date check?
c
if the criteria is ‘does symlink exist’, and they’re dynamic (not known ahead of time), you could create a custom filecollection that only includes symlinks, using that as @OutputFiles.
n
"Symlink exists and points to path X" is the full check needed
Since e.g. 
@types/bar
differs when it's linked to 
/home/user/.pnpm/@types/bar/0.0.1
vs 
/home/user/.pnpm/@types/bar/0.0.2
c
hmmm. that’s tricky. Presumably changing from X -> Y would dirty the build. Don’t think Gradle’s built-in stuff will work there, as it doesn’t have first-class support for symlinks. However, you could create a custom object representing a symlink (name, target), a Provider that resolves all the symlinks in <wherever that is>, and use a list of those as @Input to the task.
n
That sounds like a good idea, but how can I avoid 
pnpm run build
getting triggered since Gradle will believe that the output of 
pnpm install
has changed? Do the same thing with the outputs (i.e. custom provider) ? Never used a custom provider with files, so I'm not sure about the pitfalls
hmmm.. nvm, I think I get it. I just set the output of 
pnpm install
to be a guard file with the hash or something of all the symlinks and their targets
Thanks a lot! I'll try this tomorrow (getting late here)
👍 1
t
If 
pnpm install
is really fast, then maybe you could do it in the same task as 
pnpm run build
, and then use the 
pnpm-lock.yaml
as input rather than the 
node_modules
.
n
It is faster than checking up to date. Using pnpm-lock.yaml is a really good idea! Since I always run 
pnpm install
(as opposed to how it used to be with 
npm ci
), that should always be correct. Thanks!
Throwing the code I wrote and replacing with just a simple 
input.file()
. I like this
Just a caveat to look out for anyone following in my mud tracks: I will need to investigate what happens when using pnpm workspace dependencies. The 
pnpm-lock.yaml
will probably not be updated when an upstream workspace dependency is changed, thus not triggering the build of this subproject. I will most likely have to do 
tasks.named("pnpmBuild") { inputs.dir("../../other-project/src/main/javascript") }
to get this to work reliably 😞
.. or maybe 
other-project
pnpmBuild
-task will export a consumable configuration (non-cached) with the javascript output just so I can have that as input for the 
pnpmInstall
-task
d
From experience with the gradle-node-plugin and some larger projects I recommend just letting install do its own thing not caching it or even tracking its state. Anything after that, add 
packages.json
as an input and the install task as a dependency and reap the benefits of caching
n
We're actually using gradle-node-plugin as well here. When moving to pnpm, we will still use it but basically just to use the npmSetup task (for installing npm). I don't see how just looking at "package.json" in the build is not good enough to be cache'able? Like I wrote above, even with 
pnpm-lock.yaml
I assume (until I've tested it) that pnpm workspaces will not update neither 
pnpm-lock.yaml
nor 
package.json
when just a workspace dependency changes.
Ok, progress report: Created something that checks if 
pnpm install
is necessary by checking the correct 
importers
section of the pnpm-lock.yaml, because 
pnpm install
takes about 40 secs. For some reason it's much faster when doing every repo by itself compared to all together. Probably something O(N²) in there somewhere with dependencies.
d
@nisse is that something general enough to be relevant for other builds? If so it would be awesome if it could be contributed to gradle-node-plugin, we just merged a PR adding pnpm support which is going into the next version
n
Let me try the next version first? Do you have an guesstimate on when it's going to be released? (I won't hold you to it)
d
I need to do some more work on the tests (since the entire test suite is now too heavy for GitHub actions) so essentially When I Have The Time™ which I hope is during this week or during the weekend at the latest. That is if I don't find any issues with it, but it's pretty straight-forward so it should be easy to look into. But if you want to test it out early you can clone the repository and use composite builds
n
Also, the javascript developers insisted on having it as one task (ie it taking 40 seconds), so I have reverted it to not look in the pnpm-lock, instead just creating a pnpm task in rootProject which is depended upon by the different javascript projects
It's ok to call us crazy and that we're doing a stupid design, but we are working on making this fairly large monorepo which will have multiple docker images and k8s yaml files pop out of it with multiple node and frontends(html+js) So we will build multiple independent javascript projects where some will require input from Gradle in the form of e.g. .proto-files and graphql-schemas. The best way I've found to handle it is to implement a rootProject pnpmInstall task that Npx/Pnpm tasks will depend on. so a run would for project1frontend:assemble would look something like
Copy code
:nodeSetup
:npmSetup
:pnpmSetup
:pnpmInstall
:project1:frontend:nodeSetup
:project1:frontend:npmSetup
:project1:frontend:pnpm_build
:project1:frotnend:assemble
What we have now "works" even though ofc the npmSetups aren't that nice, and it's on the list to replace 
npx
with 
pnpm exec
to avoid them
d
When I started with gradle-node-plugin I worked at a place where each of our npm installs took 30 seconds and we had 4 of them on account of a mono-repo-ish layout (luckily running node projects in parallel) Normally I'd advocate splitting things into different projects and setup dependencies between them, but each project requiring a 
(p)npmInstall
is going to incur a cost, at least until I get around to finishing the build service/worker api stuff
n
the pnpmInstall cost per project was very low according to my unscientific feeling, especially since you can run 
pnpmInstall
parallelized if you run it per project with Gradle. It also allows for better granularity so that I don't need to pnpm install in the project that I don't care about right now. Because even if pnpm just symlinks, any dependency only available in those projects are obviously downloaded only if those projects are pnpmInstall'ed. But meh, moot point. CI will take 30 seconds per job longer and they'll be happy. As a bonus I can rub their nose in it every time they complain about build times. 😉 Just to be clear, I'm talking about setting the flag in the root .npmrc 
recursive-install=true
Looking at your source, it seems that our setup would be supported by the pnpm task out-of-box, without duplicating any functionality, so yay! 🎉 Especially like that PnpmExecRunner doesn't depend on 
pnpmInstall
👍 I would've like to have an option for --frozen-lockfile for 
pnpmInstall
, but that's something that isn't available in the plugin's 
yarnInstall
either (only available for 
npmInstall
via 
node { npmInstallCommand = "ci" }
), so that's nothing new.
d
It might be worth noting that 
PnpmExecRunner
is the "Run something through pnpm" and not necessarily a 
pnpm exec
, it's the core functionality that wraps Gradle's exec And unfortunately, unlike 
npx
and 
pnpm dlx
I think 
pnpm exec
would depend on 
pnpmInstall
since it runs something from the context of 
node_modules
, but 
pnpm exec
could probably easily be implemented through a 
PnpmTask
n
The problem is that we won't nor can't use the 
:project1:frontend:pnpmInstall
, we must depend on the root 
:pnpmInstall
. But I agree, a 
pnpmExec {}
construct that does "the right thing" would be very easy to implement for us, especially as we already have that 😃
I'm going to take a short break and then try to replace the plugin with the snapshot version. Never used composite builds for plugins, but should be fairly easy
I was wrong. Of course. 
Cannot include build 'gradle-node-plugin' in build 'buildSrc'. This is not supported yet.
d
that is a bit trickier, if you're ok with a dirty working directory and you're using kotlin I think you can just copy the entire plugin source-code in there and add the two dependencies to your build-file 😁
Or you have a perfect opportunity to migrate from 
buildSrc
to a composite build 😉
n
Nah, I just added 
apply(plugin = "org.gradle.maven-publish")
and the run 
./gradlew publishToMavenLocal
👍 1
We're moving away from a plugin to buildSrc because of a nasty bug in the kotlin compiler that increases our build times and makes the build output useless
Hmm.. is it possible to get PnpmSetupTask to stfu? I used to do the following:
Copy code
tasks
    .withType(PnpmTask::class.java)
    .configureEach {
        execOverrides = {
            standardOutput = LogOutputStream(<http://LogLevel.INFO|LogLevel.INFO>, logger)
            errorOutput = LogOutputStream(LogLevel.ERROR, logger)
        }
    }

tasks.withType(PnpmSetupTask::class.java)
    .configureEach {
        execOverrides = {
            standardOutput = NullOutputStream.NULL_OUTPUT_STREAM
        }
    }
FYI: 
--silent
seems to do the trick, and everything works well with new plugin. 👍
d
@nisse 3.4.0 is released and the only thing added from 3.3.0 is the pnpm support
n
💥 🎉
4 Views
Open in Slack
PreviousNext
Powered by