I have a task that downloads a large file (basically required assets for a subsequent JavaExec task). During development, I test by publishing my plugin to my local Maven repo and pulling it in my test project, but I don't change the version. Despite setting path sensitivity for the large file task input (as a regularfileproperty), every plugin change results in the task cache being busted and that large file being re-downloaded, despite it not moving or changing at all. Is there a method of development I'm missing that avoids this?

Wait, isn't your path a task output?

Yes, it is - the download task outputs a file property which is wired to the file property of the next task

What are the inputs of your download task?

https://github.com/hygradle/hygradle/blob/trunk/src/main/kotlin/dev/hygradle/internal/task/DownloadAssets.kt#L26 , the only inputs with any cache bearing are version and patchline and they don't change between runs

every plugin change

what do you mean by taht?

That gets wired to https://github.com/hygradle/hygradle/blob/trunk/src/main/kotlin/dev/hygradle/internal/task/ExtractAssets.kt to extract from the zip bundle, and then wired to https://github.com/hygradle/hygradle/blob/trunk/src/main/kotlin/dev/hygradle/internal/task/run/RunHytaleServer.kt

It's probably the plugin change

what do you mean by taht?

When I make a change to the plugin source, publish to local maven, and then sync again in my consumer project

No real way that I know about. You can track https://github.com/gradle/gradle/issues/31482

Yikes, fair enough. Do you do any further checking beyond if the file exists, or is the file path/name pinned to something deterministic from the R8 version?

The R8 version is the git sha so I'm relying on no collision there

The task implementation classpath is always an input to the task too, because if the classpath changed, the implementation and thus behavior could have changed and thus the result could be different, so the task needs to be rerun. If you know your output is in place and up-to-date, you could for example use

-x yourTask

to skip the task explicitly.

So if I made any change to the plugin source, even if it doesn't touch the task, it'll bust the cache? So even between release versions of the plugin this will happen for users

The task implementation classpath is always an input to the task too, because if the classpath changed, the implementation and thus behavior could have changed and thus the result could be different

I'd love to be able to opt-out of this. Most of my tasks implementations delegate to another classloader. The build script classpath has no importance at all in the final outcome.

The task should probably have an outputDir and remove the stale files or else the directory will grow forever but oh well...

This doesn't mean I actually have to consume that directory in any dependent tasks, right? I can just clean the dir of any not read in x days or whatever archives

So if I made any change to the plugin source, even if it doesn't touch the task, it'll bust the cache? So even between release versions of the plugin this will happen for users

Even if some totally unrelated plugin changes, or the user changes some convention plugin. Another plugin or a user convention plugin could for example also add a

doLast

action to your task and so on. If the classpath changed, Gradle simply cannot know whether the output might eventually be up-to-date.

This doesn't mean I actually have to consume that directory

If you want to be pedantic, you can probably transform the directory to a

Provider<File>

before passing it to the next task. But I would probably just keep growing the directory ^^

If you want to be pedantic, you can probably transform the directory to a

Provider<File>

before passing it to the next task. But I would probably just keep growing the directory ^^

Oh, you mean you'd use an output dir instead of a file?

Yes, the output of the download task could be a directory containing a single file and removing all the stale ones

I suppose I could just use

filecollection.singleFile

to assert that in my consumer task too

Yep

Fair 😄 I might just do naive download avoidance for now then. If the release/patchine hasn't changed, and an archive exists in that dir named

<release-<patchline>.zip

, avoid downloading

It's complicated. I like to keep directories for outputs and always use

InputFiles

for inputs

Be aware that you loose task dependencies if you are not careful. Besides that, if you have a configurable file collection and want the actual files, use

asFileTree

on it when you use it, then it resolves to the actual files.

+1 I mostly use

asFileTree

Makes sense, thank you both! Is

asSingleFile

fine as well if I'm expecting there to only be one file, or should I be using filetree filter methods?

If you know it is exactly one file and also want a hard error if your assumption is wrong, the methods seems a fit

Is there a way to provide lazy providers to filetree filter specs? i.e. if I want to match/include on the result of 2 properties in a task combined into a filename for example

Does the http server support etags? You could store the etag when downloading the file, and attach it to the http request. Then it matters less if the task re-runs - the server will respond with "up to date" https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/ETag

No, it doesn't sadly. Naive avoidance is the best I've got :( @Martin Quick question, you mentioned early I could be pedantic and transform the output directory into a file provider when I pass it on. What's the best way to do that? Should I be passing it into a RegularFileProperty?

@Remi Gelinas I did it there: https://github.com/GradleUp/gr8/pull/45

Does Gradle recognize the output file between runs? I've got it implemented almost exactly the same, but as I mentioned ^ it doesn't realize that it was responsible for the single file in that dir

@Remi Gelinas I think so. But don't trust me on this, try it out yourself in the gr8 repo:

./gradlew -p test-plugin :gr8DefaultDownload 

[...]

./gradlew -p test-plugin :gr8DefaultDownload -i

Skipping task ':gr8DefaultDownload' as it is up-to-date.