What are the best practices about creating a task that will do something based on a remote response? Can it be cacheable?

Totally up to you. But typically not really if it depends on external state.

Depends? If your remote response is maven central it is 😄

No, it would be a API that returns a Json, so it is a external state

If you trust that url to be immutable and always return the same thing, I don't see a reason to not make the task cacehable

You could have a separate task that gets the Json and is always done and then have the real task cacheable depending on that task's output file.

> If you trust that url to be immutable and always return the same thing Indeed, no, the url will be always the same but the response will change some times

Then agree with vampire. Some task need to be never cacheable

It should be interesting to have some

RemoteTask

abstract class from Gradle that allows this in an easy way.

You could have a separate task that gets the Json and is always done and then have the real task cacheable depending on that task's output file.

So, depending on what will do the real task, it would be worth or not having two task or just one and execute it always. The real use case is generating a Kotlin file inside the Kotlin source directory based on that json that can change even the URL being the same every time.

You should not generate the file in the source directory anyway, but generate it to some dedicated output directory, configuring the generation task as

srcDir

for the respective Kotlin source directory set.

@Javi we have a similar thing in Apollo were we have a task that downloads a "maybe changing" schema. That task is not part of the

build

target. It's only provided as a convenience. We then tell users to commit the schema to make sure their builds are reproducible

Actually you could probably also use a

TextResource

from URI to get the remote file, but that might then not work too good and also download the file when not necessary at all

They typically do this in a nightly GitHub action

Or that, yes, depending on whether the build shoulds always be reproducible or always use the latest state

> You should not generate the file in the source directory anyway, but generate it to some dedicated output directory, configuring the generation task as srcDir for the respective Kotlin source directory set. Yeah, it will be something like

kotlin.srcDir(myTask)

. > But regarding a dedicated RemoteTask I don't think that makes too much sense, especially as it is easy to do already. You can use any task to do the remote operation an have the result as output and then use that as input for the other task. The problem is, IMO, it is not intuitive you need two tasks for this, people can struggle with this. > @Javi we have a similar thing in Apollo were we have a task that downloads a "maybe changing" schema. That task is not part of the build target. It's only provided as a convenience. We then tell users to commit the schema to make sure their builds are reproducible Currently we are using a script that is executed manually, so it would be a similar case.

I think there's an issue for a built-in download task

Which would not change his last argument, as it would still need two tasks 😄

I was thinking the task action would be able to detect if the input has not changed in order to be executed again or not to avoid the need of two tasks, as currently to avoid it we would need to do that manually.

I was thinking the task action would be able to detect if the input has not changed in order to be executed again or not

Your input is the state of the internet

I know, I mean it needs to be two tasks, but maybe it could be only one if it is a special task that even being one it can behave as it was two. Anyway this would be out of scope for Gradle for a lot of time.

What you want is maybe not a separate task, but a

@RemoteInputFile

input property annotation, that will cause the file to be downloaded and then used as input. 🤷‍♂️

Exactly that indeed

Feel free to open a FR or PR and see what the Gradle folks think about it 🙂

Created: • https://github.com/gradle/gradle/issues/32014

or if the file download happens in a value source, and you use that as input to your task, then the task can be cacheable :p

But that would force to download always the file, no? Even if the task is not executed? Wouldn't be a problem with config cache?

if it's a lazy property and the task isn't configured (isn't in the task graph) then it won't be downloaded

FWIW, what I've been doing when applicable is to do a HEAD request and use the Etag header as a task input and the task action is doing the GET and processing. You pay an extra cost for the HEAD request during task input snapshotting. It's worth it if either the GET request or the processing takes time. For CC compatibility, a

ValueSource

doing the HEAD/Etag works.

But yeah, when using a value source to download the file, it would always be downloaded to check whether the CC entry can be reused or not and if it cannot, it will be downloaded a second time unless that downloading does some caching, so be careful when using a value source.

I want to bring attention to a plugin that provides a download task/action https://github.com/michel-kraemer/gradle-download-task

How is it different from the undercouch download plugin which is quite popular for the same use-case?

Isn't it the same? I followed the link for it

Ah, yes it is. I thought it was under

undercouch

, not

michel-kraemer

. And additionally you announced it as a new contribution to the thread while we already talked about that plugin above, but you probably missed that. 🙂

My bad, missed it!