This message was deleted.

Sample build scan: https://scans.gradle.com/s/lryay23kvqxyo/timeline

workers can run in parallel, and tasks in separate projects can run in parallel, but tasks within the same project do not run in parallel unless configuration cache is enabled which prevents tasks from affecting each other's states

Is there a reason why the

Sync

task doesn’t use a worker? Unfortunately I’m using a plugin that is not yet compatible with the configuration cache, so that’s not an option for me.

Setting up the worker machinery costs time as well, not everything benefits from parallelism. Especially I/O doesn't since your hardware (in this case disk) is working sequentially, concurrency is what you'd want (async I/O). Obviously for that the tasks need to be executed at the same time too, but not in dedicated threads. I don't know if that's done by Gradle. 🤔 That said, a task needs to explicitly opt-in to use workers and task authors need to ensure that it actually leads to improved performance and not just book keeping overhead.

Looking into this further, something is not quite right… I’m using

Sync

to expand a

.tar.gz

archive. Using

tar

on the command line, this takes 3-4 seconds. Using Gradle’s

Sync

takes close to 90s. Any idea what might be causing this?

It’s not a fair comparison because Gradle is doing more than just unpacking. Still, the difference seems extreme. 🤔 What if you use a plain task and just unpack in the

doLast

?

you could

Copy

instead of

Sync

, which would leave out some of the processing. but still, 2x or 3x would be believable, 30x feels unreasonable. I assume you're doing something like

sync {
    from(tarTree("..."))
    into("...")
}

? I wonder if Gradle is expanding the tarTree multiple times for some reason… that would be unfortunate 😞

tl;dr: running

tar xf …

and using

doLast { sync { ... } }

produce similar results (~3s on my machine), using

Sync

takes ~86s. I’ve created a sample project that demonstrates the issue: https://github.com/charleskorn/gradle-extract-performance-issue Here’s a sample build scan: https://scans.gradle.com/s/7sroga3pjcv46

I tried it on my Mac and it’s 6s for

extractWithSync

(cca. same as all other options on my machine) and after that it’s instant since it’s up to date. Maybe you can also try with

doNotTrackState("")

for

Sync

task, but in that case task it will never be

up-to-date

. You can also try to run it with

--no-watch-fs

but I doubt it will help in that case. Do you have M1 Mac or some special disk setup?

no issues here either. 2.5s for tar, 3.7s for sync. I expect Gradle is slower because

tarTree

goes through

build/tmp/expandedArchives/**

before

Sync

picks it up, but it's within a factor of 2x which is not surprising. possibly you have some sort of virus scanning software on your machine that is causing very slow I/O?

Maybe you can also try with

doNotTrackState("")

for

Sync

task

This significantly improves things for me -

Sync

now runs in roughly the same time as the other options for a clean build (ie. empty destination directory) - scan. However, on an unclean build, it still takes just as long (as does the

doLast

implementation) - scan.

Do you have M1 Mac or some special disk setup?

Yep, I’m using an M1 Mac.

possibly you have some sort of virus scanning software on your machine that is causing very slow I/O?

This looks like it could be the culprit - temporarily disabling the anti-virus software on my machine results in the same fast speed for both clean and unclean builds. Working with it disabled isn’t really an option though - is there anything I can do to mitigate this from the Gradle side? (apart from praying to the anti-virus gods that we move away from Sophos soon 😄)

nope, antivirus sucks, it'll slow down other parts of your build too. at least as far as we've found, the only way to actually see reasonable performance that scales with the hardware is to disable it