Getting back to a topic we discussed here before (@wolfs and others). Context: • https://www.linen.dev/s/gradle-community/t/26748062/i-am-exploring-solutions-for-the-following-scenario-and-look • https://github.com/gradle/gradle/issues/32225 We have a solution in place now for tool installaion that uses

FileSystemAccess

for snapshotting. Last week we noticed that it takes 16m on one developers Windows machine (8m on my AWS Workspace) to do the snapshot of a particular tool. Turns out, the tool consists of 16k files. So this is probably a general Gradle snapshotting question/issue for cases where you have an insane ammout of files: One developer spiked to replace what Gradle is doing with a (simple) custom solution based on what they did in their previous custom build system. It's not really usable as it naively snapshots each file directly and does not make any use of the "snapshot caching" and VFS of Gradle which breaks the UP-TO-DATE case. But what they did is using

Files.walk(rootPath).parallel()...

to process files in parallel, which significantly improved the snapshot creation (first use case) for the 16k files. If I read the Gradle code correctly, files are always processed sequentially (see DirectorySnapshotter). I wonder if there is a particular reason for not using some parallism here. I could not find anything on this topic in the GH issues. I wonder if this ever came up before and if there are any thoughts on this. We are in a situation where it is hard for (certain) developers to understand why the new system is better than the old one if it appears so much slower. It is of course much better in the incremental case, but the "first use" case is what makes a bad impression on some. (I personally also think it is crazy that we have a "tool" consisting of 16k files - but legacy...)

That snapshotting maaaany files is awefully slow on Windows sometimes is indeed a known problem. Due to that

org.gradle.java.compile-classpath-packaging

was introduced, so that even with

java-library

plugin the JAR is used for consumer projects and not the single files to mitigate that problem in that specific case. See https://docs.gradle.org/current/userguide/java_library_plugin.html#sub:java_library_known_issues_windows_performance

fwiw I also tried to create a really fast parallel dir checksummer. Iirc it was possible to get a bit more speed by batching the files, so each file didn't get checksummed individually. Instead, create batches of 10 or so files and checksum each batch in parallel.

Thank you both for your thoughts.

slow on Windows

I was assuming that it may be a Windows specific issue but have not given it much thought. But yes, now I also remember the classpath topic. This is important information I will pass on. 👍

snapshot the tool as one archive

Interesting idea. Unfortunately, this won't help here I think, because the idea is to check that the tool installation has not been modified since the tool was extracted. One thought I now have is that the snapshot is not needed directly after the tool has been downloaded (in the beginning of task execution). Maybe we can do the snapshotting in parallel to running the tool the first time.

like sources, docs, or license files?

Yes that is the case. A lot of stuff that is not actually "the tool". I just realized that after I was typing the above. I did not get feedback yet, but I suggested exactly that (exclude the docs) and I think it will speed up things for this particular case.

Definitely worth investigating. I'd make an issue, and a PR. My guess is processing that many files wasn't considered.

Thanks for pointing out the

UnzipTransform

. Might come in handy in the future. If we still would use Transforms in this project, it would not be enough though as we also need to extract (for the time being)

7z

files and have files that (for the time being) are loaded from a network drive which is difficult to use through dependency management as

file:

repositories are handled different wrt. caching.

PathVisitor

is definitely not ready to be used with a parallel stream. It relies on the order in which files are being visited. Though for a certain layer (aka one directory), it would be fine to process the files in parallel. I think the basic idea was that this is IO bound, so it doesn't help doing that in parallel anyway, though your comment seems to indicate that this is different at least for Windows.

I think parallel snapshotting would help when there is a lot of files (or maybe even for snapshotting big jars), so I think it's good idea in general. We had a short discussion here and there about it when there was a snapshotting performance issue. But since such issue appears only here and there and because implementation requires non-trivial effort to try and provide evidence of its benefit, we never really investigated it.

Thank you both for the insights. Do you think "parallel processing on each layer" is something I could do as a experiment in a reasonable amount of time or does "non-trivial effort" mean that you think this is something that would require a larger restructuring/redesign? I don't know that part of the code base, so I have no good feeling for it. We would be interested in trying/spiking it to do some measurements - even if the implementation initially is not "completely correct" (whatever that means). Then we can test it with the use case we have where we see a large performance difference between Gradle's implementation and our own primitive parallel implementation (see my initial post). And then we can see if something useful comes out of that.

Directory snapshotting is pretty much contained in this class: DirectorySnapshotter.java#L126-L128. I think it wouldn't need much restructuring/redesign if you do parallel snapshotting on a file level, aka you snapshot files in parallel in one directory as Stefan suggested. It would be great that final DirectorySnapshot returns the same order of files as it would without parallelism though.

Do you think "parallel processing on each layer" is something I could do as a experiment in a reasonable amount of time?

From that I think the answer is yes

@Anze Sodja this is what I got after spending a day digging into

DirectorySnapshotter

: https://github.com/gradle/gradle/pull/37305 Maybe you can run that through your pipeline to see if this is a direction you may consider.

Thanks! I started CI tests to see what is the effect of this. Did you try on any realtime project and do you see any performance benefit?

Yes. For the concrete example where we discovered that this is a problem for us, I have a ~50% speedup. On the machine I have, that took ~430s without the change and now takes around ~210s.

So this implementation parallelizes per directory, not per file, right?

Yes. I looked in

DirectorySnapshoter

in isolation. "snapshoting a directory" is already a isolated operation. My thought was that that should be paraliasable then without changing fundamental behavior. And that it possibly has the greater effect as in large structures it is likely to have many directories.

Yeah, I guess we can't use parallel snapshots when filters are set, since they are supplied by users and are not thread-safe. I don't think it's a bug in the plugin, it's just that contract of filters is very loose

Thanks @Anze Sodja for taking the time to look at this. I was afraid that numbers would regress in certain scenarios. Maybe due to more memory required by the parallel processing (just a guess). As per your comment, I assume there ist nothing I can do effectively right now from the outside. But let me know if you think there is something. Should I create an issue for this to track it better?

I think opening an issue and describing problems you see would be a good start 👍 Could you maybe also try my implementation: https://github.com/gradle/gradle/pull/37311/? It's a complete rewrite and its also less aggressive in context of creating ForkJointPools, and it seems it doesn't regress other scenarios. It also avoid parallel dir traversal when there is a predicate (so there is no issue with Kotlin plugin). But it also parallelizes file hashing. I wasn't able to reproduce high snapshotting times on Mac with provided zip unfortunately, and so if you could try it on Windows that would help.

Yes of course. I will give it a try and report back. Would be great if there is a realistic way forward with that one !

You could snapshot in parallel, while synchronizing when using the filter, right? Maybe in a follow-up PR.

That makes a lot of sense, I think that is great idea 👍