This message was deleted.

More context: We have a 16 cores agent running our CI/CD builds for our project with over 6000 unit tests spread non-evenly in over 60 modules. Due modularization we are able to build everything with paralellization at its most. Still we wanted to try forking the tests execution inside each module expecting some improvements in the builds due better opportunities to parallelize work. However, the impact is terribly worst. I am aware of the overhead of spinning up new jvm processes, but I would never expect the values we are getting. When running testUnitTest or the jacoco task for the whole project we get these numbers: - Current status (just 1 process per module): 17min - Enable forking only for the 12 slowest modules (those taking over 2min of execution): 24min - Enable forking for all modules: 36min As advanced in the first message, I noticed the fingerprinting skyrockets when forking is enabled: from 4min reported without forking to 34min when enabling forking in just the twelve modules. My current guess is that the fingerprinting somehow starts when the first test process finishes and gets hung until all the other test processes are completed. But still this is something happening in parallel and won’t explain the huge total increase in build time… Unless the worker initiating the module task that forks the test processes gets blocked idle until all tests are completed. Might this be the reason? It’s the only hypothesis I have so far. But I am not fully convinced. Also note that when running an individual module locally, it improves from 3min to 2min when enabling forking. Any clues or any ideas about what else to look? Thanks in advance! 🙂

Perhaps the tests are generating copious amounts of temporary files in a location that is getting picked up by the fingerprinting?

Interesting, but wouldn’t that effect be the same without forking then? 🤔

Are you able to share a Build Scan where you noticed this behavior?

perhaps the forking requires snapshotting that set of files

n

times.

Hi @Eric Kolotyluk Our build scans are private in our organization, but I can share some scan dumps with you privately if you can look into it 🙂

I can't guarantee I will find anything, but if you feel comfortable sharing a scan dump with me privately, I will try and take a look.

Sure! I understand it. I am fine sharing some dumps with you. Thank you very much for the help 🙂

You might also have a look with the

gradle-profiler

to maybe find more evidence