Published a blog post about testing gradle plugins https://dev.to/autonomousapps/gradle-all-the-way-down-testing-your-gradle-plugin-with-gradle-testkit-2hmc

Good read. You have great fixtures, some aligns with my view for a better testing with Gradle. I still believe the Gradle team should offer more test framework agnostic APIs to help testing.

curious what kinds of APIs you have in mind?

👋 nice writeup I started with similar approach of auto generating the build files from unit tests but as complexity started to get unwieldy I shifted towards defining integration test projects explicitly that are invoked from unit tests using

GradleRunner

it is much slower than unit tests by definition. I didn't include it in my post, but in the repo I have a larger build.gradle that includes parallel test execution

wondering if folks have some suggestions on how to speed things up

Run tests in parallel, use minimum external dependencies in test projects.

funnily enough, one of my methods for keeping things fast is to write JVM fixtures whenever possible. Android fixtures are much slower

@Tapchicoma already doing that, it takes forever for GradleRunner to spin up a new instance (guessing it is starting a demon)

interesting, I see daemon for each test is reused 🤔

it looks like they are reused from within the same class but not across

I think you are right, but you could run different test classes in parallel

yeah thats what I'm currently doing

they're slow, but I think they're worth it. More test classes => more parallelization

@tony

curious what kinds of APIs you have in mind?

I wrote a “improved” Gradle Runner in https://github.com/gradle-plugins/toolbox which try to bring some conveniences from the internal Gradle Executor in Gradle code base (https://github.com/gradle-plugins/toolbox/blob/main/subprojects/gradle-runner-kit-api/src/main/java/dev/gradleplugins/runnerkit/GradleRunner.java). There’s a bunch of conveniences as well as some better defaults like showing stack trace by default. One of the major goal was to treat running tests using TestKit or the Wrapper the same way. The result can be asserted via the same API. The parsing of the build result can also be compared and manipulated for better assertion (such as converting a verbose output into a “rich” output or extracting the output of a single task). This is very helpful when testing my samples or testing Gradle execution from within another tool (like Xcode or Visual Studio). These falls closely to you Builder and Truth support. There’s also a need to something similar to your AbstractProject. To me configuring a project for TestKit or for ProjectBuilder should be very similar. I still haven’t figure out how to do it but I have various idea floating around. I also see a lack of support for common unit test needs like accessing an

ObjectFactory

or

ProviderFactory

instance. I really like what I did with

ProjectTestUtils

(https://github.com/nokeedev/gradle-native/blob/7dc339b659a753c46349fd3a4e405447ce1[…]main/java/dev/nokee/internal/testing/util/ProjectTestUtils.java) which saved me load of work when I enabled parallel JUnit testing on some projects but I feel some things are not in the right place still.

I find it very challenging to use ProjectBuilder for unit tests. That's one reason I generally don't bother. I'll take a look at your improved runner, that sounds interesting facilities for easier project generation would be welcome. I myself have tried at least four times to write something generalizable. We have yet another we're working on internally, sort of along the lines of "java poet" or "kotlin poet"

Regarding slow testing, GradleRunner (aka TestKit) tests are orders of magnitude slower. In Nokee, I’m mostly focusing on integration/unit tests these days as the test time is like 45 minutes already. My general guideline goes as follow: • Unit test can use real

ObjectFactory

or

ProviderFactory

but should avoid anything that creates

Project

or apply plugins. • Integration test uses

Project

instance (via

ProjectBuilder

) and focus on the configuration phase. There is an unfortunate gap at this level where you can’t reliably test using

Settings

instance or

Project#afterEvaluate

. • Functional test uses TestKit and focus on the executions. That division should help move some tests into unit/integration testing and speed up test execution. It’s also easy to fall into the trap of testing too much which I have done in the pass. At one point I had parallel tests that serially would take 10 hours to run… that was a bit overkill.

you can see two of them at https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin/tree/main/src/functionalTest/groovy/com/autonomousapps