Published a blog post about testing gradle plugins...
# plugin-development
t
👀 4
👌 4
party gradlephant 2
d
Good read. You have great fixtures, some aligns with my view for a better testing with Gradle. I still believe the Gradle team should offer more test framework agnostic APIs to help testing.
t
curious what kinds of APIs you have in mind?
d
👋 nice writeup I started with similar approach of auto generating the build files from unit tests but as complexity started to get unwieldy I shifted towards defining integration test projects explicitly that are invoked from unit tests using
GradleRunner
that being said -> as the number of integration tests grow...
GradleRunner
is pretty slow... wondering if folks have some suggestions on how to speed things up
right now I'm thinking about using JUnit tags to group stuff together and run "basic" integration test in main build and whatever other groups in parallel afterwards (using actions)
t
it is much slower than unit tests by definition. I didn't include it in my post, but in the repo I have a larger build.gradle that includes parallel test execution
in the OSS plugin I maintain, I have a basic separation of tests by package, so folks can run a subset. It's very coarse-grained, just two, but it's been useful at times. (android vs jvm packages means java devs don't have to run the android tests)
t
wondering if folks have some suggestions on how to speed things up
Run tests in parallel, use minimum external dependencies in test projects.
t
funnily enough, one of my methods for keeping things fast is to write JVM fixtures whenever possible. Android fixtures are much slower
d
@Tapchicoma already doing that, it takes forever for GradleRunner to spin up a new instance (guessing it is starting a demon)
t
interesting, I see daemon for each test is reused 🤔
d
it looks like they are reused from within the same class but not across
or maybe I am imagining things 🤷
t
I think you are right, but you could run different test classes in parallel
d
yeah thats what I'm currently doing
it is still pretty slow -> hence the plan of moving some tests to separate step in the github workflow
hoping that would speed things up (as it currently takes 15+ minutes for the GradleRunner tests...)
t
they're slow, but I think they're worth it. More test classes => more parallelization
d
@tony
curious what kinds of APIs you have in mind?
I wrote a “improved” Gradle Runner in https://github.com/gradle-plugins/toolbox which try to bring some conveniences from the internal Gradle Executor in Gradle code base (https://github.com/gradle-plugins/toolbox/blob/main/subprojects/gradle-runner-kit-api/src/main/java/dev/gradleplugins/runnerkit/GradleRunner.java). There’s a bunch of conveniences as well as some better defaults like showing stack trace by default. One of the major goal was to treat running tests using TestKit or the Wrapper the same way. The result can be asserted via the same API. The parsing of the build result can also be compared and manipulated for better assertion (such as converting a verbose output into a “rich” output or extracting the output of a single task). This is very helpful when testing my samples or testing Gradle execution from within another tool (like Xcode or Visual Studio). These falls closely to you Builder and Truth support. There’s also a need to something similar to your AbstractProject. To me configuring a project for TestKit or for ProjectBuilder should be very similar. I still haven’t figure out how to do it but I have various idea floating around. I also see a lack of support for common unit test needs like accessing an
ObjectFactory
or
ProviderFactory
instance. I really like what I did with
ProjectTestUtils
(https://github.com/nokeedev/gradle-native/blob/7dc339b659a753c46349fd3a4e405447ce1[…]main/java/dev/nokee/internal/testing/util/ProjectTestUtils.java) which saved me load of work when I enabled parallel JUnit testing on some projects but I feel some things are not in the right place still.
t
I find it very challenging to use ProjectBuilder for unit tests. That's one reason I generally don't bother. I'll take a look at your improved runner, that sounds interesting facilities for easier project generation would be welcome. I myself have tried at least four times to write something generalizable. We have yet another we're working on internally, sort of along the lines of "java poet" or "kotlin poet"
d
Regarding slow testing, GradleRunner (aka TestKit) tests are orders of magnitude slower. In Nokee, I’m mostly focusing on integration/unit tests these days as the test time is like 45 minutes already. My general guideline goes as follow: • Unit test can use real
ObjectFactory
or
ProviderFactory
but should avoid anything that creates
Project
or apply plugins. • Integration test uses
Project
instance (via
ProjectBuilder
) and focus on the configuration phase. There is an unfortunate gap at this level where you can’t reliably test using
Settings
instance or
Project#afterEvaluate
. • Functional test uses TestKit and focus on the executions. That division should help move some tests into unit/integration testing and speed up test execution. It’s also easy to fall into the trap of testing too much which I have done in the pass. At one point I had parallel tests that serially would take 10 hours to run… that was a bit overkill.
I would be curious to see your four attempt to project generation. That would be good learning experience. I’m sure there are some good things in there.
another is the basic one in my blog post, and the fourth is an internal fixture that I can't easily share
👍 1