Hi there, How are you handling test "matrixes" ? Currently I have seen and practiced different approaches, which are not mutually exclusive. 1. Using properties to modify something, one need to pass the

-Pxxx

to the cli, this is quite flexible, this is ok for CI pipelines, workflows, but it's discoverability is not great locally. And clearly don't handle well the idea of running a matrix. 2. Using tasks, (eventually with an aggregating task) it requires to registering each task (this can always be done programmatically, e.g. via plugin) 3. Using test suites, which also leads to test tasks. The challenge is balancing discoverability, local execution speed (because some runs might be too long for feedback loop), local availability of the required dependencies (some can be only available in CI env), ability to run/exclude a single "matrix config" locally. Things I've had to deal with in regard of matrix, different dependency versions, different toolchain, different "properties" that could affect the exercised code (e.g. via a file on disk, a system property, jvm arg etc.)

I'm personally using properties: I don't want to run the whole matrix on my machine, CI is there for that; but I can easily run one particular combination when needed (test conditional branches in code; reproduce and debug test failure on CI; etc.) My use case is limited to Gradle plugins though, testing against various Gradle versions and JDKs. For that use case, if you want to easily run the full matrix, then have a look at gradletest.ysb33r.org

thx

Imho, most variants should better be separate variants, not project properties. It's better for up-to-dateability of tasks. If a task for example is not cacheable and has a project property as input, then if you set the property to A, then run, then to B, then run, then set to A then run, then set to B then run, the task is run 4 times. If it is separate tasks, then it is run 2 times because the 3rd and 4th run are just up-to-date. As a rough general rule, if there is a finite amount of options, I'd prefer separate configured tasks. Project properties are better suited for "free-form parameters". So if you for example want to test with different runtime environments, or run tests with a different setup but same classes, like running differently tagged tests, then add targets to a test suite. If it is about different dependencies but same code, either also multiple targets and the tasks configured, or maybe different test suites. a.s.o

Yeah like separate suites/tasks, but they don't always run so you can still run the builds in parallel in CI

If you mention variants, that also mean that it should be "encoded" as tasks. Those can be easily added to some aggregating tasks. The issue with a big matrix is the number of tasks. > So if you for example want to test with different runtime environments, or run tests with a different setup but same classes, like running differently tagged tests, then add targets to a test suite. If it is about different dependencies but same code, either also multiple targets and the tasks configured, or maybe different test suites. a.s.o Part of the challenge comes when you combine these. I.e. having multiple dependency versions, multiple JDKs, multiple "setup properties". For the same compiled test

What is problematic about the "number of tasks"? As long as you leverage task-configuration avoidance, they should not generate much overhead.

As long as you leverage task-configuration avoidance, they should not generate much overhead.

That's mostly true, but this harder to track with the number of tasks, in my current project I have more than 33k.

Well, like always, the truth is a tradeoff. If you want the advantages of multiple tasks, you have the disadvantages of multiple tasks. If you want to avoid the disadvantages of multiple tasks, you also loose the advantages of multiple tasks. Which way you go depends on your exact situation and preference. 🙂

Yeah