Hi everyone, I have a philosophical question about testing strategy — specifically, how many tests are useful, and when it becomes too much 😅 In

nf-core/phaseimpute

, our subworkflow uses several channels, some of which are optional. At the moment, we have a minimal test (single input, one chromosome, only the required channels), and a maximal test (multiple inputs, two chromosomes, all channels enabled). I’ve recently added a new optional channel, and now we’re wondering whether we should duplicate the existing tests — one version with the channel and one without — to ensure backward compatibility. The downside is that the number of tests would grow quickly, and I’m concerned about gradually increasing the load on GitHub Actions. Are there any guidelines for designing good unit tests for subworkflows? Thanks!

I think people are doing actual thesis on this

Basically you should strike a balance between test coverage and how much time it takes to run the tests. Ideally, you should be able to run your tests under just a few minutes so that you can quickly evaluate if you've broken anything after a change. If your tests take longer than that, the developer's experience quickly degrades as it's not possible anymore to quickly iterate. With Nextflow this is maybe even more important because the error messages are often quite obscure (even though this is getting better). However, writing fast tests is not always easy since there is so much overhead for booting up the whole pipeline: even with stubs it takes around 40s to run the pixelator pipeline, which is way too slow. What we ended up doing is tagging some tests with good coverage with a

smoke_test

tag (eg tests that use all the optional channels and parameters) and we wrote a couple scripts to easily run only these tests, so that developers don't need to wonder too much what to run and how. This makes it easy to run quick diagnostics. Another trick you can do is provide scripts to shard or parallelize the tests locally, to make it easier and faster to run everything. Finally you can also skip the slowest tests for regular PRs and only run them for releases. Alternatively, assuming your modules and the software inside are well tested , you can run most of your subworkflow tests with stubs for regular PRs.

Thanks for the help. We currently are at 5min of test when merging to the dev. So I think we shouldn't add too much new test.

The subworkflow will eventually be migrated to nf-core/modules. This means that the test is technically going to be run just once (when changes are made upstream, in modules). Not everytime you make changes in the pipeline, as nf-tests from nf-core modules and subworkflows are excluded from the whole testing in CI. Since it is going to be in nf-core modules, and potentially available to other pipelines, I think the coverage should be high and also strive to demonstrate backwards compatibility by keeping the previous test and also add new ones if we are trying to add a new functionality.

pushing code to modules for unit tests is definitively a solution