Maybe thats interesting for that SIG: I have added to nf-core/ampliseq a sort-of-benchmarking (not released yet, but soon). I implemented scripts for a comparison of computed to expected outcomes, as described here, added (down-sampled) mock community data as test dataset, and use a (CI) test profile to ensure that future code doesnt degrade performance. I would like to use mock data & comparison to expected with all test data, but thats sort of for the future. I havent implemented the comparison to expected for that case solely, but its a nice side effect, I think.

Thanks for sharing, this is really interesting for the SIG! Comparing outputs to expected results on mock community data, and running it as a CI test so performance can't quietly drop, is exactly the kind of practice we want to encourage. One idea I have for the Benchmarking SIG is a blog, or a similar space, where people across the community can share approaches like yours. Your work could motivate other pipelines to do the same, and parts of it might even be reusable. I'm planning to discuss this at the Barcelona hackathon, and I'd love to use your ampliseq setup as an example to support the idea. Would that be OK with you?

and parts of it might even be reusable.

I am not sure its indeed reusable, because its quite specific to that pipeline. But there might be overlaps with other pipelines for nf-co.re/ampliseq/dev/parameters#… or nf-co.re/ampliseq/dev/parameters#…. Theoreticaly would be nice to have a consensus evaluation.

I'm planning to discuss this at the Barcelona hackathon, and I'd love to use your ampliseq setup as an example to support the idea. Would that be OK with you?

Sure