Sorry, I don't know why it didn't work 👇🏻

Can someone invite Johannes to the Slack? 🙂 (I don't have any contact information of him)

I invited him now: Johannes.Koester@uk-essen.de

Hey ho!

Hi Team Are there any notes or recording of the meeting? Much appreciated!

Thanks for sharing the notes. Here at Plant&Food Research, NZ, we have been developing a NextFlow benchmarking pipeline for structural variants (>50bp) using long read (ONT, hifi) simulations. Our focus is mainly on ploidy and dosage but we are also interested in combinations of mappers, callers, variant length, read length, read quality, etc. Keen to share more details and contribute.

Yesterday, I had the pleasure of meeting @Maxime Garcia and @Phil Ewels at the Oslo Bioinformatics Workshop and learned about this channel. I'm currently comparing the performance of NGS pipelines on CPUs and GPUs (currently at the early stage, comparing a few processes). I wonder if there is an interest in benchmarking CPU vs GPU pipelines and would be extremely happy share thoughts on this.

Any plans to connect to https://openebench.bsc.es/ ?

Hey all, we just merged this: https://github.com/nf-core/sarek/pull/1272 which (if I didn’t mess something up in the GHA), takes the results of the sarek full size tests, uploads them to the Zenodo and submits them to NCBench (https://ncbench.github.io/report/report.html, https://github.com/ncbench/ncbench-workflow). In addition, we added a new test profile that allows you to run the agilent test data (NA12878) directly. Hope this helps a bit 🙂

Without having really looked deeper into it, might https://openebench.bsc.es/benchmarking/OEBC005?event=OEBE0050000000 be interesting for the sarek crowd?

This polly is closed. @Kübra Narcı has a polly for you!

Hey all, I have been working a bit on the Structural Variant benchmarking pipeline (I call it svbench) for a couple of weeks, able to collect some ideas for a draft pipeline. Since @Nicolas Vannieuwkerke is also working on a structural variant calling pipeline we will start to align our pipelines together and share some ideas. We are planning to have a meeting between 12-15th February to discuss the ideas and I would like to invite any of you who would be interested either to share ideas or work with us for the pipeline! so here is a pool to decide on meeting time ☝🏻

I seem to only be able to choose one timeslot 😁

@Kübra Narcı has a polly for you!

Your results are in! 🎉

Hello all, I just sent a meeting invitation to responders to the pool on up. But if you still wanna join here are the details: 12 February 11-12 am Join Zoom Meeting https://dkfz-de.zoom-x.de/j/8747027533?pwd=Tf4iCNaaAyYGbYoV46wahJgvqkiPxN.1 Meeting ID: 874 702 7533 Passcode: RG8kq5

set the channel topic: Special interest group about BENCHMARKING

set the channel description: Benchmark EVERYTHING

Hello all, here (https://github.com/EUCANCan/oncoliner/tree/main) there is a cool project for somatic benchmarking. I did not go over it myself yet. It is the project in our last meting Johannas was suggesting it to wise.

@Kübra Narcı what is the status of this special interest group?

do we really need one, or do we cover it most with #C06K0KEAMTN?

Good question 😄 benchmarking is not only about variant benchmarking but I am not sure how many people is interested. It might still can be interesting to hear benchmarking stories from people?

@Maxime Garcia 👋, finally I went further and submit https://github.com/nf-core/proposals/issues/122. Who would like to see this become an official special interest group?

Congrats on the NEW official special interest group @Kübra Narcı! To finalise it formally: 1. I've made a template PR for adding the new group here: https://github.com/nf-core/website/pull/3895 . Please could you fill it with content! 2. Go to #C081F8J2X8R to schedule your bytesize introducing the group 🙂 3. Optionally, feel free to make a short blog post PR to the website to also announce via the blog (and if you do, let one of core team know to make an #CE6P95170 post on your behalf 🙂 )

Hello everyone, I am excited to get this group active again! I have edited the PR that James initiated for us here: https://github.com/nf-core/website/pull/3895/changes. I plan to organize a kick-off meeting after the hackathon. In the meantime, I would like to ask: is there anyone else who would like to take a leadership role in this group? As mentioned previously, we want to take full advantage of benchmarking in a broad sense. This means going beyond just variant benchmarking to include all other related topics. I think it would be great to have representatives from those different areas as well. I have included the following pipelines since they were mentioned before: s*arek, drugresponseeval, diseasenodulediscovery, and scspatialmap*. Is there anyone else interested in getting involved?

Hi everyone, 🙂 I was looking at this SIG project page for the next hackathon, and I was wondering if you guys are already aware of mribeirodantas.github.io/nf-diff/latest

Hi all! 👋 Who's joining the nf-core hackathon in Barcelona (Oct 15–16)? I'm putting together a tentative plan for the Benchmarking project. These are the goals so far: • Update the SIG Benchmarking page • Set up regular SIG meetings • Benchmark infrastructure and resources • Work out how nf-diff fits into variantbenchmarking • Collect reusable benchmarking components • variantbenchmarking development (good first issues included!) If you're thinking of coming, I'd love for you to join. Ideas and comments are welcome too, even if you can't make it in person!

Maybe thats interesting for that SIG: I have added to nf-core/ampliseq a sort-of-benchmarking (not released yet, but soon). I implemented scripts for a comparison of computed to expected outcomes, as described here, added (down-sampled) mock community data as test dataset, and use a (CI) test profile to ensure that future code doesnt degrade performance. I would like to use mock data & comparison to expected with all test data, but thats sort of for the future. I havent implemented the comparison to expected for that case solely, but its a nice side effect, I think.

Hello everyone! We are announcing regular monthly meetings starting on 11 November! Our first topic is going to be on variant benchmarking 🎉 You can read all the details in our new blog post: nf-co.re/blog/2026/benchmarking-sig-launch We are currently scheduling topics for upcoming meetings and looking for speakers. Please let me or @GeorgiaKes know if you would like to showcase your benchmarking project, present a topic, or lead a discussion. Let's reenergize this group! 🚀