I added a few comments to the PR explaining my problems and thought processes 🙌

I think we have to

gunzip

the vcfs - but what would be the best practise to do that? I want to use test data but there is only gzipped vcfs with structural variants.

Hi! I’m curious about this pipeline, is there a test output report I could have a look? I would like to see what is reported and how. Thanks!

Hello, I am trying to run the pipeline using test profile. When it tries to pull singularity image for

VARIANT_EXTRACTOR

module, if fails returning a

255 status

and the following message:

FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for <oras://community.wave.seqera.io/library/htslib_pysam_tabix_pip_variant-extractor:a12ef217eccf6ba8>: no layer found corresponding to SIF image

Command used to launch pipeline:

nextflow run variantbenchmarking/main.nf \
-profile test,singularity \
--outdir results

Are you aware if there is any known problem regarding the container? May I be missing some configuration on the launch?

hey! How far along is this pipeline? is it stable and tested enough to be used with the dev branch?

IMG_6702.jpg

Is there a parameter set how I can run the pipeline to only do SV Benchmarking?

Hello all! Since I think I have been coming close to publish the first release of this pipeline, I am looking for 2 dedicated reviewers party parrot To be honest I am not quite sure about the process itself, but it would be great to have a start at least. It would be a great opportunity for whom would like to use the pipeline and possibly to contribute with new functionalities 🙂

anyone?

Hello everyone, it took a while but I want to announce that variantbenchmarking pipeline has its first release party parrot🚀 🎉Kudos to everyone who contributed! Especially to @Nicolas Vannieuwkerke and @Maxime Garcia, thank you guys, I really appreciate your support and help! That was just a start, I will keep the wheel going 🙂 It still needs improvements of course, so let me know if you wanna contribute or give it a try.. 🙂

Hi I updated the JSON schemas of the pipeline, can someone take a look? https://github.com/nf-core/variantbenchmarking/pull/168

Hi! I haven’t tried the pipeline yet but I think it looks great. Is there any existing functionality or planned development for assessing concordance between test VCFs (e.g., replicate comparisons)? I think this would be a useful feature.

Hi everyone, is this pipeline suitable for benchmarking of variant callers that detects somatic structural variants SVs in tumor samples? I saw in the docs that the pipeline does not include a tool specifically dedicated to somatic SV benchmarking, so I’d like to confirm whether it’s appropriate for this use case, thanks

Hey folks I'm trying to figure out something about what the interplay between the

docker.registry

parameter is and the custom images in the repo. The registry is set here: https://github.com/nf-core/variantbenchmarking/blob/1.2.0/nextflow.config#L228 And running in docker only mode the bcftools/reheader module pulls from

biocontainers/bcftools:1.20--h8b25389_0

which maps to: quay.io/biocontainers/bcftools:1.20--h8b25389_0 however with custom/bedtools_intersect we're pulling from

<http://community.wave.seqera.io/library/pybedtools_pandas:6eae56b7891a94c6|community.wave.seqera.io/library/pybedtools_pandas:6eae56b7891a94c6>

which is a full URL to a wave registry. Is there some internal logic I'm not aware of that's overriding the docker registry set in the first link I set above when community.wave is pre-pended in the container name?

Hi everyone! I am trying to use nf-core/variantbenchmarking with GIAB data and rtg-tools option for benchmarking and I am getting this error: Error: An IO problem occurred: “Invalid file pointer: 14104788689 for HG001_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_PGandRTGphasetransfer_appchr_snps_cov.reheader.vcf.gz” I am not very familiar with nextflow so any help would be much appreciated! Also, are there any test files and test commands available for this pipeline? Apart from the samplesheet I could not find any working example in github. Thanks!

I'm noticing that the VariantBenchmarking pipeline fails if you're comparing a test and query file of the same name. I imagine the underlying issue is actually one that nextflow should address but has anyone run into this issue before? Building some automation to test releases will regularly require doing comparisons between files of the same name so I imagine others have run into this before.

Hi everyone! I’m looking to validate germline CNV calls from WGS data using the

variantbenchmarking

pipeline (with calls generated via CNVKit), but I’m having trouble identifying a suitable truth set for comparison. Has anyone had experience with a reliable CNV truth set that they’d be willing to recommend or share? Any help or advice would be really appreciated! 🙏

Who dis?

Hi,

Hi everyone! I’m running nf-core/variantbenchmarking and hit an error in PREPARE_VCFS_TRUTH:BCFTOOLS_REHEADER_TRUTH (SEQC2): Error: missing FORMAT fields, cowardly refusing to add samples. Failed to read from standard input: unknown file type. My truth VCF high-confidence_sSNV_in_HC_regions_v1.2.1.vcf.gz. Could the pipeline skip reheader, or is there another way to handle this? Also, when I tried running the example test profile (somatic_snv) from GitHub, the pipeline fails during parameter validation because the test input files no longer exist. Thanks for your help!

Hello all, we have a new release (1.4.0) now https://github.com/nf-core/variantbenchmarking thanks to patient reviewers @Florian Heyl @Louis Le Nézet 🎉 some important updates: • Truvari bench update to 5.3.0 • svync update to 0.3.0 • Added a python script to plot indel distributions, thanks to @Victor Perez • UPSET plot for TP/FP/FN numbers • Added svtk/standardize from GATK to standardize SVs to BND format • nf-co2footplot plugin is added • Concordance analysis with GATK concordance (only for germline snvs) • Updated multiqc reports, thanks to @GeorgiaKes

random question im looking at the description of the pipeline. so you prepare a sample sheet with svs,cnvs,snvs. you need to run it 3 times with a truth svs, truth cnvs, truth snvs? is this a correction interpretation? and I would just swap params.vartype for each test? does that mean it ignores the "non-relevant" variant types in the sample sheet?

Hi @Kübra Narcı! I would like to start working on the task Add rtgtools/cnveval #266. I noticed it has been assigned since January and there hasn’t been any recent activity. Would it be okay if I take this task?

Hi all! I hope you're enjoying the hackathon. I wanted to share an idea for discussion and possible future implementation. What about using Depictio for visualization of the pipeline outputs? I noticed that nf-core/ampliseq already uses it: https://demo.depictio.embl.org/dashboards. It could potentially keep the pipeline lighter by moving most visualizations there. Want do you think? 🙂

Hi all, I am looking to benchmark germline CNVs from WES data generated by the DRAGEN v4.4 pipeline. I am using HG002 truth sets (NIST SV v0.6, CMRG v1.00, and the latest v5.0), along with benchmarking tools such as Wittyer and Truvari. So far, I have benchmarked small variants using hap.py and structural variants using Wittyer and Truvari. However, I am wondering whether it is appropriate to use an SV truth set to benchmark CNV call data specifically. Are these truth sets suitable for evaluating CNVs from WES, or is there a more appropriate reference for this purpose?

Thanks for providing the variantbenchmarking pipeline, which I have run without problem. I am unsure how to interpret the various upset plots and wonder if there is a descriptive document somewhere. Thanks.

Hello, I am using the pipeline to benchmark small variant amongst deepvariant VCFs. These VCFs are generated with same reference genome. I wonder what filter_contigs do and could I skip this. It seems this step only supported for pre-defined genome.

depictio Depictio dashboard for variantbenchmarking: your feedback wanted Hi all! 👋 I am building Depictio, an open-source platform from EMBL to explore pipeline results without writing a Shiny or Streamlit app each time. A pipeline gets a dashboard template: a short definition that turns the outputs of any run into an interactive dashboard. I discussed an earlier version with @Kübra Narcı, but the project has evolved a lot since and I am now sharing it more widely, so I would like the view of other variantbenchmarking users and developers too. ✨ What it does • MultiQC reports (aggregated across runs) next to result tables, images, phylogenetic trees, maps... with linked filters • Reusable: point the template at any new run of the same pipeline and you get the same dashboard • Fully editable: modify any dashboard from the GUI or in YAML, so the template is a starting point, not a constraint • Versioned, with comments and annotations on any component • Runs as a Python package, with Docker Compose or on Kubernetes, and can serve as a companion web app for a publication 📊 For variantbenchmarking: which callset recovers its truth set best, on precision, recall and F1, from the rtg-tools vcfeval, hap.py, som.py, Truvari, SVanalyzer and Wittyer results. 👀 What I need: I have built 25 templates and want people who really use these pipelines to review and criticise them. What is useful? What is misleading? 👉 Dashboards: germline small variants, germline structural variants and somatic SNVs (no account needed) 💬 Comment on the dashboard (Google sign-in) or in the GitHub discussion dedicated to variantbenchmarking. ⭐ A star on the repo is welcome if you find it useful. Thanks! 🙏 🎤 I am also presenting it today at the Nextflow Summit (virtual agenda) if you're interested in the project.