https://docs.seqera.io/multiqc/reports/customisation#software-versions-yaml-file states:

If you prefer to provide versions in a separate YAML file, MultiQC will also look for filenames ending in

*_mqc_versions.(yml|yaml)

in your search path, for example

rnaseq_mqc_versions.yml

or

mapping_mqc_versions.yaml

. The content of these YAML files should be a YAML dictionary, similar to the previous examples but without the top level

software_versions

.

...but it appears that at least most nf-core pipelines/modules use

versions.yml

, and not

*_mqc_versions.(yml|yaml)

. At least for nf-core/rnaseq, the following is used:

softwareVersionsToYAML(ch_versions)
        .collectFile(storeDir: "${params.outdir}/pipeline_info", name: 'nf_core_rnaseq_software_mqc_versions.yml', sort: true, newLine: true)
        .set { ch_collated_versions }

Is there a way to just use

versions.yml

instead of requiring this somewhat complex step extra step of using

softwareVersionsToYAML

?

so we add the software because you could have multiple software in a single container: https://github.com/nf-core/modules/blob/master/modules/nf-core/bwa/mem/main.nf#L48-L52

so we add the software

You mean adding the software to the file name? Really, versions should be named by container or conda env, right?

so each modules produce a

versions.yml

file

All of them gets collected and produce a single

softwareVersionsToYAML

is not built into nextflow, and there seems to be no standard docs on collecting versions into one yaml file

you are right,

Any plans on making this process more integrated into nextflow (docs and/or build into the nextflow codebase)?

So our plans at the moment would be to migrate that into a plugin. But yeah, definitively some extra-documentation would be good

Any thoughts on:

Lastly, calling the executable to get the version for every single job is inefficient, especially for 10's or 100's of 1000's of jobs.

Maybe using a version introspection system divorced from processes would be helpful, in which the user provides a command that is run once per container/env (similar to Snakemake).

I would agree with you on that bit. I'd say this way of getting the versions is not the most optimal one

Yeah, it's something I miss about Snakemake: obtaining software versions was much more efficient and streamlined

I mean you could always "trust" the conda declarations on use that directly

So

cmd

would just be run once per process?

so I'd say this ties in with

topic

It would be helpful if https://www.nextflow.io/docs/latest/process.html#eval-output-eval included examples of where

eval

is useful

but the uniqueness of running the cmd is a good question

Is the general thought that

eval

would be used to obtain software versions?

It is my thoughts

So how would that work for creating the yaml for multiqc?

I don't think it's general yet

https://nextflow.io/docs/latest/reference/channel.html#topic is definitely helpful for aspects like

versions

. My current code for a simple pipeline:

ch_version_files = ch_version_files.mix(STAR_map.out.versions)
        .mix(SALMON_quant.out.versions)
        .mix(TXimport.out.versions)
        .mix(ALIGNMENTS_distances.out.versions)
        .mix(BAM_MARKDUPLICATES_PICARD.out.versions)
        .mix(STRANDEDNESS_fromSpliceMotifs.out.versions)
        .mix(BAM_RSEQC.out.versions)

The whole idea behind

topic

is to avoid having such multiple lines

ch_version_files = ch_version_files.mix(STAR_map.out.versions)

Note that all of this ☝🏻 is not specific to MultiQC. There are several things going on here: • MultiQC natively pulls software versions from logs where it can • MultiQC can take a file of version numbers if you supply it • nf-core likes to have process name <> software name association for version numbers, and jumps through a bunch of hoops to do it