Got a thorny one I'd like input on. Might just be "go away Jon, you're doing something weird", but I'm going to ask anyway. It's about versions. It's long, so for the lazy among you: TL; DR: Should nf-core modules emit meta-tagged versions, so pipelines running MultiQC more than once per workflow can build per-run versions YAMLs without a workflow-global barrier? To expand... Modules have always been engineered so they don't pass meta through to the versions channel (now topic). Any summary process consuming that channel has no way to link a version back to individual iterations of a process, so we have to aggregate without meta, which means we always block execution of any process (mostly but not exclusively MultiQC) reporting those versions. That was fine with one MultiQC report per run. In fact it was kind of the point. But a couple of us have started running MultiQC multiple times per run, and it means NONE of those MultiQC processes can run until ALL their inputs are ready. I've taken this to the nth degree with recent #CE8SSJV3N changes: one report per sample in a certain mode. That mode is designed to let rnaseq work at scale, churning out sample-wise results without agglomeration. But with versions in the mix, no single sample can finish and execute its final step until every sample can. My current workaround: don't feed the versions YAML into per-sample MultiQC at all in that mode, and trust that folks can track back from the workflow git SHA if they need provenance later. The 'proper' fix would be to emit metas from the versions topic in all modules:

tuple val("${task.process}"), val('fastqc'), eval('fastqc --version | sed "/FastQC v/!d; s/.*v//"'), emit: versions_fastqc, topic: versions

to

tuple val(meta), val("${task.process}"), val('fastqc'), eval('fastqc --version | sed "/FastQC v/!d; s/.*v//"'), emit: versions_fastqc, topic: versions

Most folks would instantly discard that info at the workflow level and uniqify by process, but those of us who need it could unblock our workflow completions and pass meta-specific versions to reporting. Topic handling at workflow level is easy enough now that this doesn't feel major, it just needs an update to all modules. Again. Thoughts?

@Maxime Garcia @Franziska Bonath isn't #C06PHJM0Y3D running mqc multiple times? how do you handle this?

I would bet money they've got blocking behaviour, or they're not adding versions

so indeed #C06PHJM0Y3D is running multiqc multiple times, let me check what we're doing

you probably don't care if you're running it a handful of times

that being said, before even checking anything, I am in favor of adding the meta map so that the versions topic stays on par with the multiqc_files topic

This is fine by me, but keep in mind that this would need an update to the template first before we can implement this 🙂

Mine cover all processes too.

just one question

No, I want sample-wise multiqc

where we separate groups of sample and have a global multiqc

... or some other meta-wise MultiQC in future

yeah, that's exactly what we have in multiqc

Point me at it?

Yeah, I think you still have the blocking there, you're still using

collated_versions

in

multiqc_extra_files_per_tag

Oh, I see what you mean

Like I say, I've taken it to the extreme, but I do think we've baked in something unnecessarily rigid, and the solution is relatively trivial...

I am on-board with adding the meta map to the topic tuple

this might solve a different issue too, that I am not sure we have encountered yet but one could imagine a closure for the containers were some samples are run with one version and some with another...

That did occur to me also. It would be scientifically more correct to retain a process association, even if we mostly simplify by removing it in most cases.

one could imagine a closure for the containers were some samples are run with one version and some with another

That sounds like a dirty hack I would love to build

I would also be in favor, but that would mean to update all the modules ... again 🫠

I could probably get Claude to do it in batches of N

I’ve seen the blocking behaviour but didn’t think it had to do with meta/filtering. My understanding is that

channel.topic

blocks until all the providers of this topic are exhausted. In my case, I run MultiQC near the end of the pipeline, not exactly at the end. I had to identify which modules run after MultiQC and make these modules output no versions topic.

at some point you want a log of the process details rather than a versions file? Like if you just dumped an audit log of what Nextflow did, it would contain all this information without having to consider a "versions" file a process output

I think it's sensible that each process reports the version actually run, in a way tied to the actual task that was run, rather than to the generic process name.