With the minimap2/align (and other read mapping modules), the output BAM files all receive the meta object of the input reads channel. However, often I want the opposite, and would like to get the meta object from the reference in the output channel (as I am usually working with mapping a single set of reads to multiple assemblies). Seems like there could be a couple methods for doing this - either a) fix it in my pipeline myself - merge the reads + reference into a channel,

multiMap

, and then enter them into the module, or b) add an explicit input channel to flag to the module which meta to emit (reads, reference, possibly their combination?). Does anyone have strong feelings about b), one way or the other? Do we prefer busier module code and tidier pipeline code, or busier pipeline code with streamlined modules? edit; for reference, here's a subworkflow where in one instance I need the 'reference' meta and in another I need the 'reads' meta: https://github.com/sanger-tol/genomeassembly/blob/dev/subworkflows/local/purging/main.nf#L43

I would say in this case since it's not very common to want the reference meta as output, you could modify the module in your pipeline and use

nf-core modules patch

I do currently have the module patched as above - but I was curious if there was more general interest in this functionality as I tend to find patches don't play well with module updates

yes it's true that the patch doesn't always resolve conflicts with very big changes. But I would guess changing only one value in the output channel shouldn't cause many problems?

Ah, but sometimes I also want it the normal way... 😅

ah ok! this complicates thing 😄

I have thoughts when dog walk finished!

The problem with the multiMap approach is that it requires (currently?) that you know what meta keys are needed to join the channels, so you can pull them out accordingly for the join - if you want to do this in a shared subworkflow, where you can't necessarily make assumptions about meta contents, that becomes more difficult

Ah I see...

If you need it in a shared subworkflow then I would just make the subworkflow have whatever assumptions you want (ie id should be the unique key, or id + reference etc) then whatever workflow calls the subworkflow can handle preprocessing the meta into the right format before and after the subworkflow

Musing on this a bit more - personally, I do think this should be the responsibility of the module to output the desired data, rather than finagling the code around the module to “trick” it into putting out the desired data. I think mapping a single set of reads against multiple sets of contigs is not that unusual a pattern - off the of my head, this might be a pattern that you’d want to use in mag, magmap, genomeassembler? For example, assemble the reads into assemblies with different tools, add a

meta.assembler

key, and then map the reads to all the assemblies - you’d want to keep that assembler key in the output. That said, the code I suggested above is indeed wordy! Another way to solve the problem might be to add a new

ext

directive, available in all modules where there is more than one meta object, that is set by default as:

def out_meta = task.ext.out_meta ?: meta

That takes one line of code in the module, is generalisable to all module cases, and gives pipeline developers full control over the required outputs of a module. And its customisable on a per-case basis if you import the module with different names.

This would absolutely not be a good idea in my opinion.

ext

variables are already not particularly intuitive and the fact they are set in config makes the transparency to users and other devs very poor. Plus, you would really need to have the option for all meta in a process (and there can be many). I think you would be better off trying to coalesce support around injecting a reference meta value in any modules directly, although that would also require gaining support for some meta key within nf-core that may now already be widely used.

I take your point about discoverability, and if there really is no interest in this functionality I'll just go ahead with a mixture of either patching local modules, or forking the nf-core module to our shared module repo where we need the functionality across pipelines/subworkflows. But I do think this would be nice to support in nf-core directly.

Plus, you would really need to have the option for all meta in a process (and there can be many).

I think this is solved by the above, as you can compute any arbitrary combination of metas with it, e.g.:

withName: processWithFourMetas {
    ext.out_meta = { meta + meta2 + meta3 }
}

That assumes there is only ever 1 meta in your output channels though? (I think this is mostly, maybe even always currently the case, but it’s not guaranteed)

I do think this should be the responsibility of the module to output the desired data, rather than finagling the code around the module to “trick” it into putting out the desired data.

The problem is unfortunately the diversity of what data people want to emit etc. The reason why we've been very VERY careful about messing with meta is that you can be sent down a lot of pipeline wrangling to change your pipeline's meta fields just to match each module's/subworkflows assumptions as to what each meta is called.

For example, assemble the reads into assemblies with different tools, add a

meta.assembler

key, and then map the reads to all the assemblies - you’d want to keep that assembler key in the output.

That said, @Maxime Garcia has wanted to come up with a controlled vocab for parameters and meta terms for a long time... we could annoy him by being first and develop a controlled vocab for metadata terms within #C070369GP7T for example 😉

The more I think about it though, the more I wonder whether this is a fundamental issue with

nf-core

modules design (which would likely be a huge fight/non-starter to overturn). It’s the same issue I take with the

fasta

and

fai

channels being separate (which is actually something I’m surprised doesn’t break your usage) which is that, unless the channels are single-value the fasta/fai would get incorrectly mixed

It’s the same issue I take with the fasta and fai channels being separate (which is actually something I’m surprised doesn’t break your usage) which is that, unless the channels are single-value the fasta/fai would get incorrectly mixed

Yeah this will very likely change soon ™️ anyway... Ben is not happy about it either but we are waiting for some more fundamental changes to Nextflow before making this shift

If you pass multiple different reference files, there is no way to ensure each one is used with each of your input samples is there?

If you pass multiple different reference files, there is no way to ensure each one is used with each of your input samples is there?

That's why I suggested multiMap

I'd love to go to 1 input with meta for data and 1 input with meta for references

But how do you exsport that?

I'd you need to re-export then it's input data

But I don’t think multiMap solves that does it? You could still end up with e.g.

ch_in = (sample1, sample1, sample2, sample2)
ch_ref = (ref1, ref2, ref1, ref2)

But get them actually processed as:

sample1, ref1
sample1, ref1
sample2, ref2
sample2, ref2

This is what we did for `bwa aln`: https://github.com/nf-core/modules/blob/master/subworkflows/nf-core/fastq_align_bwaaln/main.nf

If you pass multiple different reference files, there is no way to ensure each one is used with each of your input samples is there?

In my case, the input sample is a value channel, so it gets reused. But yes, otherwise you would need to multiMap. But that's true in both orientations - a bunch of samples against a single reference, a bunch of references against a single sample, or a bunch against a bunch

I'd have the reference id in the meta map of my input, and use cross/combine

Are we suggesting that

multiMap

guarantees channel order?

It does (or should do)

Because if not, then this simply does not safely work for multi vs multi.

As it makes 'sub channels' of the same item in a channel

It does (or should do)

Ok, I always just assumed that no channel order could ever be guaranteed but happy to be wrong here

IT's how I've been using it everywhere after being advised (I think?) by Ben

It guarantees channel order insofar as it splits a single channel into multiple channels, no? You have to combine the data you need into a single entry first before splitting

YEs

Maybe it’s just output channel order that is completely random then. I assumed it was any channel. (Becasue multimap channels are still channels)

I don't know if joins in general preserve order - but I would expect otherwise multiMap works on a FIFO principle

I wasn’t sure about how

multiMap

was implemented and I could imagine a scenario where parallel processing of the input channel would have allowed some weirdness.

ch_in = Channel.of([1,2], [10,1], [5, 20])
ch_in.view()

ch_in.multiMap{
    a, b ->
    A:
        [a]
    B:
        sleep(b * 1000)
        [b]
}.set{
multi
}

multi.B.view{"B->${it}"}
multi.A.view{"A->${it}"}

This results in the pairs being printed in the same order they entered though so I think it probably is order-safe. I’m a bit leery of this still though, given how much we drill in to people to not rely on channel order (one of the major reasons for

meta

in the first place)

Hmm, OK - point taken re:

multiMap

! I would still like a way to get all the input metadata out of a module (feels important for data provenance if nothing else) but see that there's not a straightforward way of doing this/there are workarounds, so I'll tackle it on my own end for now (I would ideally like to be join-agnostic!).

As I've now got lost within the thread: you didn't want to add all the info after the pre-multiMap join into the first

meta

entry (for the primary input channel) was simply because of joining afterwards?

I wanted to turn the subworkflow into a shared subworkflow (for our repo, not nf-core), so I ideally wouldn't want to make /any/ assumptions about the contents of the meta object. But joining requires knowing something about the meta contents

yeah I just don't think it's really possible unless you somehow have an additional input channel which describes what the meta keys are for each of the input files?

Maybe that's the best solution

Oh wait that's your option (b) 😅

My option b) was to modify the module itself, not the subworkflow

Ah... but why not also at the subworkflow level?

Indeed!

That keeps the modules very simple/clean

My preference would be to shove the jank into the module so that the pipeline code is very readable and clear 🫢 But I might be alone in that

Ha, my instinct is the opposite

The less weird channel manipulations, the better

In the sense that modules can be mostly standardised with common structures as they are the 'atomic' thing, whereas HOW you combine the modules is much more 'custom'

The less weird channel manipulations, the better

I used to feel this way but I am now much more used to using and reading channel manipulations and they feel cleaner now. You can also write code that is just channel manipulations and then bung it in a subworkflow and test it independently if you really want

This is true - but I think modules can be flexible enough to support different use cases, especially if there's a consistent interface. I really wouldn't want to add various sets of weird epicycles to a module either

Normally I have: pipeline workflow (clean) -> higher level subworkflow to link subworkflows (clean) -> lower-level subworkflows to link modules (messy AF) -> modules (clean)

Agree with everything from James there

I'm right!

All my channel magic is typically in local subworkflows

And it's also why shared subworkflows has been tricker/less popular - it's worked mostly with genomics because many of those steps ARE very routine for large areas e.g. human cancer genomics - you are only ever dealing with single reference genome basically

everyone else who deals with 'real' (messy) biology

I feel slighted, human cancer genomics can be messy too! 🥹

Come back to me when you have to handle 100,000s of uncharacterised microbes who do whatever the fuck they want and not just within their own freaking species!

@Daniel Lundin pointed out in the meta-omics meeting today that he has a similar use case for the diamond/blastp module: https://github.com/nf-core/modules/pull/7403

@Jim Downie you know you are efficient

late to the party, but I agree with the

multiMap

approach. Here’s how we go about doing this in eager: We have a custom util function (

addNewMetaFromAttributes

) that pulls parts of the metas of a channel into a new meta, that can then be used for a

join

->

multiMap

chain. The multimap generally also removes the created “join meta”.

/*
This function can be applied to the contents of a channel row-by-row with a .map operator.
It assumes that the first element of the channel row is a map of metadata. It will then create a new metadata map
that consists of the source_attributes of the original metadata map, but named after the corresponding target_attributes.
The new metadata map is then prepended to the channel row, becoming the new first element.
If the remove flag is set to true, then the original metadata map is removed from the channel row.

Example:
ch_my_channel=Channel.of( [ [id:'id', sample_id:'sample_id', single_end:true, reference:"hs37d5" ], "bam", "bai" ] )

ch_my_channel.map{ row -> addNewMetaFromAttributes(row, "id", "new_attribute", false) }
// This will create a new channel with the following rows:
[ [ new_attribute: 'id' ], [id:'id', sample_id:'sample_id', single_end:true, reference:"hs37d5" ], "bam", "bai" ]

ch_my_channel.map{ row -> addNewMetaFromAttributes(row, ["id", "single_end"], ["new_attribute", "endedness"] , true) }
// This will create a new channel with the following rows:
[ [ new_attribute: 'id' , endedness: true ], "bam", "bai" ]

*/
def addNewMetaFromAttributes( ArrayList row, Object source_attributes, Object target_attributes, boolean remove = false) {
    def meta = row[0]
    def meta2 = [:]

    // Read in target and source attributes and create a mapping between them
    // Option A: both attributes are Strings
    if ((source_attributes instanceof String) && (target_attributes instanceof String)) {
            meta2[target_attributes] = meta[source_attributes]

    } else if ((source_attributes instanceof List) && (target_attributes instanceof List)) {
        if (source_attributes.size() == target_attributes.size()) {
            for (int i = 0; i < source_attributes.size(); i++) {
                // Option B: Both are lists of same size
                meta2[target_attributes[i]] = meta[source_attributes[i]]
            }
        } else {
            // Option C: Both lists, but uneven. Error.
            throw new IllegalArgumentException("Error: The target_attributes and source_attributes lists do not have the same size.")
        }
    } else {
        // Option D: Not both the same type or acceptable types. Error.
        throw new IllegalArgumentException("Error: target_attributes and source_attributes must be of same type (both Strings or both Lists).")

    }

    def new_row = [ meta2 ] + row

    // If replace is true, then remove the old meta
    if (remove && new_row.size() > 1) {
        new_row.remove(1)
    }

    return new_row
}

Feel free to use/adapt this to your use case. 🙂