Is there a way to manipulate a module output channel to transpose outputs within the module? I’m looking at the case of

bcl-convert

. The

fastq

ouput channel contains a list of all demultiplexed fastq’s. I would like to be able to transpose this list to a queue channel containing

[meta, [fastqR1,fastqR2]]

You don't want to make a downstream module, or custom module? There are some cases that just can't be handled well in the strict framework

For large flowcells we’re talking about ~ 4TB of fastq data, I want to avoid having to stage it as much as I can.

Understood. I would include the code in the module, and use a patch personally ( both module and subworkflow )

I’m not sure I get what you mean?

My understanding was that you wanted the code in the

map

to be moved elsewhere without incurring the staging or memory overhead that's currently happening on the head node. Since you want avoid the memory intensiveness on the head node that rules out native processes, and to avoid the staging overhead, the code should be moved to within the

script

section of the preceding process. Although now seeing the two preceding

if

statements, it looks like you want a separate process anyway that would just take the output and iterate over each fastq to apply the modification from the

map

. Naively I thought this was something

samtools

does right? Some of the channel operations would need to be kept there though to make the appropriate tuples

turns out

bcl-convert

outputs a file called

fastq_list.csv

that contains all the info needed to construct a readgroup, so we can associate the data with the correct files that way, without having to read each fastq file. @Simon Pearce, perhaps this could also be a fix for the issues you’re having with #CL88J906S? This fix does preclude other demultiplexers such as

bcl2fastq

(which is legacy anyway) or

bases2fastq

for Aviti sequencers

Personally I'm only interested in bclconvert at the moment 😉 . We could make any extra output from the bclconvert if that file isn't sufficient (and/or make a. newoutput channel). Are you able to make some changes? You have an open PR with bclconvert to topic too

I’m working on a POC right now. You can add custom fields to the illumina samplesheet that get transferred to the fastq_list.csv file, so I’m planning to hijack that feature to get all the data I need the recreate the readgroup tuple for our pipelines.