And the second: In preparation for the hackathon, @Lili Andersson-Li has been doing some investigations and testing into the new workflow outputs publish system (https://www.nextflow.io/docs/latest/tutorials/workflow-outputs.html#overview), to see if we can help support pipeline chaining (as we will have a hackathon group on this for #C070369GP7T). This has been referenced as a possible more efficient way of generating samplesheets than a lot of custom code and manual 'reconstruction' of

--output

paths in a samplesheet as we would have to do now. However with the new workflow output system, we've encountered a possible blocking problem. It appears there is no efficient way to support generatation of samplesheets for multiple downstream pipelines in a single pipeline run This is because there appears to be no way to efficiently generate multiple index file (i.e., samplesheets) for each channel output, as you cannot make multiple index files from a single publication 'event'.

'processed_reads' {
        path  "bbduk"
        }

        index {
            path 'processed_reads.csv'
            header true
            sep ','
        }
    }

One method we discussed was to generate a single channel for each downstream samplesheet you wish to generate. However, in the current system, each channel's index creation event MUST be coupled with a publication event (i..e, copying files). Therefore if you want to generate 2 samplesheets, you will need to specify three publish/copying events (even though they are identical): the main output, then one for each if the 2 samplesheets - which obviously is very bad from a IO point of view.

'ch_mag_samplesheet' {
        path  "bbduk"
        }

        index {
            path 'processed_reads.csv'
            header true
            sep ','
        }
'ch_metaval_samplesheet' {
        path  "bbduk"
        }

        index {
            path 'processed_reads.csv'
            header true
            sep ','
        }
    }

One option we have now discussed is instead pushing the problem 'down the road' with a 'pull' method, whereby instead of 'push'ing a pretty much already-compatible samplesheet for the downstream pipeline, we just generate a index file (i.e. samplesheet) 'as is' from the upstream pipeline, and instead it's up to the downstream pipeline to 'pull' work out a way to 'consume' the output from the upstream pipeline (e.g., convert and validate the contents). However, this is not such a good user experience: this would be potentially be more annoying in that a user will have to potentially do more custom curation of the output of the upstream pipeline to add missing information, and the developer will need to make multiple

schema_inputs.json

for each of the different upstream pipelines (and their versions) etc - whereas in the 'push' you can have a basically 'pre-made' template that is clearer for the user to 'fill in' (if you follow)

Naïve question, can you just throw duplicated publish paths to

/dev/null

?

It's all groovy level, it's nothing to do with bash (so ti say)

/dev/null

is just a file on disk (in Unix) - so you should just be able to specify that as the output file path

Ah got you

Yes, i can try it

Actually, would that work? Or would it just write

/dev/null

inside the index file...

Yeah exactlyu

have you asked Ben?

That's the next plan (ooo today)