I wanted to remind myself of the difficulties here in daisychaining so decided to implement the metaomic pipeline that @James Fellows Yates presented at the barcelona summit last year. https://github.com/mahesh-panchal/nf-metaomics-daisychain. Including the pipelines has been trivial, but the issue is I'm not really sure which files I need from which pipelines that go into the others. https://github.com/mahesh-panchal/nf-metaomics-daisychain @James Fellows Yates said @Lili Andersson-Li and @Daniel Lundin could provide more insight into which files I need to go from one pipeline to another, for some other tools like taxprofiler and metatdenovo, etc. I'm looking for which flags need to be enabled for a pipeline and where the outputs get written, that I can then make a samplesheet or something for the next pipeline.

I think one has to take this on a use-case by use-case basis. For the pipelines I know, I can imagine that chaining output from #C02FC6VFQG1 into #C045UNCS5R9 could be one that's relatively straightforward. The former outputs a table of counts that I presume the latter can consume. Another option is #CE9MS66BS to #C01U1E95JQ5, where the former outputs a set of MAGs that the latter can use as mapping targets.

The meta pipeline will be configurable to use which ever pipelines one wants to chain, so that's not an issue. It's more that if a downstream pipeline needs input from an upstream, although, they can all be used standalone if one wanted too. Do I need to do anything special to get output from metatdenovo into differentialabundance or is there a samplesheet?

For the MAgs to magmap you would presuambly want the bins from

GenomeBinning/

, maybe even the refined bins in

DASTool

Currently there's no built in support for chaining in #C02FC6VFQG1, i.e. no output of a samplesheet for any other pipeline (we're not working on this, but planning), so one would have to point to e.g.

<results>/summary_tables/<assembly_name>.<orf_caller>.counts.tsv.gz

. However, I don't know how #C045UNCS5R9 works, so I'm just guessing this input could be used.

Thereisn't for anything at the moment - that's what Sofia was also asking about. @Joon Klaps started testing this to make it much easier to roll out our initial system which if it now works then it'll make that MUCH easier

@Mahesh Binzer-Panchal We are interested in chainning #C031QH57DSS and

gms/metaval

together. So metaval will take the output from taxprofiler as input.

Regarding #CE9MS66BS, James is correct. In #C01U1E95JQ5 there's a "genome sheet" that takes a name (accession), path to contigs and, non-mandatory, a gff file path.

Sofia and I are working on the same project. We are happy to collaborate on pipeline chaining @Mahesh Binzer-Panchal

I said we haven't tested the fixed new publishing system from Ben, but maybe you and her could collaborate on this?

So the idea would be to make samplesheets then for a given workflow? My plan was to just extract the necessary inputs from the folder structure and form an input within the daisychain, but adding samplesheet outputs directly to the child workflows would also work.

So the idea would be to make samplesheets then for a given workflow? My plan was to just extract the necessary inputs from the folder structure and form an input within the daisychain, but adding samplesheet outputs directly to the child workflows would also work.

With an input you mean feeding the other workflow a channel directly? Our original idea was indeed to use the new output declaration to let nextflow resolve the correct locations etc and have samplesheets ready that then can be used for various other pipelines

My plan was to just extract the necessary inputs from the folder structure and form an input within the daisychain, but adding samplesheet outputs directly to the child workflows would also work.

That's up to you, but yes the samplesheet output idea was our original concept. This way people who don't want to automate the whole thing but still make it easy to trigger the next one can benefit

So nf-cascade uses a channel to capture the results folder. I can then extract whatever files I want from that, which could be an input samplesheet or a collection of files that could be turned into a samplesheet. That channel is then passed as input to the next workflow in the daisychain

@Joon Klaps you didn't add the fancy new

publish:

thging in this PR did you?

I guess I didn’t push it and went straight to report the issue

Do you think you still hae it smewhere

No, gitpod environment is long gone

sad panda

This is just a side project for me too, so I'll see what I'll have time for after the hackathon, but I'll check it out. At the moment I'm just trying to trace what needs what now

Ok!

Also what's the

gms/metaval

pipeline?

non-nf-core pipeline

ah, ok. Do you want that in this metapipeline?

I think GMS would be exrtremely interested in a metapipeline

It’s a taxprofiler post processing pipeline.

What am I missing in terms of samplesheets needed here? https://github.com/mahesh-panchal/nf-metaomics-daisychain/issues/1

Is that ALL POSSIBLE combinations?

I think it's missing things

(note eager you can skip for now as it's still dSL1, unless you use the

dev

branch which is now DSL2)

Sorry, just to double check. The samplesheet for gms/metaval needs to be constructed by hand from the output of taxprofiler right?

Exactly. I need to create metaval samplesheets using taxprofiler ouput.