https://nf-co.re logo
Join Slack
Powered by
So what's your opinion on this? If Pipeline C take...
# wg-meta-pipelines
m
So what's your opinion on this? If Pipeline C takes either input from Pipeline A or B, who is responsible for supporting the samplesheet to directly to C? For example Fetchngs produces samplesheets for Taxprofiler, Rnaseq, etc, but what if the pipelines are not in the same ecosystem, e.g Taxprofiler and gms/metaval. Which pipeline should be responsible for the samplesheet creation?
j
While it's less optimal in terms of keeping in sync, I think it should be the upstream pipeline. The upstream pipeline knows what information it already has internally, and can how to prepare it correctly for the downstream
Also they are in control where the files etc. goin it's own output directory
But that's me 😬
m
So Devil's advocate here. Why should you implement/maintain an output samplesheet for someone else's pipeline that you don't know if it's going to be maintained or not?
j
Because the downstream pipeline will have to make hacky bash or python scripts (I expect) to be able to parse and find the output files in the output directory of the upstream pipeline - it would be harder to do that in the downstream pipeline, as you will be missing a lot of the meta information that is not stored in the file names of the files
👍🏽 1
e.g. let's say taxprofiler --> mag, taxprofiler uses sequencing platform as information. This is also needed by MAG, but how would the sequencing platform information be given to mag from taxprofiler, when taxprofiler only has that information in ti's 
meta
maps
m
pipelines should provide samplesheet for the next pipeline
m
even if it's outside the same ecosystem?
m
yeah for me rnaseq for example should produce samplesheet that can be used by a downstream pipeline
In my idea of a meta pipeline for sarek. fetchngs produce the initial samplesheet. that is used by the align pipeline which produces a samplesheet with bam which is used by the variant calling pipeline that produces a samplesheet with vcf which is used by the annotation pipeline that produces another samplesheet with vcf (but annotated_
j
Adding a module to build a samplesheet from the meta-map does not even solve the problem, right? Don't we need to know specifically the required columns that a downstream pipeline pipeline expects if this is meant to be generalizable?
m
With nf-schema you can just read in the columns you need if you output a generic samplesheet.
m
> even if it’s outside the same ecosystem? > yeah I don’t understand why nf-core would feel obligated to output specific samplesheets for my pipelines that are outside of nf-core
j
Well in this case @Lili Andersson-Li developers both nf-core/taxprofiler and her non-nf-core metaval 😉
There doesn't have to be an obligation I guess
j
> if you output a generic samplesheet. Meaning you use generic column names between all pipelines? Sounds hard to enforce outside of nf-core.
Maybe part of the solution could also involve one-off manual stitching/mapping of columns (at time of setting up a meta-pipeline).
m
Meaning you use generic column names between all pipelines? Sounds hard to enforce outside of nf-core.
If someone wants their pipeline to chain off another and the only difference is the column headers in the samplesheet, I think the downstream pipeline would happily change their input requirements.
m
I've been thinking with @Nicolas Vannieuwkerke about adding a check and version the samplesheet specs
m
I think the issue is less the versioning, and more the input naming. If Pipeline C can take output of Pipeline A or B, but A and B use different column names, we've got an issue.
n
I think it's more an issue of standardizing column names across all pipelines
m
That's fine in nf-core, but not generally
Or does EDAM or somewhere have an ontology for that?
j
I'm suprrised @Maxime Garcia hasn't jumped in and gushed all over this
standardizing column names across all pipelines
😆
shakefist 1
🤣 2
n
It's an issue though 😅. I've gone over so many schemas during the hackathon
Some pipelines have 
id
as the sample ID name, others have 
sample
, others have 
patient
(that's just one column)
j
I think it'll be hard to standardise to be honest
n
Yeah that's a fact! Just pointing out the problem here 😛
💯 1
m
I did jump on it, that's what I meant by the check and version the samplesheet specs
I'm just going it another way
😆 1
l
gms/metaval
takes the same column names as 
taxprofiler
and there are some new columns in 
gms/metaval
Open in Slack
PreviousNext
Powered by