This could even be its own pipeline

Honestly it could 😅 but I think for now a megasubworkflow would be more flexible I have some commens on the above that we can talk aobut during the meeting today! Mainly we should maybe define a clear scope (I'm not sure about hte amplicon specific bit yet), and I think the bowtie2/minimap2 is less good than e.g. hostile or deacon or something like, and why the seaparation of adapter clipping and merging ? (most tools do this togetrher in one step) But we can discuss in a wider round then!

I've never played with fastq.. I'm a protein fasta guy, so this was my first take according to what people were suggesting + a little bit of chatgpt for the categories/stages :p

OK cooL!

I would honestly prefer it to be a standalone pipeline but I guess its trivial to make one that just wraps the subworkflow. From a methods standpoint, it seems like it would be easier to describe it as “we preprocessed the raw data with nfcore/preprocess. Clean fasts we generated assemblies with nfcore/mag, and published the preprocessed data and mags with nfcore/submission”. From an execution standpoint, a standalone pipeline would make it easier to organize the results, and less likely for multiple copies to be generated inadvertently. Say someone is using both mag and taxprofiler: they wont be able to execute both concurrently cause one will have to wait for the other’s processes (including the preprocessing subworkflow) to complete. Further, its not intuitive to have to dig for generalized preprocessed data within a nongeneral pipeline’s output (eg chaining mag from taxprofiler’s preprocessing results, when the two have nothing to do with eachother). For the duplicated work argument, I could easily see situations where someone wanted to run mag, didn’t know someone already preprocessed the data within taxprofiler, and ends up rerunning the same preprocessing (which isn’t great for resources) or preprocessing with slightly different parameters (which is great for resources OR reproducibility/provenance).

IMO it should be both? For better or for worse, mag/taxprofiler/etc. are designed as fully end-to-end pipelines, in part I presume because it makes them beginner friendly - no need to figure out how to pre-process your stuff, jump straight to some analysis with your shiny new data. So we can have a big subworkflow (or IMO, a subworkflow wrapping some smaller subworkflows 😉) and that can be wrapped in its own "nf-core/metaomicscleanfastq" or similar

nf-core/qualitycheck? Could potential scale outside metaomics

I was mainly thinking about scope of options - presumably you do different stuff if you're playing with RNA or variant calling, etc?

I agree that end-to-end solutions can be nice for beginners, I definitely wouldn’t want to raise the barrier of entry. Personally I shied away from implementing the nf-core pipelines in the past because I didn’t like the baked-in preprocessing for mag or taxprofiler — I needed deduplicated data, those pipelines didn’t have it, and I didn’t want to end up with double preprocessed data in my methods section ( my own host removal and deduplication -> mag’s host removal -> the rest of mag etc). Many of the existing preprocessing options can be turned off (eg mag’s clipping tool) but there are some that cannot. Maybe designing it as a subworkflow and having a minimal wrapper pipeline for it (nf-code/quality-check) would suit both sets of people best.

For mag, it is on the list of issues to properly implement a set of flags to skip each type of pre-processing

sorry, I’m not trying to pick on mag specifically! Just the general pattern of baking qc decisions into downstream analyses seems fraught. Plus, perhaps it would make it easier to support pipelines if we don’t have to support the QC steps.

No worries, didn't think you were picking on mag - just wanted to highlight some of the complexities involved! But in general, I would also prefer that all QC steps could be skipped (including FastQC, pre- and post-).

Please take all these ideas with a grain of salt, I’m new to the community and haven’t seen half the use cases! From an outside perspective, it seems the fact that more than half of taxprofiler’s metro chart is qc should be cause for a standalone pipeline for easier support, testing, and reporting. Detaxizer has multiple QC steps, as does metadenovo, mag, ampliseq, demultiplex, bactmap’s fastq input path, etc

Ok kids are finally asleep, good to see a good convo going on! Like I think we've talked before @Nick Waters eventually nf-core does want to go towards the direction of more modular pipelines. However Nextflow doesn't natively support pipeline chaining, so actually it woudl be much more painful for users - particularly as many users are not as computer savvy as most of us are - thus like Jim says it's important to cater to more novice users with end-to-end pipelines. But that said, there is no indeed reason indeed not to make a pipeline out of the subworkflow! Then in each pipeline (mag, taxprofiler, sarek etc), switching all the custom subworkflows for preprocessing data to a single subworkflow would make it MUCH easier to make the whole thing skippable, and thus when training chaining exists to start from there as an entry point. But for now I think getting the subworkflow ready will be much less work than a fully fledged pipeline (which has a lot of additional overhead). Also name suggestion:

nf-core/preprocessngs

? then we have

nf-core/fetchngs

then

nf-core/preprocessngs

?

NGS in some contexts = short read PE sequencing, and not e.g. ONT/PacBio long reads - is that something we should worry about for a name?

Yeah I realised that when typing it, but

fetchngs

can do both AFAIK...

Maybe just plain ol

nf-core/preprocess

?

I think we would need seq somewhere in there