So I chatted a bit more with @Evangelos Karatzas after the meeting and we made this doc to better plan out the preprocessing subworkflow idea: https://hackmd.io/@nf-core/S1NVXogilg Rather than combining different seqeuencing techs into one, we could have one common subwork per sequencing tech, the first being an Illumina one: What does everyone think of this (more ugly, but more easily protypable) mermaid diagram:

Kicking off different subworkflows based on, for example, short or long reads is a good choice. Much better than littering the code with hundreds of conditions testing for that along the way.

I think this looks good, but I think there could easily be further sub-sub-workflowification - which may or may not be platform specific - that would improve maintainability. Depending on the number of tools in each box, you might want that bit to be separated - short read adapter removal could also be its own subworkflow, for example. Other bits might not be platform specific - host removal might be something easily shared across long and short reads with the same tools. Otherwise, the only comment I would have is whether run merging is a necessary component, or if that’s something that needs to be flexible enough that it should be handled by a pipeline. Because otherwise you have to start making assumptions (here’s my wee bugbear again 😇) about the contents of the meta object…

Yep, I like the idea of sub-subworkflows. Maximum modularization

Yeah, Vangelis and I thought about that yesterday (About run merging) too... that and hsot removal

Pretty nice. My only comment is that adapterclip_merge has loads of tools that are for the same purpose. I agree with James that every team has a favourite tool for this but it adds loads of complexity to the subworkflow (and maintenance burden). Borrowing from classic programming langs, could we build the sbf with one tool (or 2) and define some kind of interface/API/conventiont so people can swap tools?

I dont think so, as the outputs channel structure of each tool might not be exactly teh same

yeah, to maximise adoption you need to have people favourite tools

Precisely

also.. my gut feeling is that most of those tools are easy to maintain (no massive consumption of resources, massive usage by the community)

I dunno, we try and see what happens 🙂

yup... I will think about my plug-and-play thingy anyway drevil

e.g. keep adaptermreoval

could start with these, yes

Pretty much