Hi all! Inspired by something I just saw in #C0905QCFTPE... what are your preprocessing steps you do for reads for any time of DNA based meta*omics analyses? Please dump in the thread below 👇

For PacBio HiFi reads, we just run adapter removal - don't really do anything else. Will occasionally do some kind of mapping+filtering in complex cases with minimap2

(also this might be relevant for you @Nick Waters and 'may' save you some time)

Usually we don't have a genome for the host!

"Our" pipelines (#C02FC6VFQG1 and #C01U1E95JQ5) use the template provided Trimmomatic plus bbduk to remove anything the user specifies. Mostly meant for rRNA or other stable RNA species.

Template povided trimmomatc? What do you mean by that?

The QC we got from the template when starting the projects.

There is trimmomatic in the template?!

Sorry, Trimgalore.

OK I don't remember that ever being there... but I own't go down that rabbit hole to keep this thread on track 😬

If there are shared subworkflows among pipelines, please consider pushing to nf-core/modules 🙏

Shhhh

(I wanted to see how consistent they all are, and if a good proporation of our pipelines are doing basically the same thing, it could be a common project to build a subworkflow for that and then we replace it across all our pipelines)

We are also considering pushing some monstrosity modules (chained modules that are more efficient than calling them one by one) as I've also seen some of those in the past.. we'll talk about this

In our pipelines (not-nf-core) we use fastp for basic qc stuff, we found it much nicer that trimmomatic (also fastp is maintained). There is also a fastp for long reads. We have to some extensive QC on reads mostly, people submit all sort of horrible things to public archives. Chris from our team even coded a tool to try and get the primers used as this info is often missing - not cleaned from the public archives -- https://www.biorxiv.org/content/10.1101/2025.07.04.663168v1 If this is of interest I can ask Chris to present this in one of the metaomics meetings.

What other QC steps do you use?

fastp is the basic one.. depending on the pipeline we do: • https://github.com/EBI-Metagenomics/amplicon-pipeline/blob/main/subworkflows/ebi-metagenomics/reads_qc/main.nf - ◦ some seqfu to try and catch plainly wrong fastq files ◦ a whole subworkflow to trim primers (we use DADA2 - so we need to be certain we are getting the "right" hypervariable region) • https://github.com/EBI-Metagenomics/miassembler/blob/main/subworkflows/local/short_reads_qc.nf ◦ we remove phyx (used in illumina to calibrate).. and we remove human data (this is probably applicable to us because this is more for "regulatory" reasons

Would you be willing to join an nf-core effort to standardise this, or are your own using too many custom modules?

I think it would be useful for us too, we can customise things in our end. I'll see who else from our side is willing to join, it would be a great learning opportunity for other people too. So, yeah.. we can join

Oh don't worry this is not under any time pressure

perfect

Partly inspired by my idea from 2024: https://github.com/nf-core/modules/issues/4906

I have a viral metagenomics pipeline for detection of known and novel viruses (emphasis on eukaryotic viruses). If interested see this pollen virome paper https://www.nature.com/articles/s41467-022-28143-9. Unfortunately, the pipeline is difficult to setup and run since it is mainly configured to work on my local HTC and my coding skills are subpar. The pipeline can take fastq reads (unfortunately, no paired end support) or a BAM file from which unmapped reads are extracted (BAM file support came from my work on TGCA BAM files for viral discovery in cancer) The pipeline first runs cutadapt with a rather long set of adapters to handle the many library preps I've encountered over past 15 years. I use options

-n 15 -O 5

Then I run prinseq-lite.pl with several options like low complexity filtering (entropy method) with cutoff 60, length cutoff 30, and quality cutoff 15 for average quality filtering and trimming of the ends of reads Then I remove host reads by serially mapping the reads against one or more Bowtie2 indexes. I use bowtie2 with the following options --very-sensitive and --local (the goal is to reduce host reads as much as possible for downstream mapping and assemblies). At this point, the processed reads enter two parallel paths 1) mapping to NCBI Viral Refseq with bowtie2 for identification of known viruses and 2) an assembly pipeline followed by a BLAST pipeline for identification of known and novel viruses.

ampliseq: fastqc & cutadapt (required, no other tool suitable) detaxizer: fastqc & fastp (any other tool might be fine as well)

For non-amplicon sequencing, we do duplicate removal. this seems to be omitted from most microbial workflows but with the extremely high duplication rates from many modern sequencing machines I don’t know why its not more common. Its core to human genomes workflows with GATK, for example. We use clumpify from bbtools rather than fastp cause it can be configured to catch the types of duplicates found in a given sequencer. I’ve searched here for threads mentioning deduplication and it looks like it has been discussed previously but I’m curious what pipelines the logic has made it into. Other than that, we use standard qc steps like fastqc, quality trimming, host depletion. We’ve noticed low-complexity reads sneaking though so have been interested in implementing some sort of entropy/complexity filtering but I haven’t yet.

Oh, that is a good point - for our PacBio read sets, we do run deduplication when we have PCR-amplified libraries (for example, ultra-low input libraries). This happens with

pbmarkdup

that is very interesting. This reminds me of one aspect .. benchmarking different approaches, should we do some while trying to standardise?

I think there have been various benchmarking papers on whether dedupping is worth the computational overhead and it wasn't always necessary for some contexts (because the amplification was normally roughly uniform (enough) across everything)

This paper suggested that the time spend doing duplicate removal was worth the speed, memory, and storage gain for downstream assembly/binning. That said, the figures mostly talk about the speed gain vs raw data, rather than including the actual time spent deduplicating haha!

Exactly

Do we not need to have multiple shared subworkflows? Can imagine that we would certainly want long/short read separated?

I can in a pinch but I do have a backlog of funcprofiler stuff to sort out as well as our workflows for the in-house demuxing — does anyone else happen to have more bandwidth? If not I can coordinate but progress might be slower than optimal 🙂 Do we have opinions about the order of tool execution?

Yes, I'm envisioning short read first as that's the most common and the someone else could do long read one

Yes, I was planning on getting taxprofiler’s preprocessing modules/subworkflows incorporated as a subworkflows in funcprofiler

Ok but I guess what I'm suggesting is just to take the taxprofiler skeleton and make the offical subworkflow

Gotcha yes, so splitting out all taxprofiles preprocessing logic into a new nf-core workflow, and amending with everything mentioned here?

Or wait @Evangelos Karatzas, why did you ping @Ekaterina Sakharova before?

Yes, I'm envisioning short read first as that's the most common and the someone else could do long read one

But also, might we want to separate e.g. adapter removal and quality/duplicate filtering from host removal, also? (the latter could come as a subworflow-ised version of detaxizer)

Is this something on your radar @Ekaterina Sakharova?

samplesheet - to - samplesheet kind of thing

But otherwise theoretically yes could have host removal separately

We were discussing this topic just the other day, for our various pipelines' QC harmonization

Would it be sufficient to write some community-approved workflow.oncomplete logic to spit out the appropriate samplesheet for the subsequent steps? Like have a preprocessing workflow write a detaxize samplesheet, and have detaxize write a taxprofiler/mag/funcprofiler samplesheet? It would keep the configuration out the core repos at least

There is core nextflow functionality coming towards that, but we did/do have our own method in #C070369GP7T that stalled due to a nextflow bug. We may pick that up as a project to reboot during the Barcelona Hackathon though (as discussed)

The plan is -> check if subworkflow already in nfcore -> if yes, use that (might patch minor .diffs) -> if not, use our own -> if more than one of our pipelines use it, then standardize that and also push to nf-core

I see, but what about jump to last one straight away and just have the community work it together from day 1?

Definitely, that's the reason you started this thread anyway :p

Then do YOU guys want to lead the development as per https://nfcore.slack.com/archives/C070369GP7T/p1755530357704489?thread_ts=1755514158.328599&cid=C070369GP7T ?

It doesn't have to be one subworkflow, we can have various with minor differences like we currently do I guess. Just to make this clear, we are talking about subworkflows,right? Not about a qc pipeline

Yes subworkflow

So this would results in the addition of two subworkflows to https://github.com/nf-core/modules/: something like microqc_short_read and microqc_long_read, and we encourage pipelines to consider swapping out their ad hoc qc steps for them?

Can we first spot the existing ones? 1. FASTQ_TRIM_FASTP_FASTQCL: https://github.com/nf-core/modules/tree/master/subworkflows/nf-core/fastq_trim_fastp_fastqc 2. FASTQ_FASTQC_UMITOOLS_FASTP: https://github.com/nf-core/modules/tree/master/subworkflows/nf-core/fastq_fastqc_umitools_fastp 3. ?

Wilk Double Check in tomorrow but was not many when I skimmed

sorry I was late for party 😁 I think Martin has already covered most of tools we use. I wanted to add

seqkit sana

might be useful to check broken fastq files

I would add to the list as maybe: FASTQ_CREATE_UMI_CONSENSUS_FGIBIO (removing UMIs)

has nf-core ever considered a massive QC workflow with everything that people use for QC where you can set as input tools you want to use and it will turn on those 😄 sounds crazy I know but having a lot of duplications in subwfs also seems not very useful sometimes

Soooo you may have missed me pinging you a few times in this mega 105 thread: https://nfcore.slack.com/archives/C070369GP7T/p1755514158328599 But that's exactly what I was proposing 😄

ok I seee, I’m new here and not following much yet, but I will get better

it would be interesting to discuss 🙂 I definitely won’t be able to coordinate

What if @Evangelos Karatzas (who I'm voluntelling you) mentored you on it 🤔 (but no pressure 😅 - just it's one of those things where everyone thinks it's a good idea but no-one wnats to take a lead...)

I think we can work together (he has more “free”/nextflow time than me)

Only if she logs in Gather

oh no…

Well you two can hash it out and if you agree @Ekaterina Sakharova then we can chat and give you a road map

lets have a chat first I can’t promise now because I have a lot of work and family 😁

OK no worries - and absolutely no pressure (despite all my messages 😆 I'm just excited)

Let's do thiiiis!! Team MGnify to the rescue 🐕 🔎

Ok well then I interpret this as @Evangelos Karatzas volunteering the to coordinate this initiative. @Ekaterina Sakharova you can help him when he needs 😉

Trap card activated

trap

shared a GIF