Hi everyone, I'm very new to bioinformatics and I'm currently working with the nf-core/fastquorum pipeline for Duplex Sequencing data. I'm analyzing a dataset with approximately 94 million reads from a 376 bp target region containing 10 nt UMIs. The pipeline runs successfully through alignment, but repeatedly struggles during the fgbio GroupReadsByUmi step. A 1% subset of the data (~0.94 million reads) completes successfully, but the full dataset either runs for a very long time or fails due to resource limitations. I've already spent some time troubleshooting memory allocation and runtime settings, and I've been able to resolve some scheduling/resource issues, but GroupReadsByUmi remains the bottleneck. Has anyone encountered similar scaling issues with Duplex-Seq data or extremely deep sequencing of a very small target region? Any suggestions would be greatly appreciated. Thanks!

GroupReadsByUmi groups reads by their mapping position, and since you’ve got 94 million reads crammed into a 376 bp region, they all land on basically the same spot. The group at that position gets huge, and the tool’s time and memory grow far faster than linearly with depth. It’s also single-threaded, so adding more cores won’t do anything. Update fgbio to the latest version, give the step one fast core with plenty of RAM, raise --min-map-q to drop reads and shrink the pileup, and double-check that your 10 nt UMIs are being parsed correctly. A long single-threaded step like this, you end up paying for a big instance while a single core does all the work. I’m soon launching our executor (nf-ccloud) that resizes the container to match the task!

Thanks, this is very helpful! When you say "plenty of RAM", do you think 128 GB would be a reasonable starting point for this dataset? Also, regarding runtime: since groupreadsbyumi is single-threaded, I assume it may take longer with a single core. I was thinking of requesting 96 hours (our partition allows a maximum of 7 days), but I'm also trying to avoid triggering AssocGrpCPURunMinutesLimit. Do you think 128 GB RAM and a 96 h walltime is a sensible starting point, or would you recommend something different?

Yes, 128 GB is a good starting point. The most important thing is to ask for just 1 CPU. GroupReadsByUmi can only use one core, so extra cores do nothing and since that QOS limit is basically cores × time, requesting 1 core keeps you well under it. So 1 core + 128 GB + 96 h is a sensible combo. Before you commit, time your 1% subset on a single core and scale it up. But pad it a lot, because this step gets slower than linear when all the reads pile onto one tiny region. If that maths says it won’t finish inside 7 days, the answer isn’t more time it’s fewer reads (down-sample, or raise --min-map-q) or a newer fgbio. It’s a classic HPC headache, honestly: booking a big-memory node for days just so one core can grind away. Happy to give you early access to our elastic Nextflow executor if the cluster becomes the bottleneck, but plan the above and you should get going.

Hi again! I followed the suggestion and submitted the full dataset run with 1 CPU, 128 GB RAM and 96 h walltime. Unfortunately, the groupreadsbyumi job was cancelled after reaching the 96 h time limit. Increasing the time further is difficult because I then run into our cluster's AssocGrpCPURunMinutesLimit policy. I noticed that the output grouped.bam file initially increased in size, but at some point after the job started its size stopped changing, even though the job continued running until it eventually hit the time limit. For reference, on a 1% subset of the same dataset, groupreadsbyumi completed successfully in 14 minutes using ~2.45 GB MaxRSS.

Most of those steps (alignment etc.) scale with read count but parallelise across cores, give them more CPUs and they finish quickly. The only genuine problem is GroupReadsByUmi: single-threaded, and scaling worse-than-linear on your single-locus data. That’s exactly what your “grouped.bam grew, then stopped growing while the job kept running” shows, it reached the ultra-deep target region and got stuck grinding on one massive pileup. (And at ~2.45 GB MaxRSS, this is a compute wall, not memory so you can drop the 128 GB right down.) Since you can’t add walltime (QOS) or cores (single-threaded), the only real lever is shrinking that pileup. Run 2–3 subset sizes (5/10/25%) and time just GRBU at each. Three points give you the curve, so you can pick the largest fraction that fits inside 96 h and downsample to it (samtools view -s). At this depth you have far more coverage than duplex consensus needs, so you’ll keep plenty of families. Raising --min-map-q, and double-checking the 10 nt UMIs are parsed correctly (mis-read UMIs balloon the graph), both help too.