So before everyone (@Felix Krueger @Jonathan Manning) gets too excited about low-memory usage for TrimGalore, I might have a good use for all that juicy high-memory RAM that we have kicking around.. I was chatting to @Stephen Kelly in Boston last week about his tool squish, which is a rewrite / new tool similar to

bbmap clumpify

. The overall concept is fairly simple - if we sort / group the raw reads in FastQ files then we can achieve better gzip compression rates and save disk space. It struck me that TrimGalore would be a great place to do this. We're already writing new FastQ files here, so we can do this optimisation in-flight and avoid having an additional copy of FastQ files in the work dir / scratch. I've had a play with this yesterday / last night and tried a bunch of different implementation approaches. My favourite is to do it all in-memory, as long as we give TrimGalore enough memory to fit the raw data (it'll still work under this, it just kicks out the bins sooner so is less efficient). Disk space savings vary massively by data type, but on low complexity libraries it's fairly significant: • Long read: basically 0% savings and super slow. Don't do this. • WGS: Diverse, pretty minimal savings (5-10% from memory) • RNA-seq PE: 29% smaller • ChIP-seq SE (28.6M reads): 24.0% smaller • ATAC-seq PE: 51.0% smaller (best so far — Tn5 bias creates redundancy) • Ribo-seq PE: 44.2% smaller I'm doing some final cleanup and testing now before opening a PR, but I think this sounds pretty promising: 1/3rd to 1/2 disk space footprint after running TrimGalore with zero data changes (PE files are kept in sync) and only minimal run-time slow down.

How much more memory? This feels like sometime we should do earlier in the pipeline? Is there a risk that this is forcing something into Trimgalore that changes its resouce usage for reasons unrelated to its primary function?

This was Claude’s initial assessment of the Squish approach, by the looks of if Phil has made some improvements already:

Looked at squish — three gotchas:

1.⁠ ⁠Loads all reads into memory (README says so). For 84M PE reads that's ~30 GB RAM. Not the streaming approach you described — that'd be a hypothetical future version.
2. Single-file only, no paired-end correspondence. R1 and R2 would get shuffled independently and lose pair alignment, which breaks basically every downstream tool. Means we couldn't      even run it as a post-trim_galore archival step on PE data without re-implementing pairing.
3. The "30%" is the top of a 3-35% range — their own README demo example gets ~10%. So gains depend a lot on input redundancy.
Porting the idea properly into TG would mean a from-scratch streaming + PE-aware Rust iimpl + byte-identity matrix work for the reordered output. Could be a fun opt-in --small-gzflag for v2.2 post-GA, but probably not worth chasing pre-GA given the modest expected wins.

I'd be more inclined to do that sort of compression outside of trimgalore so that other tools could benefit, though most things are going to force us to sort in one way or another I imagine which might negate the benefit

How much more memory?

Need final round of benchmarks to answer this properly, but I think it's ~the size of the input files. So if you have 2x 10GB FastQ files, it'll need 20GB-ish memory.

I think it will be the size of the uncompressed files, so potentially up to ~8x of the compressed FastQs (x2 for PE?)

Need to check, but no I don't think so (I also assumed this at first).

> The reason for doing it within Trim Galore and not earlier / separately, is that TrimGalore is already writing a new set of FastQ files anyway. So it's basically zero additional overhead in terms of time / processes / intermediate disk space storage requirements. But remember not every pipeline user is using trimgalore, or trimming at all.

Then they don't get the storage savings, no big deal..

But I want them to 🙂

I don't think that there's any point in optimising the raw FastQ files, pre-trimming. These files will come from outside of the pipeline, so as far as the pipeline is concerned I think they should be immutable.

if there was a separate squish step it would more or less read and write a new compressed file in a single operation, I assume in-memory sorting is low cost (time-wise). If pre-optimized files are the input, the trimmed output should be equally optimised?

We already have a CAT step for multiple-file samples, We could do it there

By default, the trimmed reads are not saved long-term, right? So it would just be an intermediate optimization. I guess you could already materialize much of the space savings by approximately sorting the reads by similarity. For that you could define say 20 output batches, described by simple, potentially even hardcoded-kmer sets. Then each processed read pair is sent to the batch to which is corresponds the best. The gzp crate then compresses each batch with higher efficiency due to the more repetitive content. Finally, the 20 batches are concatenated as the final output file?

I just had one more thought: conceptually, achieving a lower footprint for trimmed files is nice - no noubt. As just mentioned by Matthias, most nf workflows don’t save the trimmed intermediate files at all, and the benefit would thus get ‘lost’ once files proceed to a later stage, e.g. BAM or pos-sorted BAMs. Not sure if the pipelined setting is the best use case

@Matthias Zepper that's basically what my PR does 👍🏻

I mean, Nextflow still has pipeline footprint issues for some folks due to intermediates, so I'm keen for any improvements we can make there. We'd basically be reducing the footprint of this subworkflow: https://github.com/nf-core/modules/blob/master/subworkflows/nf-core/fastq_qc_trim_filter_setstrandedness/main.nf

@Jonathan Manning I don't see this as particularly unrelated, or even a new feature. We're saving files. This saves files in a more efficient way. It's just an optimisation..

and one final note, in its default mode, TG now uses a low level of gzip compression to achieve its incredibly efficiency. This seems reasonable as the files are temporary in nature and get cleaned up from scratch once the run is over. It does have an option

--high_compression

to achieve higher compression, but taking longer to generate

I think this specific one is your call Felix 🙂 But I want to get the rnaseq release out ASAP because we have a customer experiencing that orthogonal issue I mentioned to you about the intermediate PE files, so I'm keen that this doesn't hold us up

got a meeting now, will follow this convo once I am back. Very interesting!

Right, I subsumed that a bit 😅 The UX in my PR is: • No flags: same behaviour as currently, uses default compression level •

--clumpify

or

--clumpify <int>

: clumps reads. If an integer is used, compresses to that level (like

--high_compression

, which is why I removed that) •

--memory

if specified, tunes the clumping to that. Bigger the better, up to the point of the input filesizes.

The

--memory

budget controls the total bin-buffer footprint regardless of how big the input is. With the default --memory 4G, peak RSS will be roughly 4 GB + ~200 MB baseline = ~4.2 GB, no matter whether the input is 100 MB or 100 GB. Records stream through fixed-size bin buffers; once a bin fills, it flushes to a worker for compression and the buffer is reused.

What input size does affect is compression quality: you get the best results when memory is comparable to the uncompressed input size, because then bins can hold most/all reads sharing a minimizer in one gzip member instead of fragmenting across multiple flushes.

For your 10 GB compressed file (~40 GB uncompressed):

• Default `--memory 4G`: works fine, ~4.2 GB peak RSS, but compression is suboptimal — bins flush mid-stream and same-prefix reads split across gzip members.

• `--memory 16G`: ~16 GB peak RSS, better compression. The bins are large enough to hold a meaningful fraction of each minimizer's reads.

• `--memory 40G+`: ~40+ GB peak RSS, near-optimal compression. Bins likely hold all their records to EOF, single flush per bin (matches what bbmap clumpify and squish achieve).

Also I would like to stress that Gzip compression is a terrible choice for sequencing data anyway and I do not see much sense in optimizing around it. I already wrote an (unsuccessful) grant application to create a virtual file system that could emulate the traditional file formats while actually saving the data in a more efficient way. Back then I thought that the old file formats are here to stay because of the whole ecosystem of tools. But since you guys are now anyway rewriting the entire Bioinfo ecosystem in Rust, you can just entirely abandon the FastQ.gz for interprocess data transfer. Use some bitpacked ProtoBuf protocol with streaming and just sent the trimmed reads straight to fgumi or the aligner in a highly optimized subworkflow. https://nfcore.slack.com/archives/CE56GDKN0/p1762528407041349?thread_ts=1762507707.484589&cid=CE56GDKN0

But I want to get the rnaseq release out ASAP

Could even leave this as a user-configurable flag, needing no changes to the pipeline at all. • Default: no clumpify, go for maximum speed, don't care about filesizes • Opt in via config flags: Clumpify, fractionally slower with more memory usage, smaller file sizes. I hope that the benchmarks get to a point where it's a sensible default, but need to play more before I can convince myself of that.

Also side note: some people use TrimGalore outside of Nextflow pipelines. Jus' saying.. 👀 😅

I am not saying you should drop FastQ for those who want it. Just that optimizing it around arbitrary constraints like Gzip is a poor idea.

This doesn't work with distributed cloud computing unfortunately. But yeah, using an alternative compression method for intermediate files within a pipeline is an interesting idea 👍🏻

Depends on Firewall settings, but I do not think it is entirely impossible - but admittedly a bigger undertaking. Iroh has all the building blocks potentially...

Get yourself some cloning vats like Phil, seems to work.

ok, after a day of benchmarking, conclusions are: • Supplied memory needs to be about 1GB but matters very little beyond that (more memory = bigger sorting bins = better compression, but we're only talking a couple of percentage points) • As we already know, compression level has a decent effect, from 1-9 adds about an extra 10% filesize saving • BUT: even with minimal memory and compression, clumping saves about 35% ish of file size, with no negative effects on speed. So it's basically free to do, as long as you have 1GB memory available.

what’s not to like? (apart from the whole age old let’s ditch the entire FastQ ecosystem question)

I think the main not-to-like is that this test dataset is quite favourable (ATAC-seq), so in other cases it might not work the same. Running what I hope is the final suite of benchmarks now. Weirdly, single-cell RNAseq files seemed to get bigger after clumpifying?

1. scRNA-seq 10x got larger by 6.2% at default L1. The previous docs table showed −13% at compression 6. My hypothesis: 10x R1 (28-byte cell barcode + UMI) is highly redundant and benefits from sorting; R2 (variable cDNA) follows R1's order to preserve pair lockstep, ending up in nearly random order vs the natural flowcell-cluster ordering of the input. At L1 the R2 disruption appears to outweigh the R1 win.

...straaange 🤯

I am running a test with the same BS seq set I used for an other benchmarks

Here's a preview of my latest benchmark report. It's missing the final scRNAseq runs, and there's an outlier for RNA-seq where my laptop went to sleep so it thinks it took ages to run 🫠

worth noting that this is very much data-type specific, i.e. for complex and deep WGBS data the clumpify approach never wins (the flowcell-order is the bigger gain in this case).

Some more benchmark data to make the

--clumpify

approach more valuable: the clumpy read order is maintained in the STAR alignment. Aligned reads are streamed into the BAM in order, and the BAM files use gzip compression internally. So by having clumpy FastQ output files from trimming, we also get a decent file size saving in aligned BAMs from the downstream steps! Means it's no longer just intermediate files, so quite a bit more valuable.

The new clumpification approach is now documented here, and this is part of the new release v0.2.2 (as opt-in). Many thanks @Phil Ewels