Testing Picard vs samtools vs sambamba DONE! (this issue) @Phil Ewels, took a few extra days - sambamba had some surprises in store - but here's the full picture. Three-way: Picard vs samtools vs sambamba (same 8 ENCODE samples, 16 vCPU, t=1/4/8/16, 3 runs each). All three tools produce identical duplicate counts on all 8 samples. No divergence anywhere. sambabma is the fastest (x5.5 vs Picard), but requires some tuning. sambamba out of the box OOMed at 32 GB server. Once you add three memory tuning flags, peak RSS drops to 1.4-3.6 GB - roughly 10x less than Picard. The second surprise: sambamba also hit the OS file descriptor limit mid-run (default 1024 isn't enough). Fixed with

ulimit -n 65536

. Once properly configured, it was rock solid across all 192 runs. Also ran a quick I/O test - suspected the thread scaling plateau (both tools gain only ~1.2x going from t=8 to t=16) was network storage bottleneck. Moved BAMs to local SSD and RAM, but got only 8-15% improvement. So the speed and memory efficiency is real, but not automatic: it requires explicit configuration. All data, scripts, and a setup guide for a budget-friendly Hetzner server, including the sambamba config details: https://github.com/WeTheAgents/rnaseq_data/tree/main/phase3_three_way

Very nice analysis! 👏🏻 Thanks for this!

Yeah, the ulimit stuff is a bit of a problem to deal with, I don't see a way of handling that neatly in a way that won't bother users.

yeah my thinking too. But a 1:1 switch from picard to samtools shouldn't add complexity?

Yeah, that could work, and it might be one less container

Does anyone run markdups single threaded? 🤨

IIUC the thing tested here is actually:

samtools sort -n | samtools fixmate -m | samtools sort | samtools markdup -s

Sigh. Rust rewrite of sambamba?? 😆

😄

Or would be nice if we could fix the bad things with it upstream somehow, presumably it's written in C?

Certainly for the immediate future I'm not convinced a replacement is justified. Sorry @Alexander Noskov, it was excellent work!

Sigh. Rust rewrite of sambamba??

Why not. It'll take more than 24 hours thou 🖖

One more note on Sambamba where we encountered an interesting behaviours a while back. There is a chance this has now been changed, and mit might not be relevant for the RNA-seq workflow, but I thought it fits the “you need to consider…” notion: When re-sorting position sorted BAM files by name (the equivalent of

samtools sort -n

it sorted R1 and R2 in a random order, while

samtools sort -n

consistently sorts the read in the R1 followed by R2 order (taking the FLAG into account). This was an issue of methylation extraction back in this issue).

thanx Felix! Got it, will keep in mind

Sambamba is apparently written in.. D? I've never even heard of this language until now 😅 https://dlang.org/

forget about it. I contacted the creator Artem Tarasov, and he told me it was a student's assignment, and he thinks that we should go with samtools. Anyways, gimme a couple of days to test Rust markdups, seems promising after the first night of effort.

Awesome, you're off already 😅

yep. hope to clean my karma a little

And there can be a tonne of edge cases, like @Felix Krueger said

very happy to

Sanity check: are there any other duplicate marking tools that I don't know of? Has anyone else already started a Rust project for it?

Well, Artem pointed at two bigger projects, but it's not a 'segment by segment approach'. He advices to have a look at https://pola.rs/, and they even began a bioinformatic integration at https://github.com/wheretrue/biobear. Project seems to be stalled thou

Note that markdups is something we need to be super careful with in a rewrite. It's more critical than QC for example, as it's directly affecting the data.

Definitely this.

I contacted the creator Artem Tarasov, and he told me it was a student's assignment, and he thinks that we should go with samtools.

Scary how often folks rely on software with this sort of history (cue the XKCD cartoon with the software stack) At the risk of putting a damper on things, I'm not convinced there's room in the BAM/SAM handling ecosystem for a rewrite tbh, these are core things everyone knows and relies on, it's hard to swap out something like that in a widely used pipeline like #CE8SSJV3N without scaring folks. At the very least the burden of proof would be REALLY high, and there would have to be very dramatic gains for the overall pipeline execution time for us to want to adopt it, which I'm not sure is the case here.

Got It. Still fell like pot-committed, so I'd give it a shot anyway. Implied odds are just too big to ignore.