https://nf-co.re logo
Join Slack
Powered by
Rewriting STAR in Rust -> STAR 2.0 <@U01GQAH4X1N> ...
# rnaseq_dev
l
Rewriting STAR in Rust -> STAR 2.0 @Rob Patro & scverse initiated a rewrite of STAR into Rust a few weeks ago which is currently mostly led by James Ferguson. It's supposed to be faithful of course, but also implemented in such a way so that it can be extended with new ideas of Rob that will potentially make it more efficient & accurate. We want to ensure that the new version will be easily extendable and maintainable by the community. Given the size of the task, a lot of the implementation will be Claude guided. A few industry partners are supporting this effort with credits & later potentially engineering hours. This effort will be kicked off officially super soon but the current POC can be found here: https://github.com/Psy-Fer/ruSTAR Given how much STAR is used for various pipelines in nf-core and your recent rustqc efforts, we are wondering: 1. Are there specific pain points in current STAR nf-core usage (memory, startup overhead, output format quirks) that the rewrite should treat as first-class problems to solve? 2. Would anyone here be interested in testing their pipelines with the new STAR version as soon as it's in a good enough state? Contributions are of course welcome. If you're interested in dedicating a substantial amount of time/effort, I'd be happy to put you into touch with the team.
🦀 2
🤩 3
👋 1
neon cat spin 1
m
In my perception, STAR has some reputation of being memory hungry. I have never really looked into that, but if memory could be optimized without massive efforts and relevant slow-downs, that would certainly be advantageous. A probably quite doable and imho useful feature request would be an aligner that accepts input reads already saved in an unaligned BAM file. In terms of storage space, that would be so much smarter (and solve issues with additional information such as UMIs), yet FASTQ.gz proves to be incredibly durable. I bet the majority of #C050TCYSCFM pipeline runs are just writing data downloaded from repositories in BAM format to FastQ temporarily prior to alignment.
j
That would be a great and pretty straightforward feature.
Also hello, I am here now. ruSTAR is going well. Been at 99.9% parity for a while on SE and PE alignment, but just been hunting down subtle edge case bugs.
🎉 1
👋🏻 1
p
Hi all! I just had a go at running 
-profile test
on Platform using a docker image off my fork. Got some of the way there, but it failed due to the #CE8SSJV3N pipeline command using 
--quantMode
Copy code
error: unexpected argument '--quantMode' found
    tip: a similar argument exists: '--runMode'
  Usage: STAR --genomeDir <GENOME_DIR> --readFilesIn <READ_FILES_IN>... --runThreadN <RUN_THREAD_N> --outFileNamePrefix <OUT_FILE_NAME_PREFIX> --sjdbGTFfile <SJDB_GTF_FILE> --runMode <RUN_MODE>
  For more information, try '--help'.
Full command that the pipeline tried to run:
Untitled.sh
I'm trying it with reference genome building at the moment..
j
Quantmode hasn't been enabled yet. I'll add it now. It doesn't interfere with my current bug squashing and algorithm polishing steps
🚀 1
p
Ok I tried to run a genome build but got this (after a few aborted tries):
Copy code
[2026-04-17 01:26:20.311] [ERROR] Error (helpers.c:655): metadata file already exists and is not empty
[2026-04-17 01:26:20.312] [ERROR] Error (exec.c:129): failed to initialize dump metadata
Not quite sure where that's coming from. But it's late so I need to get some sleep.
j
Okay, quantMode is added in the latest push
🎉 1
p
Great, thanks! I'll probably wait for the CI/CD PR to be merged if that's ok, as then I can just grab the new container names (saves git shuffling to achieve the same on my fork)
✅ 1
m
Wow, things are really moving fast here. Awesome work @James Ferguson with the rewrite. Happy to hear that you liked the proposal with alternative input from BAM. This ties into a long-standing distaste of mine for some file formats that predate the genomic area and that I perceive as inefficient. Just for raw reads, there are at least a dozen file formats that provide better compaction for fewer compute than the generic .gz compression. Considering that a typical pipeline will run through multiple de/recompression cycles and write that data back to disk, this is a huge potential to save compute, energy and to easy on your spinning disks. https://nfcore.slack.com/archives/CE56GDKN0/p1762528407041349?thread_ts=1762507707.484589&cid=CE56GDKN0
j
I am one of the creators of the slow5/blow5 files. Trust me, I get it 😅
🙇‍♂️ 3
🙇🏻‍♂️ 1
m
Initially, I assumed that there is no way to change the ecosystem of tools that has grown around those file formats in decades, so my approach was that of a Virtual File System that could emulate classical file formats without storing the data actually in those (I have written unsuccessful grant applications along that line). However, now that it seems that there actually is the possibility (and reality) of all major tools being reimagined and reinvented, maybe the interfaces could also receive a second thought. Since devising file formats is obviously a beast in itself (you seem to know way better than I do), maybe a clean generic internal I/O API is a possibility? Possibly even with a plugin system? Then developers could relatively straightforwardly implement support for their favorite file format themselves. Maybe people will even integrate with network streaming using hts-get or Quic or directly built on PolarsBio/DataFusion etc.
j
I mean, the simplest way to do that is to accept stdin streams.
decouple the data format handling altogether. Let the user mangle the data into whatever format the tool accepts to run.
p
I think a middle ground is ideal - support the most common formats, also support stdin
Low effort, easy maintenance, common formats cater for 95% of folks, still single binary and easy packaging / install
j
Yep. That's what I tend do to
👍🏻 1
Also I love that xkcd commic. It was included in one of the reviews of the original slow5 paper. The funniest thing is, ONT responded by building pod5. Got us to test it before they released it, then ignored every recommendation we gave them, which then all came true. After we got them to confirm the file format was stable, we benchmarked it, and showed how much money people were throwing away using pod5 for basecalling compared to blow5, especially on cloud compute.
😣 1
Man, i would love to see the face of a PhD student looking to build a new tool in bioinformatics using rust read that paragraph 😅 I think they would have a panic attack.
😆 1
You are right, we can always do better, but don't let perfect be the enemy of good enough.
silentdiscreteapprovingnod 1
p
I think that there are quite a few folks in the Rust bioinformatics field working in this area as well, right? eg. Oxbox, Noodles, the other one that I always mix up with Oxbow. Not all quite the same thing as we're discussing here, but I like the idea of letting others work on that problem and just slotting in crates as and when..
j
Outside the scope of this pipeline, but I was always a STARSolo fan, seeing as it does what CellRanger should really have done 🙂 . Assume there are plans to support and drive that functionality too?
Are GPU integrations on the cards as well? So we could combine both RUST and GPU speedups?
The main barriers to adoption in this workflow will be trust-based, since it's basically the core operation of the pipeline and folks have (sometimes justified) mistrust in AI rewrites. I think the antidote to that will be a bank of rnaseq test data samples from the panopoly of types the pipeline handles, showing parity in all of their results across a few common parameterisations.
j
ruSTAR has been built using debug tooling where it traces reads through STAR and ruSTAR to compare the outputs exactly. There are some slight differences in randomness in multimappers, of which one is chosen as primary, but that's not really an error. STARsolo is on the cards, but need the STAR functionality first before extending it. Not sure GPU is really going to make that much difference tbh. But others can try that.
p
On to the next missing argument all the things
Copy code
error: unexpected argument '--runRNGseed' found
Feature request: would be nice to parse all CLI flags and print all unexpected arguments 🫠 😆
(probably actually quite difficult to do given the CLI parsing lib etc? 🤔 )
Checked each flag against src/params.rs. Missing from ruSTAR:
1. --runRNGseed — RNG seed for random primary selection among ties. Not in params.rs.
2. --outSAMattrRGline — read group line (ID:..., SM:...) for SAM @RG header. Not in params.rs.
3. --quantTranscriptomeSAMoutput — transcriptome SAM output mode (BanSingleEnd etc.). Not in params.rs.
Also worth flagging (parses but feature not implemented):
4. --quantMode TranscriptomeSAM — the flag parses (it's a Vec<String>) but only GeneCounts is implemented (params.rs:721). There's no transcriptome BAM output path yet — listed as Phase 17.3-ish
in CLAUDE.md. Running with this value will accept the arg silently but won't produce Aligned.toTranscriptome.out.bam.
Everything else in the command (--genomeDir, --readFilesIn, --runThreadN, --outFileNamePrefix, --sjdbGTFfile, --outSAMtype BAM Unsorted, --outSAMattributes NH HI AS NM MD, --readFilesCommand zcat,
--twopassMode Basic, --outFilterMultimapNmax, --alignSJDBoverhangMin, --outSAMstrandField) is already supported.
Want me to add the 3 missing CLI flags as no-op stubs so the nf-core/rnaseq module at least runs end-to-end, and track the real TranscriptomeSAM work separately?
j
Or just wait till I get there 🙂
Won't be long
I've had a slight regression because i found claude was lazy on a section, that has caused a few edge cases, but it's a pretty bad architectural issue. Taking a moment to get it fixed
p
Just doing the final 
/simplify
before putting in 3 PRs for 3 flags (figured I'd run them over lunch as we're on different time zones)
Feel free to ignore / close if you'd rather handle yourself!
Quick thought: at the moment we have a bunch of annoying complex logic in the #CE8SSJV3N pipeline because of AWS-iGenomes. I set this up years ago and it refuses to die. The pre-built STAR indices there are now super old, and only work with old versions of STAR. So we have all this logic to run STAR with different containers / versions depending on what reference genome is used and whether it's iGenomes or not.. sad cowboy If the Rust rewrite could work with a range of STAR indices builds, including cough some really old types, that'd mean we could strip out all that gross extra logic and have a much cleaner pipeline again. Note that currently ruSTAR seems to be incompatible with recent regular STAR genome indices, which is what brought me onto this topic..
STAR's index has splice junctions pre-inserted into the Genome file (extra 
nInsertedSJs × sjdbLength
bytes). Your extra 1005 bytes = 5 SJs × 201 bytes (sjdbOverhang=100, so sjdbLength=2×100+1=201). ruSTAR's loader (
src/index/io.rs:96
) strictly requires the Genome file to equal 
chrStart[last]
, which doesn't include STAR's SJ-insert padding.
j
claude been having fun with this bug
🥵 1
p
Good job Opus 4.7 dropped today 😅
j
Anyway. Trying to do a faithful port of STAR first, so it's 100% locked in. Then handing the project over to scverse to do whatever they want (or you) to it
👍🏻 1
👍 1
ruSTAR isn't curently at a "stable" state. So testing it now probably isn't ideal
p
Yup, no worries. I was hoping that I could get it to run in the #CE8SSJV3N pipeline so that we could get some results, then use those snapshot results as a validation target to work towards. But no dice yet because of the reference genome thing.
I could generate new refs using ruSTAR, but then I need to mess around with additional container packaging to get samtools in as well, to generate the Fasta index file.
j
A lot easier for me to just fix that
I'll do a sprint on these things after this architecture sprint
p
Sounds great 👍🏻 Yes I did consider trying to put in a PR for the genome build thing but it sounded complex and significant, so I decided against it.
ok I'll give it a break for a few days then (or maybe a couple of weeks or so, as we have the Summit coming up). Shout if there's anything I can help with, or you get to a point where you think it'd be good to try to run though with the pipeline tests again 🙂
👍 1
a
So we go Rust... Inspiring 🖖
w
Hello people! So, I've been poking around the nf-core Slack for a while and currently neck-deep in a rewrite of maxbin2 - sounds related to what's happening here, so I figured I'd say hi
🤩 1
j
okay finally got all those changes ported into dev.
Copy code
SE: 99.989% (8798/8799) — 127 tie-breaking reads excluded from denominator                                                                                                                              
PE: 99.831% (16566/16594) — 176 tie-breaking reads excluded
🎉 1
thanks @Phil Ewels Now I can finish off the last few little things, and I think it should be ready to hand off to Rob and scverse crew
p
Nice! Is this on 
dev
? Or did stuff go into 
main
?
j
This is all on dev that in now working on top of.
👍🏻 1
p
Once it's on 
main
and we have a container build I can kick off a benchmark / validation run..
j
Some went into main. Haven't pushed up stream yet
p
Or could update the CI to build images for 
dev
as well I guess
j
Just give me the weekend to zip through the last few little things and I'll push up to main
Then trigger everything to run there
Then you are good to test (or do it on your fork if you are impatient 😅)
p
That's fine by me! I'm out all next week for the Boston Nextflow Summit anyway, so no rush!
✅ 1
j
22k lines of code later 😅 https://crates.io/crates/rustar-aligner
🙇 1
with an inital commit and everything passing.
Need to talk to the scverse peeps about further development. It has almost everything except STARsolo.
Seeing if they want me to add that or leave it to them
p
Amazing! 👏🏻
How come you renamed it from rustar -> rustar-aligner ?
And how up to date are the docs, such as DIFFERENCES.md ?
How come you renamed it from rustar -> rustar-aligner ?
Ah right, rustar is taken on crates.io
It'd be good to delete the old 
rustar
github packages container: https://github.com/Psy-Fer/rustar-aligner/pkgs/container/rustar (which will just be getting increasingly out of date from now on)
✅ 1
m
How about making the star a starfish instead?
p
ooh I like it!
How's this?
✅ 1
Updated..
m
Nice. But ultimately this is James call and that of nobody else.
j
Yea nah that's sick 😄
was renamed from ruSTAR to rustar-aligner because rustar was taken on crates.io and this seemed an appropriate name
👍🏻 1
everyone says "STAR aligner" anyway
I deleted the old rustar package
🙏🏻 1
p
I'm messing around with a docs website now.. Would be nice to have something a little easier to read
Shout if anyone has any design preferences 🙂
j
Yea I didn't bother making one, because to be honest, that's just not my wheelhouse. Look at the kuva docs to see what I mean. I really need to clean those up 😄
p
I'm the opposite, I learnt to code with web design, so making the website is the fun bit 😅
👍 1
j
I learnt to code writing RATs, keyloggers, and online game anti-cheats (detect cheats other players use, and reverse theml lol).
😄 1
🙇🏻 1
m
I think Codehike is a nice suite to build tool docs because you can nicely introduce additional arguments etc. in a storytelling manner. You can find starters and examples here. Unfortunately it is React. While it is what AI models actually can do best, I prefer other UI frameworks. 🤷
p
I think codehike is nice for libraries / training etc, where you want to build up code. But not sure it's so necessary for CLI tools...?
I usually build with Astro Starlight these days, then customise the theme
m
Fair. It makes sense to stick what you know best. I don’t have a tool that would have a sufficient amount of arguments to justify some interactivity 😄. But STAR has quite a number of arguments (although not as many as BBMap), so having some interactive tutorials in addition to a static reference could be nice. For example having the resulting SAM with the alignment on the side and then showing how setting the flag e.g. introduces secondary alignment positions or e.g. how changing the gap penalty affects the resulting alignment. This is admittedly more of a general lecture about alignment rather than a specific tool documentation.
p
m
It fits the theme of an AI rewrite to also have Claude design building a website in a couple of hours, but it is also quite scary. I don’t know why it is specifically Claude Design that triggers this feeling in me, but perhaps because I am a visual person and seeing code appear or listening to generated music hits different than seeing an entire website appearing…you chose a nice color theme, Phil! (I guess you did, but maybe Claude did also that…) https://nfcore.slack.com/archives/C03EZ806PFT/p1770293403751979?thread_ts=1770044837.930779&amp;cid=C03EZ806PFT
😅 1
p
I chatted to Claude, got it to make 5 different prototypes, then 5 more, then combined aspects from 3 of them into an 11th prototype, then refined that, then applied it to the Astro theme, then refined it some more.
So it wasn't exactly "push button, website come out"
But it was definitely accelerated, and I wouldn't have had the patience to do stuff like the little wavy line animations etc. otherwise
m
I have so far only seen your screenshot, so was unaware of the animations. And yes, I am aware that we are not yet at the “_push button, website come out stage_” - I am still using Cursor, but I have not managed that even for simpler things. But I always question my usage skills in the process, too. I see so many Skills.md shared on GitHub, read about connecting additional MCP servers and notice tools capable of turning codebases into knowledge graphs with automatic subagent spawning etc. that I always assume the really productive people are all having some super-fancy setup that is to blame/credit for actually getting things done. I have tools FOMO 😆.
cattyping 2
j
Pretty much all of that is unnecessary unless you are doing a lot of hands off work, which I don't. When I use Claude it's iterative, building small features bit by bit and testing them as I go, with a lot of tweaking and feedback, and a fair bit of manual code writing as well. Takes me a while to do things but I get better results then just telling it to one shot something (which is almost always terrible)
Currently these tools wielded by an expert creates more correct outcomes than when not. So not out of a job yet 🤣
p
@James Ferguson - minor nit, but any chance you could rename the release from 
rustar-aligner
to something like 
v0.1.0 rustar-aligner
? https://github.com/Psy-Fer/rustar-aligner/releases/tag/v0.1.0 I know the tag has the version number in it already, but having the number in the release title makes it a little easier to see from the repo homepage / releases list etc.
j
Oh I normally do that. Not sure why I brain farted on that one. Will fix when back at PC
🙏🏻 1
done
❤️ 1
p
Thanks!
I'm not working today but let me know if you have any requests on the website stuff and I can take a look tomorrow :)
j
Just trying to get a meeting setup in zulip with scverse peeps to talk about this stuff. might be worth if you come too? (are you on that zulip?)
p
Just joined
party parrot 1
r
Caps-sa branch can build the human STAR index in 5.1G of memory ;P
🥳 3
5 Views
Open in Slack
PreviousNext
Powered by