https://nf-co.re logo
Join Slack
Powered by
I know you all like the `fq` tool we made—I'm not ...
# rnaseq_dev
c
I know you all like the 
fq
tool we made—I'm not sure if this is useful, but, today, the same individuals on my team that made that just released a new RNA-Seq quantification tool that uses a (the?) typical HTSeq configuration. The average runtime is 5 minutes, and it aims to be a drop-in replacement for 
htseq-count
(it's a true quantifier). Though lacking in updated documentation in the README, the repository is here: https://github.com/stjude-rust-labs/atlas.
Another note, though we've done our due diligence internally, this is not yet independently tested or published. So it's more of a "hey, we're doing this thing" comment for awareness, not a "I think this should be integrated into the RNA-Seq pipeline within nf-core".
m
Interesting! I have to take a look. From a data engineering perspective, I am however surprised that you opted for a relational database like PostgreSQL to store the count data in the server. Except maybe in the metadata, there is nothing relational about count matrices. Why don't you use the data structures from Polars for computation internally? And Icechunk would be a nice storage abstraction, although I think the direct Rust bindings are not available yet.
m
That sounds interesting
cc @Jonathan Manning
j
Can you provide a little more detail about how the quantification is done? Is it 'just' counting, as done by htseq-count/ featurecounts etc? htseq isn't really used by the pipeline any more (except for one or two remaining peripheral points), having been replaced by Salmon (or Kallisto).
m
Valid point, my excitement was more about the associated storage solution, since R loads everything into memory. Such a tool could be of interest with really big datasets or when aggregating quantifications across multiple pipeline runs. @Clay McLeod: Also mind that there is already a really fast htseq-count replacement in Rust: https://github.com/NKI-GCF/gensum
j
(and featureCounts, which is basically the same thing)
c
Can you provide a little more detail about how the quantification is done? Is it 'just' counting, as done by htseq-count/ featurecounts etc? htseq isn't really used by the pipeline any more (except for one or two remaining peripheral points), having been replaced by Salmon (or Kallisto).
Yes, it's "just" counting. You can see the commands we used to compare them to get a sense of what parameters were used. This is essentially the default way that we do counting on a t-SNE, and we wanted to greatly speed that up. We're going to also be accelerating various other parts of DESeq2 to make a very fast t-SNE generation pipeline.
Valid point, my excitement was more about the associated storage solution, since R loads everything into memory. Such a tool could be of interest with really big datasets or when aggregating quantifications across multiple pipeline runs.
@Clay McLeod: Also mind that there is already a really fast htseq-count replacement in Rust: https://github.com/NKI-GCF/gensum
Thanks for sharing this, I was not aware of it.
These are good questions about the relational database. The data model required to faithfully communicate the necessary information to do anything non-trivial tends to be largely relationally driven. Beyond that, 
atlas
will aim to do things like storing and serving multiple different transformations of the same dataset and generate a t-SNE against a reference dataset for you. In this way, at least in its current implementation, we felt a relational database was a good place to start.
To be clear, that piece isn't totally fleshed out yet.
My understanding was that the primary reason Salmon/Kallisto is useful was to avoid a wholesale alignment step. That being said, I see the nf-core pipeline aligns the data using STAR before passing into these tools. While I am not an expert on this, I am aware that there are other benefits in their methods that might justify the choice over pure speed—is there a writeup/paper that anyone is aware of that succinctly explains the benefits of using these tools over "just" counting? I can trawl the papers themselves, but it seems like something where the information might be more readily condensed.
m
Of course, I do not know what the ratio of metadata vs count data will be, but if you intend storing and serving multiple different transformations of the same dataset that imho makes an even stronger point for looking into storage solutions for large matrices specifically. Zarr (the format served by Icechunk) for example is quite popular in climate sciences and can store billions of values and hand out chunks/slices of that data both in rows and column segments. If your biological question would be to find a cells with a very similar gene expression signature, a vector database (thanks to the AI boom, there are a gazillion options) might make sense, because they usually come with native integrated similarity search using faiss or something the like. > The data model required to faithfully communicate the necessary information to do anything non-trivial tends to be largely relationally driven. But even for the relational metadata, a graph database would most certainly allow for simpler queries than a traditional tabular relational engine. Are you aware of the multimodal SurrealDB, which is written in Rust? If the data is relatively small, you can directly embed it to your Rust code and otherwise connect via the Rust SDK. It could store both, you metadata and your vectors.
p
@Clay McLeod there was a looooong discussion in #CE8SSJV3N about dropping featurecounts some years back. It should be in the slack history I think if you want to dive into it.
Mini recap here
j
Multi-mapping reads are normally cited as the killer for straight counting, particularly for big gene families. Salmon, Kallisto etc model the counts in (I'm rusty on the 'how' of that), counting approaches typically have to discard them or do uniform assignment, both of which are problematic.
c
Okay, I see. I do buy the argument for transcript-level abundance. I'll try to find the thread to learn more of the "fundamental limitations" to quantification using these approaches—perhaps it's wrapped up in the multi-mapped reads things that Jonathan states here, which has downsides as well (at least, as I understand the approach the Salmon/Kallisto take).
Open in Slack
PreviousNext
Powered by