Dear all, I am new to nextflow, but am enthusiatically writing my own modules for modular custom pipelines. When it comes to bulk and single-cell multiomics, tools are divided between R and Python, raising the question of input file formats. For example, nf-core modules exist for differential expression with limma, edgeR and DESeq2. These consume tsv-files and are therefore language agnostic. Then the decoupler-module does the same. But I use decoupler with single-cell datasets, which are large, sparse, and shouldnt be converted to tsv or csv files. Processing for multiomics may require the aforementioned differential expression in R, but processing before and after is preferred (by me) in muon, and intermediate processing steps may again be used from R. I let muon modules emit h5mu or zarr files, which then require conversion or extraction steps for several tsv/csv files (expression, metadata), or back from several files to anndata/mudata, which works for bulk data, but not large single-cell matrices. How may I handle this better, and in a way that will hold up modularity with other existing (limma, deseq2) and future nf-core modules, i.e. , muon, sva, RUVSeq, and more? (looking at the mlomix proposal using some of these) Modules and subworkflows for common conversions adn extraction like nf-core/anndata?

Not sure if I fully get what you are going for, there are database formats that you can use with R and python, e.g. arrow or duckdb. I am also not sure why those "shouldnt be converted to {t,c}sv", is that simply because it is inefficient / takes space?

Thank you for your reply. Yes, inefficiency and storage are part of what I meant, saying it "shouldnt" be converted. I was thinking of exporting a sparse single-cell matrix as a dense expression table, which exceeds my available memory in the dataset sizes I work with. My main question was more about interoperability between modules. In my custom pipelines, I’d like to combine Python steps using AnnData/MuData with R tools and existing nf-core modules. That often means extracting expression and metadata, then reconstructing an object for later steps. Arrow or DuckDB could help with exchanging tables, but I would still need to handle the mapping between those objects and the inputs each module expects. Would reusable extraction/conversion modules or subworkflows for example between AnnData and SingleCellExperiment, or for preparing inputs for existing modules, be useful in nf-core? Or is the usual approach to keep those adapters specific to each pipeline? (There is the nf-core/modules/anndata module which handles conversion from anndata to Seurat and SingleCellExperiment, but from what I have seen not the other way around. Is this something that nf-core would like to handle in conversion modules? Or how would you do it?

Ok i see, i misunderstood. I think @Nico Trummer might have a better idea if such modules are useful than I do

Thanks for tagging me, this is a central challenge in #C077PUX8DBP. Or I should rather say 'was', because since anndataR has been released in bioconductor, it is pretty convenient to handle now. Basically the way I do it is I keep it as AnnData (h5ad for now, maybe zarr in future) all the way through the pipeline. For the R modules, I read it from the AnnData, convert it internally to SingleCellExperiment/Seurat/whatever and then in the end store it as AnnData again. This made dedicated modules for format conversion unnecessary for me. The existing nf-core module for the conversion from anndata to Seurat/SCE was introduced back when conversion was still more problematic. It is not used anymore in #C077PUX8DBP. Before I write too much here, maybe you just want to have a look at how the things you mentioned in your original message are done in #C077PUX8DBP. A lot of things you mentioned are there: R/Pytion handling, decoupler, edgeR (or rather edgePython), deseq2 (or rather pydeseq) and many more things that are adjacent to what you are writing

Ah awesome, thanks both! @Nico Trummer, this is the exact confirmation on conventions I was looking for. I just took a look at #C077PUX8DBP and saw how the new scdblfinder module uses adnnataR directly on the .h5ad input rather than relying on R-specific formats or intermediate files. That's so much cleaner than having separate conversion steps. Good to know too about using edgePython and pydeseq to keep things unified. Thanks a lot for pointing me there.

Feel free to recycle my modules. I did not find the time to put them all on the shared #CJRH30T6V registry yet, you can still copy them