I'm planning to create a rapids-singlecell module to GPU accelerate a few pipelines that use scverse tools like scanpy. However, there isn't a single scanpy module but rather a few that each provide individual steps like PCA, ... . I'd like to avoid having to create a module for the 50+ functions that RSC provides. I'm already astonished that this is best practice for scanpy because scanpy offers so much more than filtering, hashsolo, scrublet, and PCA. I know that other tools like neighbors, Leiden etc are used in various pipelines. Is a "master" rapids-singlecell module in scope or how should I go about it? @Nico Trummer Sorry, I'm a Nextflow boomer that last really implemented stuff in 2019 or so when modules were just about to become a thing.

Well, so in general nextflow (and nf-core) are both generally more focussed on CLI-based tools rather than scanpy or rapids Now a module usually comes with both the environment and a bit of logic (for CLI tools this is usually just the call of the respective tool and tweaks can be made via

args

). Now for scanpy I tried to always do one module per function that I needed and the parameters to this function can be passed via args. But there are some annoying things with this: 1. Scanpy has two internal levels of hierarchy (pp/tl/etc and then the actual tool). nf-core only allows one level of hierarchy per tool, that's why it's scanpy/pca and not scanpy/tl/pca 2. Between nextflow modules, the only way to pass data is writing to disk. Scanpy/Rapids were designed to keep data in-memory between processing steps. So we end up with quite a bit of IO overhead 3. The only real alternative to having one module per function would be to have a scanpy module that has the exact function to be used as an input - e.g. something like

SCANPY('tl', 'pca', args, adata)

which could work in certain cases, but then if you need any wrapping logic that is specific to certain functions, it would soon become a nightmare. And for example #C077PUX8DBP has a lot of these, for example outputs that should be passed to multiqc. So I think nextflow and scanpy/rapids will never be best friends, and I am annoyed by this too. Still I have not been able to come up with a better approach than creating one module per tool within scanpy, and I don't have any better ideas currently. But I'm very open to discuss, maybe we can find something better

Yeah, I know how things work here but I was hoping for something nicer. I mean one alternative is to have the modules mostly just be containers that set up some settings but then the actual work is done with custom scripts in

/bin

. This is also antipattern because ideally, there shouldn't be any such scripts. I'd argue they're a "nextflow code smell". Performance wise, I do fear however that having such individual modules that always load & write is stupid. RSC is most efficient when you put the matrix to the GPU once (cupy) and you keep it there for all GPU computations. Every time you read & write, you have to move the matrix to the CPU (and back) which is expensive. I actually think this is an even bigger problem that we need to discuss. Maybe we just need to work with local modules that act as supermodules for a whole set of steps for a pipeline? Maybe something generalizable aka a proper nf-core module is antipattern?

Scripts in

/bin

are not bundled with the module - e.g. if you do

nf-core modules install scanpy/pca

, the script in

/bin

won't be included. That's why we are currently mostly using the

template

syntax, which stores the python in the module directory. In future there will probably also be a slightly different approach (module-specific bin-directory), but that does not really have an effect on our discussion So there are two sides concerning the performance thinking here: 1. Nextflow allows parallelization, e.g. running different doublet detection tools, integration methods, and similar things in parallel. Also it allows to process multiple datasets in parallel 2. Large modules allow using less IO operations, thus better performance for 'linear' workflows Now the idea of official nf-core modules is to make them small, general building blocks that can be shared across different pipelines. With local modules, of course there is more flexibility. But then there is two extremes: 1. Having a single supermodule that basically is the entire pipeline (which makes it useless to have a nextflow pipeline in the first place) 2. Having one module per task, lots of IO, but following the general nf-core structure better and potentially more parallelization Now the optimal solution will probably be somewhere in between. Something where one module contains what in nf-core rather would generally be considered a subworkflow. But defining the boundaries of these modules probably requires a lot of thinking beforehand

I understand that parallelism is desired but for RSC, it's so fast that the host<->device transfer cost is higher than missing out on parallelism. I mean scanpy/RSC pipelines are linear in many cases anyways.

Well, then a nextflow pipeline is probably not what you want

Parallelism is ony one advantage of NF. It's also about reproducibility, portability, ... which weighs equally

Maybe there is a way to make rsc work as a subworkflow. That way we can keep everything in memory until after.

Let me know your thoughts on this, it's currently the thing that sounds most feasible for me

Maybe what could be useful is a CLI tool that sits between nextflow and scanpy/rapids, provides the CLI to nextflow and internally runs an entire subworkflow.

Then, we could implement a shared nf-core module that encodes a single call to that intermediate CLI tool and fully comply with nf-core standards. It could also abstract the difference between rapids and scanpy away

I see, so all the RSC steps would need to be in the same module if they want to share memory.

Exactly. That's why to fully benefit from RSC, we will have to build quite complex modules. Shared nf-core/modules are generally designed to be atomic building blocks of pipelines, and what we will need to do to fully benefit from RSC will be the opposite of atomic. Which would not be a problem for pipeline-specific (local) modules in #C077PUX8DBP, but in order to contribute a module to the shared #CJRH30T6V repo, stricter rules apply. And I guess to make RSC fully useful for the nf-core community, the goal will be to make it available in #CJRH30T6V. To make this possible, my proposal is to keep the modules atomic by moving the complexity elsewhere. So I guess one usecase of RSC would be: Given an embedding, compute the neighborhood graph, do clustering and create a UMAP. Now we could implement a CLI tool that allows something like this:

scverse clusterumap file.h5ad --rapids

. Then we could build a shared, atomic nf-core module that does all of the desired actions in-memory. And with the

--rapids

flag it could work with both scanpy or RSC as backend

Let me know your thoughts on this, it's currently the thing that sounds most feasible for me

> Maybe what could be useful is a CLI tool that sits between nextflow and scanpy/rapids, provides the CLI to nextflow and internally runs an entire subworkflow.

> Then, we could implement a shared nf-core module that encodes a single call to that intermediate CLI tool and fully comply with nf-core standards. It could also abstract the difference between rapids and scanpy away

https://github.com/nictru/scanpy-cli in a way? I guess that's why you partially implemented it at some point. It wouldn't be the nicest interface but it would probably work.

Well, what I imagine would be one level higher in abstraction. The scanpy cli I built still has one CLI endpoint per function. What I imagine here would be basically one CLI endpoint per in-memory workflow, potentially spanning many functions.

Yeah, I got it. That's why I wrote "in a way" haha

Given that we're talking about like 3 pipelines here at the moment, I am somewhat tempted to just implement local modules that do just what we want them to do

That's probably the most pragmatic way

I wonder whether we should ping some nextflow devs to make them aware of this design issue? Maybe it's something for the future roadmap. Sharing resources between steps

Now the timing is a bit suboptimal, as I was planning to do a 1.0 release of #C077PUX8DBP soonish, and this is probably a major overhaul Also there is still some things that need consideration: • How is the scanpy/RSC handling implemented? How can we keep the same featureset available for both CPU and GPU users with a minimal amount of redundant code? • Where are the supermodule boundaries?

That's a discussion we can continue in #C077PUX8DBP in a new thread.

I wonder whether we should ping some nextflow devs to make them aware of this design issue? Maybe it's something for the future roadmap. Sharing resources between steps

I can't imagine this to be possible, because it would require having shared memory between different containers (and this does not mean just docker, but any supported containerization engine) and also one would need to be able to influence how tasks are assigned to machines (and if I understood AWS batch correctly, a new instance is launched for every task). Also the resuming might be weird. Well I see a lot of obstacles here and I think nextflow was just not made for this, but I am not a nextflow dev. Having their opinion would sure be interesting. Maybe @Ben Sherman

I agree that there are significant obstacles but if there's a real problem, there can be a real solution.

I have long considered doing some nextflow magic to fuse process calls into a single process to get the best of both worlds. Maybe you could even use pipes to connect outputs to inputs without disk IO. But at beat that is still IPC which will not be as fast as shared memory So my inclination is also to just write one big process. You really only need separate processes if you need different resources for each step, which is not relevant if the goal is shared memory I also agree it can still be useful even as a single step pipeline (containerization, portability, reproducibility, lineage, etc). With 26.04 you don't even need the pipeline -- just run the module directly with

nextflow module run

Nextflow module registry also makes nf-core modules just one namespace among many. So you could share these modules via a different namespace if they are worth sharing but don't fit nf-core requirements