Hello there! Help needed! I have a very general question: what is the best way to deal with out-of-memory (OOM) image processing/analysis? These days, I believe a lot of Nextflow users are also working with spatial genomics data, which can get out of either RAM or GPU-RAM memory pretty easily. I know Janelia (@Konrad Rokicki @Cristian Goina ) has a very promising solution which runs cellpose as a subworkflow with 3 steps: 1. spin-up dask/spark cluster 2. pass the server address to cellpose wrapper which distribute each chunk to each worker and then merge the data 3. shutdown the cluster This is a great solution but if we'd like to extend this template to other segmentation methods, the amount of work can be daunting. Since a new wrapper will be needed for each methods. Thus, I am wondering is there a solution that scales and doesn't lead to the overhead of using dask/spark?

cc @Jeremy Muhlich who has been thinking on this during #C047HKB4RPC development

I think a general solution is not possible because the approach for merging the chunks is going to be different for each tool, or at least for each class of tool. (e.g. instance segmentation as a class of tool, with many implementations)

many thanks @Jeremy Muhlich and @Adam Taylor

the approach for merging the chunks is going to be different for each tool, or at least for each class of tool.

exactly want I think it could be! the there's a finite number of class, right? We can then define one template for each type of class. So, I'd say still doable? I've got this module for merging segmentation outlines (https://github.com/cellgeni/nf-modules/blob/main/subworkflows/sanger-cellgeni/tiled_segmentation/main.nf#L48), but the merging is not perfect, as you rightly mentioned. I believe @Quentin Blampey’s work in SOPA has a cleverer implementation. and another one for merging points in WKT format on spot data (https://github.com/cellgeni/nf-modules/blob/main/subworkflows/sanger-cellgeni/tiled_spotiflow/main.nf#L25) For the future, this list of classes could be potentially extended to: 1. image-in/image-out type of module? a. chunking (2D-5D?) b. applying local image processing c. returning locally processed tiles d. merged the processed images i. concatenate if no overlapping (much easier) ii. different merging strategy for different types 2. image-in/object-out type of module? a. chunking (2D-5D?) b. apply object extraction/enhancement(?) c. returning locally processed object d. merged the processed objects. This can be application specific. Off the top of my mind, this covers quite some use-cases already in our case. Do you have any other senarios?

The tools usually have a substantial start-up time, so with a separate pre-chunking step you'd be incurring that cost on every chunk instead of once. Also you'll need to store a duplicate copy of the image data in the chunked layout, albeit just temporarily in the work dir.

> The tools usually have a substantial start-up time This is true for deep-learning based tasks with model loading etc.. But will be much faster for simple operations, e.g. local peak-calling. > you'll need to store a duplicate copy of the image data in the chunked layout Without modifying the tool, yes, I agree. That's why we always add a wrapper that simply loads the tile to process and then pass that in-memory in the same module instead of two modules and on-disk serialisation. Overall I think we are inline that whichever tool we need to work on. It will be better if they support running on a subset of the image instead of the whole image. Is this what you meant by modifying here? > modifying each tool is a better approach

Could the chunking be done by transforming the images into zarr and then, within the tools, implementing the parallel processing of zarr chunks…?

That's exactly what I propose. Tiff files can be internally chunked as well and accessed just like zarr, which is what I did in my own cellpose chunking wrapper.

Ah, okay, I see, thank you. I think we wanted to use this with ome.zarr (@Luis Kuhn), not sure if this is the same as the tiled storage (i’m not very experienced with all the file formats)

I think microscopy tools should be able to read and write both formats, ideally.

so, if I understand correctly. (for python developers) It could be as simple as asking developers to support some standardised IO pakcage (bioio, ome-types? etc), which already support n-dim subsetting?

For the file input, yes!

I am actually thinking not leaving the merging task to the tool developer, but to the pipeline developer (us). As long as they save the processed chunk data in a standardised format (WKT, label image, ome.tiff etc). It will be useful for their testing anyway as well.

> so, if I understand correctly. (for python developers) It could be as simple as asking developers to support some standardised IO pakcage (bioio, ome-types? etc), which already support n-dim subsetting? Could it be done with an interface, instead of requiring a specific package?

what do you mean interface?

I think zarr and tiled tiff are the interface

Totally agree. I guess I’m thinking there there might be room to create a higher level construct, and I’m not sure if this exactly maps to what Tong is looking for, but if it could be defined in a general way you could imagine being able to plug-in different image processing tools and different merge/reconciliation algorithms. In a sense this sort of programming model is defined by the map-reduce paradigm, but this is a bit more specific because some algorithms require information from adjacent tiles. There are tools like Daisy which try to take all this into account, but many people just use Dask which can be very tricky to do correctly because the obvious paths (e.g.

map_blocks

) are extremely inefficient. From the perspective of workflow management, I think there are some different ways to think about this: 1. The tool does all the parallelization internally, we don’t need to do anything, just invoke the tool with enough CPUs/GPUs. In this case, the “interface” is the input/output file format. This is simple and elegant but it’s limited to a single node’s memory/compute. If you don’t need distributed processing, I think this is the way to go. 2. The tool does all the parallelization, but relies on an external task scheduler (this is our approach for tera-scale data, which Tong described in his original post). In this case the interface is the file format and some (optional) command line parameters describing how to connect to the cluster. If the cluster parameters are not provided, the processing reverts to local as in (1). 3. The tool provides chunk processing, and there is some general-purpose framework at the workflow level that uses this as a building block (along with a merge algorithm) to process an entire image in a distributed manner. @Tong LI does this drive at what you were thinking about? 4. Are there other programming models that should be considered?

Thanks @Konrad Rokicki for the list. That's pretty much what I have in mind as well. I'd say some tools can do

1

. But maybe not in the most efficient way. Very few, if not none, can natively can do

2

? And yes,

3

is what I meant. because from a pragmatical point of view, I believe it's the easiest to achieve. For

4

, I believe some Java/Fiji support is needed?

It depends on whether the natural chunk size is larger or smaller than the native chunks and whether the boundaries align vs. overlap. There isn't a one-size-fits-all solution, rather best practices and common strategies.

@Jeremy Muhlich agreed. What's why I think having the tool supporting a flexible subsetting/slicing will make the task much easier. Or as what we all are doing, adding a wrapper on top of the tool that enables slicing and then call the algorithm on a chunk within the wrapper.

Here's my overlapping-tiles cellpose wrapper, in case it's helpful as an example: https://github.com/labsyspharm/mccellpose/blob/25a1471e16cd8ae6f2f4a8a9fa09ef5227320585/mccellpose/cli.py