Hi all, I’m trying to understand how Nextflow should handle Zarr/SpatialData. Zarr allows large datasets to be updated in-place without rewriting the whole object, but my understanding is that Nextflow stages inputs and expects processes to produce new outputs, making persistent in-place updates difficult. Is that correct? Are there any Nextflow pipelines or recommended patterns for updating Zarr stores in place? Sorry if this has already been discussed elsewhere!

The problem with updating stores in place is that it breaks

-resume

. Any earlier step that relies on it will see a different hash and it will trigger the process to be rerun, and end up using the wrong dataset. I'm not familiar enough with the data stores to know if that makes a difference, as it could be some other data component you're adding. There are ways to side step it, but then the pipeline loses portability. I've used a similar trick in my pipeline chaining method to make sure the child pipelines continue from where they left off if something kills a process, but it only works on HPC and locally.

Thanks Mahesh, thats really helpful and confirms what I thought might be the case that in-place zarr changes aren't ideal in Nextflow because of how pipeline caching works when you resume, thanks!

So I heard about a tool called

icechunk

from scverse. I made a quick toy to see if it could be leveraged in this setting and it seems one can. This repo explains what's needed to make it work, github.com/mahesh-panchal/nf-icechunk-records but it still needs refinement. Feel free to toy about with it if you want.

interesting. I've been prompting a

icechunk

based stitching workflow as well github.com/cellgeni/nf-modules/blob/…/main.nf So far, we only need

local

operations and it seems fine. But just wondering what would change for cloud compute environments. it seems that it wont' change much since repo is passed as

String

instead of

Path

?

If it's passed as a

Path

nextflow will see that the data store has changed. If it's passed as a

String

, nextflow only checks if the string changes which is why other caveats occur needing to test it still exists and stuff. I don't have access to cloud environments so I have no idea really how this would perform there.

So I heard about a tool called

icechunk

from scverse

Hi all! @Luiz Irber and I are the two biologists currently on the icechunk.io core team. Luiz is the one who gave that talk at scverse, and I am a former (but also forever) microscopist. We're both very excited about how Icechunk might be able to enable better science in biology. So we're very happy to answer any questions anyone might have about it. Or feel free to reach out in the future if you don't have any now