Copying data is a legitimate concern but zarr does...
# microscopy
j
Copying data is a legitimate concern but zarr doesn't solve that -- staging a .zarr will copy all the data in, just like staging a .tiff version of the same image. I think Nextflow's Fusion file system may provide data transfer efficiencies with zarr arrays but I don't know if anyone has tested that yet.
k
Exactly right. In fact, Zarr can make it even harder to copy, because now you have millions of small files instead of a few large ones. What I currently do is just write chunked data (N5/Zarr) directly to a final destination directory and plan on never moving or copying it. This is supported by Nextflow’s symlink staging method, but when creating the Zarr you have to input the parent directory for the Zarr container separately from the container name, so that the parent directory gets linked into the process dir and mounted into the container.
j
To be fair, symlink staging works fine for huge tiff files too (when handling mostly-planar images)
The challenge for staging arises in cloud environments where the input and output folders are in object storage
There, staging involves copying from the storage bucket, over the network, to the local compute instance scratch storage
Fusion file system is (as I understand it) a copy-on-demand wrapper over object storage that only copies in files as they are accessed. With a zarr, this should limit the chunks transferred to the compute instance to the chunks actually touched by the compute task.
w
Just to chime in here. One of the things which as far as I know with ome-tiff is that it doesn't support storing transformations. Particularly what happens during the image analysis pipeline is that there is data duplication which on itself can be very costly, for example because of having to store the registered images. OME-NGFF does prevent this from being a requirement. Transformations can be applied on the fly. That being said there is indeed the downside of copying / moving a zarr store. Things to look into here is whether using a zarr zipstore with efficient zipping can alleviate this. I am currently looking into this.