Hello I have a `Missing output file` error on the megatests nf-core #aws-megatests

Hello, I have a `Missing output file` error on the...

Quentin Blampey

03/30/2026, 12:49 PM

Hello, I have a

Missing output file

error on the megatests for a pipeline in development (logs here) The script runs correctly, and the output file itself is created, but somehow not detected. The output file causing the issue is inside a directory: see

path("${sdata_path}/.sopa_cache/patches_file_image")

at line 14 here. I run this pipeline on my own data using AWS Batch, and it works nicely, so it's unlikely related to AWS batch. And all the other tests from the CI are passing. Is there something specific to megatests that could explain this?

Rike Hanssen

03/30/2026, 1:25 PM

~~The path likely is in a different place on the node than it is on local compute where nextflow symlinks everything. You cannot assume aboslute paths on AWS.~~

Rike Hanssen

03/30/2026, 1:27 PM

what is sdata_path?

Rike Hanssen

03/30/2026, 1:27 PM

side node: the docker containers need to be pushed to the nf-core account on quay.io before release. Can you submit a request in #C09H6NYHR9T?

👍 1

Rike Hanssen

03/30/2026, 1:32 PM

I resumed the run. There was a spot reclamation and I am wondering if during the recovery step something got corrupted. I am not optimistic it's that but easy to exclude as an option

🙏 1

Quentin Blampey

03/30/2026, 3:30 PM

what is sdata_path?

This is a

zarr

directory containing a single `SpatialData` object. Since this format is a directory and not a single file, it made it challenging to implement many processes, but it's supposed to now work on AWS

Quentin Blampey

03/30/2026, 3:31 PM

Most processes are updating the SpatialData object, i.e. creating a new subdirectory inside this

zarr