Failed to publish file error with nfcore/mag 5.4.1 test profile in combination with S3 HI everyone, I am trying to run mag 5.4.1 with the test profile together with Nextflow 25.10.4 and our local S3 and run into the following reproducable error: ERROR ~ Failed to publish file: /vol/projects/mbeckste/nextflow_scripts/nfcore-mag/work/85/732971b497420ebeab146e9085edd2/test_minigut-bacteria_odb10-busco; to: s3://mag-test/output/GenomeBinning/QC/BUSCO/SPAdes-MetaBAT2-test_minigut/test_minigut-bacteria_odb10-busco [copy] -- See log file for details What I have tried so far to pinpoint this error is as follows: 1. Profile test; local execution; file system output $ NXF_VER=25.10.4 /vol/biotools/bin/nextflow run nf-core/mag -r 5.4.1 -profile docker,test --outdir output --> works as expected without errors!!! 2. Profile test; Local HPC (SLURM) execution; local output $ NXF_VER=25.10.4 /vol/biotools/bin/nextflow run nf-core/mag -c minimal_clowm.config -r 5.4.1 -profile docker,test --outdir output --> works as expected without errors!!! File minimal_clowm.config (attached) provides just configuration for our local SLURM and S3 endpoint 3. Profile test; Local HPC (SLURM) execution; publish output to self hosted S3 bucket s3://mag-test/output $ NXF_VER=25.10.4 /vol/biotools/bin/nextflow run nf-core/mag -c minimal_clowm.config -r 5.4.1 -profile docker,test --outdir s3://mag-test/output --> This fails with the error mentioned above. I also attached the nextflow.log from this run, although - at least for me - it gives no additional information. Interestingly, there are several PublishDir calls which all work and I also see the output files in the bucket s3://mag-test/output. Remark: We run several nfcore workflows on our platform and hence together with our S3. So far without any major problems. Also mag 3.4.0 runs without problems (although with an older Nextflow version) Any kind of help or hint how to debug this scenario further, is really appreciated.

Hi @Michael Beckstette Thanks for posting here too! I was slowly getting to your email (lots of meetings today unfortunatley). I've very occasionally seen that sort of error, but never consistently enough to pinpoint the problem. Did you see the same thing on earlier version of mag, if you tested those?

In the past we used the following versions of mag (all with Nextflow 25.04.8) WITHOUT any problems: 3.0.3, 3.1.0, 3.3.1, 3.4.0. I came across this issue, when trying to use mag 5.4.1 with Nextflow 25.10.4 with production data and was able to reproduce it with the test profile.

do you think this is related @Diego Alvarez Saravia?

Yes, we updated BUSCO to v6. And in mag v3 versions, there was the custom BUSCO module that reorganized all the files, and now we're using BUSCO as it is. Maybe it is related to BUSCO producing a huge number of small files someway? @Michael Beckstette can you can try

--busco_clean

? That should remove a lot of BUSCO files that aren't really that necessary.

I will give it a try. Although even many small files shouldn't be a problem for S3. Does the test profile work on AWS or other S3 storage providers without any problems?

I believe so, but can try again now

Thanks for the answers! I just wonder how to further deal with this. Can I somehow "influence/tune" the handling of many, tiny files in the nextflow config?

Are you running BUSCO because you need to or because it's the default? Because if it's the later, you can try with CheckM2 instead of BUSCO.

image.png

Do you mean -profile docker,test,debug ?

Yes

@Diego Alvarez Saravia Originally I tried it with production data and run into this error. Then I tried just the test profile with its defaults. So there is no absolut need for BUSCO. However, some of our users use it, because it is there and its an option....

We've seen this before: https://nfcore.slack.com/archives/CE9MS66BS/p1760555547272489 Looking at other issues regarding publishing on S3, it could be a name clashing (not 100% sure). I'm going to doublecheck the publish dirs to verify that we don't have something like that. Personally I don't use S3 at all, and if things works on Seqera Platform, I'm totally lost at this 😅

I really wonder if this can be related to our S3 storage. In particular because James mentioned that it runs successfully on AWS. We use Ceph backed S3 which is imho an also widely used setup.

I also find it strange as im running nextflow pipelines on deNBIs Openstack infra right at the moment ,storing results on the s3 buckets just fine

I now also tried it with a reduced samplesheet (first sample from samplesheet.multirun.v4.csv). Still with the same error. In addition I tried to write to a de.NBI/Openstack S3 Bucket, again with the same error. @James Fellows Yates: Can you really confirm that mag v5.4.1/Nextflow 25.10.4 with -profile test and upload to a de.NBI/Openstack S3 bucket works?

Still running, but completed 6 BUSCO runs with no error so far

This is my command line call: NXF_VER=25.10.4 /vol/biotools/bin/nextflow run nf-core/mag -c minimal_openstack_S3.config -r 5.4.1 -profile docker,test --busco_clean --outdir s3://openstack-mag-test/output --igenomes_base ./igenomes where minimal_openstack_S3.config contains the config for accessing the de.NBI Openstack Bucket s3://openstack-mag-test/output ---> ERROR ~ Failed to publish file: /vol/projects/mbeckste/nextflow_scripts/nfcore-mag/work/3c/696343610f1ec4ebff4733a46f3fe0/test_minigut-bacteria_odb10-busco; to: s3://openstack-mag-test/output/GenomeBinning/QC/BUSCO/SPAdes-MetaBAT2-test_minigut/test_minigut-bacteria_odb10-busco [copy] -- See log file for details

OK it crashed due to a transient error

it is always test_minigut-bacteria_odb10-busco

Mar-24 14:08:04.777 [PublishDir-1] DEBUG nextflow.processor.PublishDir - Failed to publish file: /vol/permanent/mag/work/35/11e46c13af0263c63baf967aebf23f/test_minigut_sample2-bacteria_odb10
-busco; to: <s3://mag-test/GenomeBinning/QC/BUSCO/SPAdes-MaxBin2-test_minigut_sample2/test_minigut_sample2-bacteria_odb10-busco> [copy] -- attempt: 1; reason: the path: <s3://mag-test/GenomeBin>
ning/QC/BUSCO/SPAdes-MaxBin2-test_minigut_sample2/test_minigut_sample2-bacteria_odb10-busco/SPAdes-MaxBin2-test_minigut_sample2.noclass.1.fa/logs does not exist
Mar-24 14:08:06.287 [PublishDir-1] DEBUG nextflow.processor.PublishDir - Failed to publish file: /vol/permanent/mag/work/35/11e46c13af0263c63baf967aebf23f/test_minigut_sample2-bacteria_odb10
-busco; to: <s3://mag-test/GenomeBinning/QC/BUSCO/SPAdes-MaxBin2-test_minigut_sample2/test_minigut_sample2-bacteria_odb10-busco> [copy] -- attempt: 2; reason: the path: <s3://mag-test/GenomeBin>
ning/QC/BUSCO/SPAdes-MaxBin2-test_minigut_sample2/test_minigut_sample2-bacteria_odb10-busco/SPAdes-MaxBin2-test_minigut_sample2.noclass.1.fa/prodigal_output/predicted_genes does not exist
Mar-24 14:08:08.572 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 248; id: 179; name: NFCORE_MAG:MAG:CATPACK:CATPACK_SUMMARISE_BINS (test_minigut)
; status: COMPLETED; exit: 0; error: -; workDir: /vol/permanent/mag/work/3d/4d3831e0b30e28330993bd2f3a4a5e started: 1774361288570; exited: 2026-03-24T14:08:06.104512419Z; ]
Mar-24 14:08:08.574 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 249; id: 178; name: NFCORE_MAG:MAG:CATPACK:CATPACK_SUMMARISE_BINS (test_minigut)
; status: COMPLETED; exit: 0; error: -; workDir: /vol/permanent/mag/work/fa/e70a70d0c21422c3ec435b3d748660 started: 1774361288572; exited: 2026-03-24T14:08:06.167511287Z; ]
Mar-24 14:08:08.694 [PublishDir-10] DEBUG n.cloud.aws.nio.S3FileSystemProvider - S3 upload file from=/vol/permanent/mag/work/fa/e70a70d0c21422c3ec435b3d748660/MEGAHIT-SemiBin2-test_minigut-b
ins_summary.txt to=<s3://mag-test/Taxonomy/CAT/MEGAHIT/SemiBin2/test_minigut/bins/MEGAHIT-SemiBin2-test_minigut-bins_summary.txt>
Mar-24 14:08:08.747 [PublishDir-2] DEBUG n.cloud.aws.nio.S3FileSystemProvider - S3 upload file from=/vol/permanent/mag/work/3d/4d3831e0b30e28330993bd2f3a4a5e/MEGAHIT-MaxBin2-test_minigut-bin
s_summary.txt to=<s3://mag-test/Taxonomy/CAT/MEGAHIT/MaxBin2/test_minigut/bins/MEGAHIT-MaxBin2-test_minigut-bins_summary.txt>
Mar-24 14:08:10.002 [PublishDir-1] DEBUG nextflow.processor.PublishDir - Failed to publish file: /vol/permanent/mag/work/35/11e46c13af0263c63baf967aebf23f/test_minigut_sample2-bacteria_odb10
-busco; to: <s3://mag-test/GenomeBinning/QC/BUSCO/SPAdes-MaxBin2-test_minigut_sample2/test_minigut_sample2-bacteria_odb10-busco> [copy] -- attempt: 3; reason: the path: <s3://mag-test/GenomeBin>
ning/QC/BUSCO/SPAdes-MaxBin2-test_minigut_sample2/test_minigut_sample2-bacteria_odb10-busco/SPAdes-MaxBin2-test_minigut_sample2.noclass.1.fa/run_bacteria_odb10 does not exist
Mar-24 14:08:13.164 [PublishDir-1] DEBUG nextflow.processor.PublishDir - Failed to publish file: /vol/permanent/mag/work/35/11e46c13af0263c63baf967aebf23f/test_minigut_sample2-bacteria_odb10
-busco; to: <s3://mag-test/GenomeBinning/QC/BUSCO/SPAdes-MaxBin2-test_minigut_sample2/test_minigut_sample2-bacteria_odb10-busco> [copy] -- attempt: 4; reason: the path: <s3://mag-test/GenomeBin>
ning/QC/BUSCO/SPAdes-MaxBin2-test_minigut_sample2/test_minigut_sample2-bacteria_odb10-busco/SPAdes-MaxBin2-test_minigut_sample2.noclass.2.fa/logs does not exist

I also noticed in our cases it's repeatedly failing to find it

Broken symlinks maybe? The module should take care of that already 🤔. Yesterday I merged: https://github.com/nf-core/mag/pull/1002 to use all variables in busco publish dirs. To me doesn't look like this issue is because of that, but can you try anyway?

i.e., @Michael Beckstette can oyu try with

-r dev

?

If it would be broken sym links why does it then work with AWS?

The only thing I can think of is there is some race condition or something

I am currently on a retreat and might be able to test it again with -r dev on our Ceph on Friday.

@Michael Beckstette working for me now with

-r dev

!!

@James Fellows Yates and @Diego Alvarez Saravia I can also confirm that it works with the -r dev option.

That's great news! Thanks @Michael Beckstette and @James Fellows Yates

I guess we should have a patch release, what do you think @Diego Alvarez Saravia . do you think there is enough?

I was thinking exactly that! This seems like a important fix (and also the other BUSCO is quite important too).