A community effort to collect a curated set of analysis pipelines built using Nextflow.

nf-core

Megatest <https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/Oq4b50j1voDvt/tasks?taskId=102|failed again>, but with a different problem:
```1:03PM INF [seqera-fusion] sending incremental snapshot signal (SIGUSR1)
[2026-09-10 13:03:14.845] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps
[2026-09-10 13:03:14.846] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
make: *** [/usr/local/bin/LINKS-make-real:122: test_hifi_hifiasm_links.tigpair_checkpoint.tsv] Killed
FUSION_GPU_USED=false
Fusion Info:
    available_disk: start:2.5TB, min:2.4TB, max:3.5TB, end:3.5TB
    clone_namespace: false
    download: 47210.36MB, drops:0, skip:0, sync:0, req:370
    fusion_version: 2.5.14-20e904e
    kernel_version: 6.1
    max_open_files: 1048576
    upload: 169466.41MB, drops:0, skip:0, sync:79, req:1402```
I do not really know what this means tbh

```Exit: 2 (FAILED) (action: TERMINATE)```


hmm, I was thinking it might be a 17X code, an intermittent fusion issue. Can you directly resume it from Seqera Platform? I've never logged into the nf-core Platform.

there were other jobs that received an 175 at around the same time

it is very possible that LINKS is somehow masking the 175 and reports 2

If this is intermittent, I would restart. I anyway noticed that I forgot 1 param, so the test needs to run from the beginning

175 is a fusion error, it should be caught at the base.config retry logic

The fusion error looks like it tried to store a snapshot to be able to resume the task from mid-way through, but AWS killed the instance before it was able to finish uploading the snapshot image

Usually happens when there is a lot of data in memory

In theory the task should be able to retry again from the start, but if spot instances are being reclaimed constantly then it can trigger a pipeline fail

It's increasingly common on AWS as the spot markets are getting really really tight

(one of the several reasons we started building our intelligent compute scheduler, which can fall back to on-demand when this starts happening)

Ok, let's see if it happens again. LINKS is a bit special, so I would not be surprised if such things caused problems

Are the spot instance reclamations logged during the AWS megatest runs? Or is this handled silently?

I honestly can't really remember :thinking_face:  <@UPC8CHDKQ> might know best when she's back.

Ok, mainly asking because I do not see any ;)

I do remember that it is not trivial to see (I know this because we have an internal project going on at the moment to refresh the UI to surface this info)

Ok, thanks, that largely answers my question

image.png

ahhh there it is.. Thank you <@U016SFULDB8>!

&gt; Are the spot instance reclamations logged during the AWS megatest runs? Or is this handled silently?
yes they are listed in the metrics tab.