I checked a (biased) selection of nf-core pipelines for their AWS Results and approximately half failed in their most recent release, see attached table. Looking at advertising our pipelines, that looks very poor. Because ampliseq isnt working I wanted to see whether I am the only one who doesnt get that to work. Just wanted to leave that here, not sure yet what I conclude from that.

I am wondering if something odd is happening with the results. I was looking at a few sarek releases where I was reasonable certain I had triggered the full size tests and they had completed but i didn't see the results on the website. Do you by chance have for any of these the platform link to the successful run?

Thanks a lot! About that oom, for both, 2.16.0 and 2.16.1, peak_rss shows for picrust almost 40GB, but 2.16.0 passed with 20GB allocated, 2.16.1 not. Seems odd. Not sure whether it makes sense to increase the resource allocation?

@Daniel Straub I am running airrflow on the non-fusion environment to get it to run. I couldn’t figure out what was going wrong with the fusion environment since particular processes were passing there for some samples but not others. That might be one aspect?

@Gisela Gabernet do you have a failed run I can check.

That might be one aspect?

Yes, absolutely. Is there a way to specify that somewhere so that the AWS test is always run without fusion?

@Rike Hanssen Here is the failed run: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/5k8tausNyT9zbs/logs

I don't think it is consistent enough to switch out yet... not everyone are having the problems it seems

Thanks Gisela, will try that out!

Gisela, I am looking at yours, I think what happened is that AWS spot reclaimed the machine and the snapshotting wasn't fast enough to ransfer the job to a new machine

Snapshot reclamation is like, 10000% more prevalent than it was a couple of years ago. I'm finding on many of my tests these days that basically any task running for more than about 30 minutes seems to get reclaimed.

ok I grabbed the recent releases from pipelines (had rerun a bunch of them in february) and had claude parse over the logs 3 times to check which of them ran on fusion, which of them definitely on plains3 queue and which had a work around.

| Category | Count |
|----------|------:|
| PASS with fusion enabled | 50 |
| PASS on non-fusion CE (PlainS3 queue) | 3 |
| FAIL — fusion-related | 6 |
| FAIL — not fusion-related | 16 |
| No full-size test run | 16 |
| **Total** | **91** |

I have spot checked the results, it be great if someone else can also randomly click around the links and let me know if any are wrongly attributed.

Checked for ampliseq, seems correct. Looking at the config, maybe that PR should resolve the issues of ampliseq in the future?

Ok, thank you for the detailed reports, in my case just switching to the non-fusion CE solved the issues but I see it is more complex than that. For airrflow I just tried resuming it several times but it was failing at random processes for random samples (which makes sense that it is related to spot reclaiming). Is there something else we can do to get them to run on the fusion CE? E.g. auto-resuming or any other solution?

We could setup a fall back to an on-demand queue or let it try more often on spot reclaims

Sounds like a good plan!

I am also finding a similar error for proteinfold, one sample worked but the other keeps failing with error messages such as:

12, 225), 'deletion_mean': (225,), 'all_atom_mask': (225, 37), 'all_atom_positions': (225, 37, 3), 'assembly_num_chains': (), 'entity_mask': (225,), 'num_templates': (), 'cluster_bias_mask': (512,), 'bert_mask': (512, 225), 'seq_mask': (225,), 'msa_mask': (512, 225)}
[2026-03-12 14:07:34.347] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps
FUSION_GPU_USED=false
[2026-03-12 13:17:34.480] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 13:22:34.433] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 13:27:34.365] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 13:32:34.414] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 13:37:34.373] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 13:42:34.371] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 13:47:34.318] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 13:52:34.374] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 13:57:34.372] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 14:02:34.401] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
[2026-03-12 14:07:34.429] [ERROR] Error (exec.c:102): incremental dump failed with code: -52
FUSION_GPU_USED=false
[2026-03-12 14:15:15.768] [ERROR] Error (helpers.c:675): no full dumps found in metadata
[2026-03-12 14:15:15.768] [ERROR] Error (dumprestore.c:335): no full dumps available for restoration

I think it is trying to rescue the spot reclaim but can't store the data fast enough. Which tool is this?

e.g. https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/5HpawGcTFSYmyZ/tasks

it is auto retrying now as far as I can see

yes, it increased the memory, let’s see

@Jose Espinosa-Carrasco your pipeline passed and this time it was able to recover the spot instance reclaim and try again: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/37pQymumIJoo1K/metrics

However, this one failed that I launched at the same time failed 👉🏻 https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/5HpawGcTFSYmyZ/logs with:

[2026-03-13 03:39:25.955] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps
  [2026-03-13 03:44:25.927] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps
  [2026-03-13 03:49:25.928] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps
  [2026-03-13 03:54:25.969] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps
  [2026-03-13 03:59:25.906] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps
  [2026-03-13 04:04:25.951] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps
  [2026-03-13 04:09:25.974] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps
  [2026-03-13 04:14:25.912] [WARN] Warn  (dumprestore.c:235): predump failed, starting new chain for future dumps

the real issue is at the top of that log:

Job attempt duration exceeded timeout

the warn logs just indicate that the snapshotting is preparing dumps just in case. However, you just ran out of requested time here I believe. the job is cancelled at 16h 22s and you requested 16h of walltime for that job

the issue here is that this job normally does not last that long in my HPC and according to the

.command.err

and the

.command.log

I can not see anything strange going on…

Why are you doing all the mv/cp operations? They tend to seriously increase runtime on compute that doesn't have shared storage like clouds. Are they necessary?

Yes 😬

for context

The action now fails to send the tests to the platform: https://github.com/nf-core/proteinfold/actions/runs/23048595518

ah woops I need to update the environment variables on github

Org level?

yes: https://github.com/organizations/nf-core/settings/secrets/actions

Sarek is running now, I guess it’s you 💥

yes but I haven't changed anything in comparison to proteinfold on the sarek repo homer disappear soooo I still don't know why proteinfold is using some other value

Could be something related to the gpu instances?

manual submission seems fine: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/4ZacaSxQl6x3nJ/logs

Not it run but I got this error:

Caused by:
  Missing AWS Batch job queue -- provide it by using the process `queue` directive

from the GHA runner?

No already in the platform: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/4ZacaSxQl6x3nJ/logs

you can have a secret on the repo that overwrites the one from the org

don't see any repo level secrets: https://github.com/nf-core/proteinfold/settings/secrets/actions

I got it

ok throwing claude at it. now that I think about it it's also weird it's plain text when it should be a secret

TOWER_COMPUTE_ENV_GPU is still 6 months old

why do we have this twice 😭

Why do we have secrets and variables?

also why the ids and not the name. Let me try with name so we can swap out CEs

I cancelled the runs will only trigger boltz

checking what is different in the fallback strategy. The syntax is the same as on the CPU one as far as I can tell, and there it works

No, I didn’t change anything

no worries, it's building an incremental snapshot

yep, but it seems to be stacked here without even starting the process 👀

Ping me if you’d like me to do some tests with airrflow as well, going back to the fusion environment 🙂

Sorry, got stuck on non squirrel tasks for once

@Gisela Gabernet It should work. I tested with sarek and it was ok. I don't quite know why the GPU based ones sometimes work and sometimes don't but the CPU based ones seem ok. Let me know how testing goes

Great, I’ll test it out and let you know. What environment should I use?

sorry I was sick. this CE should work (I updated the github secrets so it should work also if you run the action): aws_ireland_fusionv2_nvme_cpu_spot

As discussed above I attempted to set AWS tests to

TOWER_COMPUTE_ENV_NO_FUSION

and as earlier suggested by Rike

scratch=true

for QIIME2 in this PR. I started

nf-core AWS test

manually (not

test_full

, I wanted to keep computational burden low) on dev branch via github here, but the run failed again here. Something went wrong, it appears like fusion was used (inferred from aws_ireland_fusionv2_nvme_cpu_spot), so I didnt successfully modify the settings?

let me take a look. side note:

scratch=true

should only be relevant for the fusion runs as this will use the local storage on the node instead of streaming from s3

Ok, and how would I then correctly avoid the fusion setting?

if you are using the non-fusion CE, then there shouldn't be any fusion to avoid. I am checking on #CE5LG7WMB if people want to keep secrets or variables and then we can set it on the Github side appropriatly. In the meantime for testing you could also just hardcode

aws_ireland_s3_cpu_spot

for the CE name.

Ok, thanks, unfortunately I dont know where I would hardcode that non-fusion environment, so I am patient and wait for the github change (also have no clue where to start there 😄 ).

https://github.com/nf-core/ampliseq/blob/6a8f2c3a24ed5b9d4223f241b9e569b00767f3c6/.github/workflows/awstest.yml#L19 you could hardcode the string here

Like that https://github.com/nf-core/ampliseq/pull/970?

🤷‍♀️ worked on rerun: https://cloud.seqera.io/orgs/nf-core/workspaces/AWSmegatests/watch/4AAv44pz3FRKs9/logs