Hi all, does anyone here have some benchmarking data or recommendations for ESVEE resource requirements? I know it's hungry for that RAM like all SV callers, but just had it fail with OOM on 120GB and 32 threads (90x-30x T-N, running OA 2.3 from fastq with HMF GRCh38). Since we are running this via AWS Batch/Seqera Compute, we would like to optimize that step as much as possible. If anyone has data it would be welcome. Thanks!

There's a parallel discussion with another group running into resource constraints. It's not something we are seeing except for maybe the most broken FFPE samples.

Hi Oliver, I retried with 180GB RAM and also had it OOM. In the meantime I ran the same sample through OA 2.1 and it went through with ~60GB in ESVEE-PREP and ~80GB in ESVEE-ASSEMBLE. It's fresh-frozen material, target 90x depth (bamtools tells me 70x). In the OA 2.1 run purple gives me 0.15 purity (bad for many things, but for our purpose here intuitively it should lower resource requirements?). Let me know what other metrics would be useful.

So weird. This feels like a perfect sample tbh.

@valleinclan any chance we could get the data uploaded somewhere for the Aus team to give it a good whack?

These are the stats for 3 cases we have managed to run with OA2.2.0. I haven’t had time to properly troubleshoot yet:

Okay that is useful. So it is not sample specific.

@Eva Caceres how are you running OA? Nextflow version, profile, orchestrator, fusion enabled etc? Just in case there are some share infrastructure issues different from the ones HMF & co are using. Here with Nextflow 25.10.2, docker profile, running on AWS Batch with fusion enabled, with GRCh38_hmf (pre-staged). Attached the ESVEE log in case it's useful.

Nextflow version 25.10.0 build 10289 Slurm HPC GRC38_hmf (pre-staged) Oncoanalyser version 2.2.0

Hi @valleinclan @Eva Caceres - could you try this branch and lower the allocated jvm heap size as such in your config ?

process {
    withName: /.*ESVEE.*/ {
        ext.xmx_mod = 0.70
    }
}

@Rayan Hassaïne resubmitting with the provided branch and

withName: 'ESVEE' { cpus = { 28 + (4 * task.attempt) }; memory = { 80.GB + (16.GB * task.attempt) }}

Will report back

Nice to hear Jose! Could you try to add that argument to sage as well and see if it fixes things ?

how long did it take to complete?

It took 2.5h, but I did not get SAGE_SOMATIC to complete. I need more time to check what's going wrong there.

I think this is worth investigating but honestly I don’t think this could explain all the issues we are experiencing. I’ve been testing using both Seqera platform and native nextflow and I’ve been encountering similar issues. I’m currently running ESVEE as a standalone job with updated memory options (outside of nextlflow/seqera). It’s still running but if it ends up looking similar that would discard the nextflow/seqera component

Ok, got too ahead of myself 😅 Sorry.

If you can get a minimal reproducible example showing the behaviour with / without fusion then please post it in the #fusion-fs channel in the nextflow slack (or maybe better, community.seqera.io) and we can take a look. Or actually if you're a customer using Seqera compute then a support ticket is easiest as then we can talk about accessing logs.

Ok, so the standalone esvee job using xmx 70% took the same amount of time and nothing is really different, so that doesn’t solve our problem. And at least in our case, the issue doesn’t seem to be due to nextflow nor Seqera. We do have issues in our cluster with I/O because the file system is simply not optimal, so bottlenecks in I/O could be the cause of our problems. Unfortunatelly I don’t have a way to test this at the moment

hi all, just posting here the response from Seqera Customer Support after their investigation:

Thank you for your patience. We have completed our investigation and identified the root causes of the reported issues. A summary is provided below.

1. ESVEE Out-of-Memory (OOM)
The ESVEE OOM is not caused by Fusion. The Java process allocates memory both within the managed heap (-Xmx) and outside it, including native BAM I/O buffers, JNI calls, and thread stacks. With -Xmx set to 95% of the container memory, only approximately 6 GiB remains available for off-heap allocations, which is insufficient for this workload. Reducing the heap cap to approximately 75% of the container memory is the correct approach, as it provides adequate headroom for off-heap usage and prevents OOM failures.

2. ESVEE 10–12x Slowdown
The slowdown is caused by Fusion Snapshots. Our analysis shows that ESVEE allocates approximately 110–123 GiB of memory. Fusion Snapshots must dump nearly all allocated memory pages every five minutes. At an effective write speed of approximately 0.5 GB/s, each snapshot takes about 244 seconds within a 300-second interval, leaving only around 19% of the time available for computation. This behavior aligns with the slowdown observed. To address this, we recommend either disabling Fusion Snapshots for the ESVEE process or increasing the snapshot interval. Snapshots can be disabled per process in the Nextflow configuration as follows:
process {
    withName: 'NF_CHAI:CHAI_1' {
        containerOptions = '-e FUSION_SNAPSHOT_ENABLED=false'
    }
}
For ESVEE specifically, use the appropriate process selector (for example, withName: 'ESVEE' or the fully qualified name used in oncoanalyser).

3. SAGE_SOMATIC OOM
This appears to be a separate configuration issue. The process_high memory setting is currently 24 GiB, while the base configuration for nf-core/oncoanalyser requires 72 GiB for this process class. We recommend increasing process_high to at least 72 GiB.

Recommended Actions
Reduce -Xmx to approximately 75% of container memory for ESVEE
Disable Fusion Snapshots for ESVEE or increase FUSION_SNAPSHOT_INCREMENTAL_INTERVAL
Increase process_high memory to ≥72 GiB for SAGE_SOMATIC

TLDR: • The 5% memory space left between the memory given to the process and the java heap cap is not sufficient for certain i/o intensive processes. That's why the xmod hotfix by @Rayan Hassaïne works. • With fusion enabled, in this very intensive i/o processes then a lot of computation time goes into snapshotting, so it's wise to control that • I was giving too little memory to SAGE_SOMATIC, that one is on me only. But I suspect this process also suffers from the heap space cap (and perhaps others, looking at Virusbreakend right now). I hope this is useful for other people as well.

Thank you! Snapshots is not something that would have ever occurred to me.