I'm running this workflow: https://pipelines.tol.sanger.ac.uk/genomeassembly using the

pdc_kth

profile. I'm srun-ing the main process. For a task using a local executor (

SANGERTOL_GENOMEASSEMBLY:GENOMEASSEMBLY:NUCLEAR_ASSEMBLY:INDEX_STAGE

) I keep getting the error

Process requirement exceeds available memory -- req: 6 GB; avail: 1.6 GB

. My guess is that this is caused by the

pdc_kth

profile requiring a minimum of 6GB of memory. Is this correct? What is the best way to handle this? The task itself seems to be very computationally light, so I can understand why it is run locally. Right now my workaround is srun-ing and asking for 6GB of memory, but that means requesting more resources than necessary for several days of run time which seems wasteful.

I have to admit, I usually run

nextflow

in a

screen

session on the login node so the local processes just run. It usually has a fairly low footprint that you don't starve resources for others. I wonder if the executor is accessible in the task context. Then one could have a guard on that. If you really want, you can use a custom config to override the definition of the

process.clusterOptions

so it doesn't add the

--mem=6G

. In the grand scheme of things though I don't think it's that wasteful of resources, and simply adding the 6GB to srun is fine ( or do you find it's really burning your cpu allocation?)

That's also how I usually run it, but it got killed, I suspect because of the local task? But okay, the custom config sounds neat, so it would be something like this?

process {
    withName:'.*INDEX_STAGE' {
        clusterOptions = '-A projectid -p shared'
    }
}

It's not being submitted to anything so I guess that would actually be empty.

6 Gbyte was the minimum I felt I could pick without seeing random kills (e.g. because of sudden memory pressure from other processes). Granted that this was typically with a apptainer or similar. If you don't want to run on login nodes, I would certainly ask for 6 Gbyte (or however much is needed) for the monitoring job, but if you don't want to you can just specify a lower memory requirement for the process running locally?