possibly silly question: for the bbmap related steps, they pass cpu and memory reqs as part of the command something like this

bbmap.sh \
     \
    in=3114327_1_val_1.fq.gz in2=3114327_2_val_2.fq.gz \
    out=3114327.bam \
    trimreaddescriptions=t pigz=t \
    threads=6 \
    -Xmx0g \
    &> 3114327.bbmap.log

and i wondered if/ why thats necessary? i can see at the top of the associated .command.run that the memory reqs for the process (node) are specified already (in my case for an LSF cluster)

#!/bin/bash
#BSUB -o metatdenovo_work/ec/5cca944b88b416f6e19334e805f570/.command.log
#BSUB -q normal
#BSUB -n 6
#BSUB -R "span[hosts=1]"
#BSUB -W 08:00
#BSUB -M 13653

Hi, maybe we can have

-XmX

as extra argument to be specified in

nextflow.config

instead of having it inside the module. it should not be a problem, I think.

It's odd that you get

-Xmx*0*g

. In the module code we have

-Xmx${task.memory.toGiga()}g

, meaning that the value given to

-Xmx

comes from how much memory Nextflow allocates to this particular process. This is done for (almost) all processes, although the details vary between different programs. Adding a memory flag for this particular processes will thus just create problems elsewhere. Default with the profile we use is 36 GiB coming from how

process_median

is specified. What do you specify as argument(s) to

-profile

? Did you write your own config file and provide to the Nextflow call? (Please give the full

nextflow

command you used and/or paste the

.nextflow.log

file.)

lsf by default expects memory specs in mb, and thats how nextflow provides them. but my cluster expects them in gb. so it constantly thinks its being asked for insane amounts of ram, and refuses to even start jobs. to get around that, i set max_memory to something like 80.MB, which the cluster thinks is gb when part of the bsub cmd, but obviously not otherwise.

I see, you're trying to cheat Nextflow... By asking for less memory, the job submission scripts will might work, but each task will get only 1/1024 of the memory available. (This is true for most tasks. That you got an error from bbmap was likely just a coincidence.) There are better ways. I think you should look at

clusterOptions

: https://www.nextflow.io/docs/latest/process.html#clusteroptions as a way of asking for the correct amount of memory, i.e. memory in GiB instead of MiB. You will of course have to calculate this dynamically, but this can be done in a config file you provide. (In your case, this config should probably be in your home directory so that it's always read.) I'm afraid I have little experience with this, but look at the docs and ask any questions at #CE6SDBX2A.

i think i explored clusterOptions, and iirc it wasnt replacing the automatically added memory request in the bsub cmd, just adding a new line w a different request, which was still causing trouble. i dont mind continuing to investigate other and better solutions to my particular problem. but i do still wonder if the -Xmx arg is necessary? if i remove it in my local copy of metatdenovo, nothing bad seems to happen..

It is necessary, since it allows us to control the amount of memory the command can use. Allocating memory is, as I said, part of many nf-core modules, any that calls a program that has a parameter to let us set memory. Try by just adding a hardcoded memory allocation for processes in a config file and see what happens. If it's like you remember, that it's just added as another memory allocation request, perhaps you need to build your own process submission task with a config (I'm guessing wildly here). You can probably get better help at #CE6SDBX2A here, or in the Nextflow Slack workspace.

i think there is something im still not understanding here, but thats ok. thanks.

Let me try again: Each process in a pipeline is allocated a certain amount of memory. In a cluster setting this number is typically used for two purposes: Ask for a node with sufficient memory but is also specified as an argument to the actual program that gets run. In the case of BBMap, memory allocation to the program is done with the java option

-Xmx

and controls the behaviour of the program (see under "Memory" here: https://jgi.doe.gov/data-and-tools/software-tools/bbtools/bb-tools-user-guide/bbmap-guide/). If we don't do the latter many programs, BBMap among them, will not know how much memory it can use. Sometimes a default exists and this will be used, sometimes the process might take all memory it sees, which can lead to the queue manager killing the process. The

--max_memory

parameter you've used to control memory sets a ceiling for the memory allocation to individual processes when you run on a cluster. Clearer?

yes, and no tbh. though youre saying 'in a cluster setting' does remind me that i suppose not everyone is in a cluster setting, and maybe it could be trouble for a local executor.

If you use a local executor, the process memory will be used for the second purpose, i.e. tell the program how much memory it can use. It will also be used to calculate how many processes Nextflow can start in parallel. The

--max_memory

parameter has a slightly different meaning here: this is the amount of memory available to Nextflow for processes. If you set it to 80 GiB it will start two 32 GiB processes in parallel. If you set it to 20 GiB, each 32 GiB process will get its memory requirement reduced to 20 GiB and Nextflow will only start one at a time.

i tried investigating that too.. couldnt find anything in the lsf docs, and when i asked the cluster admin they basically told me to just submit w the units they wanted

Thanks! I've actually kept an eye on your discussion over at the Nextflow Slack now and then. Good that you found the problem, even if it's not yet solved.

Good that people around here are so responsive 😁