Handling user-level limits on threads/processes: I'm testing a pipeline that was hanging for me with lots of "Failed to create thread" warnings, that I was able to resolve by changing the max user processes (ulimit -u) from 4096 to 150000. At the moment I'm running locally by checking out a whole cluster node, so I'm not concerned about antisocial behaviors to other users, but it may be a factor in the future. So questions are - are there ways to set this in my nextflow config rather than in the shell beforehand? Ways to set it dynamically (e.g., 1000 per cpu)? Options to throttle the number of threads requested if I'm in an environment where I can't change the limit, or can't set it as high as the pipeline wants?

Well, if you run the pipelines the way they are designed, i.e. you use a proper config to farm the various processes out to different nodes (which is faster than running all on a single one), then ideally you should not have to mess with

ulimit

, because there is only one process running. (You might still need it if that process spawns a lot of subprocesses and/or several of those end up on the same node, but you can cross that bridge when you come to it. Nextflow has the `beforeScript` directive, which may be helpful for this, but if this becomes a recurring issue, I would ask the sysadmins to raise the default limits [4k seems very low -- on my cluster the head node has 30622 and a typical compute node 399360; heck, my laptop has 55046]). BTW, I am far from an expert on this, but I don't think threads make a difference here, the limit is for processes. If you want to limit threads, which is a legitimate need, make sure your config has a proper

resourceLimits

directive, which directly affects how many

cpus

any

process

can ask for. I would use the CPU count of a typical node as limit for

cpus

(and you can adjust this dynamically, depending on queue, QOS, etc.), so you don't ask for more than is available and get jobs that can't be scheduled. Make sure to set

memory

and

time

as well. These are global limits for any

process

. You also can use custom config using

withName

or

withLabel

to set the

cpus

[or

memory

, or

time

] for specific `process`es or categories, but I would only mess with that if you see glaring inefficiencies or some process runs out of resources. 🤞

The setting is for LightWeightProcess (which are typically used to provide threads as well).

Points taken about cluster deployment. I'm in dev/testing phase - still troubleshooting inputs and settings - so maybe it won't be an issue once I get to production. Also, still on SGE 😑, but I can probably get a config from a colleague here

There are SGE configs at https://nf-co.re/configs/, just type

sge

into the search box. You may have to modify things to fit your cluster, but you'll have some inspiration. 🤞 The

resourceLimits

limits the requests to the scheduler, it is then up to the scheduler/OS to enforce that. Some do a better job of that than others.