Hello everyone, is there an easy way to change the...
# airrflowg
Giulio Isacchini
03/13/2025, 12:09 PMHello everyone, is there an easy way to change the number of cores/threads the pipeline uses, or do I have to specify it for each command independently in a custom.config file? thanks!
Giulio Isacchini
03/17/2025, 11:32 AMAt the moment, as a workaround I made a custom.config file that modifies the processes labels as follows:
Copy code
process{
errorStrategy = 'terminate'
withLabel:process_low {
cpus = { 32 * task.attempt}
memory = { 64.GB * task.attempt}
time = { 6.h * task.attempt}
}
withLabel:process_medium {
cpus = { 32 * task.attempt}
memory = { 64.GB * task.attempt }
time = { 8.h * task.attempt }
}
withLabel:process_high {
cpus = { 32 }
memory = { 64.GB }
time = { 10.h * task.attempt }
}
}g
Gisela Gabernet
03/18/2025, 2:00 PMHi @Giulio Isacchini, it is possible to limit the resources for all the processes with this config file:
Copy code
process {
resourceLimits = [ memory: 16.GB, time: 72.h, cpus: 8 ]
}Gisela Gabernet
03/18/2025, 2:02 PMIf you’d like to set up this for all your runs, you can also store this config file under your home directory, stored in this path with the name “config”
$HOME/.nextflow/configg
Giulio Isacchini
03/24/2025, 3:43 PMthanks! My hope was the opposite: to increase the resources for a specific job (igblastn was running only on two cores). The trick above does the job, but I am not sure if there are alternative ways
g
Gisela Gabernet
03/25/2025, 4:46 PMah sorry I misunderstood, this is how to increase it for a process:
Gisela Gabernet
03/25/2025, 4:46 PM Copy code
process{
withName:CHANGEO_ASSIGNGENES {
cpus = 10
memory = 64.GB
time = 10.h
}
}Gisela Gabernet
03/25/2025, 4:50 PMBasically specifying “withName” instead of “withLabel”and the process name, the rest works the same way. If you’re not using the ” * task.attempt” there’s no need for the curly brackets any more.
g
Giulio Isacchini
03/31/2025, 2:50 PMok great thanks!
👍 1