Hi Everyone, I have run early eager for small dataset (in coverage and sample), but now i am analysing whole genome with pretty good coverage, so problem is like at 4 hrs interval everything get failed, start running again. As i have read documents, its like its running intially with some default resource, if it does not get succeed, it will increase the resource and run. here with each time, it will make new file as i have seen directly and taking extra space, also i am loosing time, as i know i have dense data with many sample, how i can increase resource directly for adapterrmoval process & AdapterRemovalFixPrefix, also for bwa aln, i have seen some example but i am not able to make configure file. i am running currently on slurm and with help of conda. thank you.

Hi Shailesh! You will need to provide a configuration file that tweaks those processes. The config file (

my_config.config

) would look sth like this:

process {
    withName: adapter_removal {
        memory = { task.attempt == 3 ? 32.GB : task.attempt == 2 ? 24.GB : 16.GB }
    }
}

This would give the adapter_removal process 16, 24, 32 GB on the first, second and third attempt rescpectively. You will just have to additionally specify this config in your run command:

nextflow run nf-core/eager [...] -c /path/to/my_config.config [...]

This config will overwrite anything in the profiles you are loading.

Thank you Thiseas for quick response and sharing conf file. So like this, bwa will also work ?

yes. You can actually provide the configuration for multiple modules within the same process block too

thats is really great. i think increasingly cpu will be helpful to me at mapping. Thank you very much.

If you are only starting to analyse this data, you might also want to consider testing eager3 with it ^_^ (

-r dev

) Input structure and parameter names have changed, so setting it up will take a bit of time. but it is designed to be much more convenient to use with deeper SG genomes, as it can shard mapping input into many smaller jobs, instead of one long one. That also means if any fail, you only have to redo that one part, minimising time losses

Sure, i will check eager3 as well, at this moment just wanted to see some stats of recent sequence, but its good to check eager3 i think and will try next months and will let you know. Thanks

> Exit code 140 typically indicates a process failed due to resource constraints such as memory, disk space, CPU, or time limits on Slurm or SGE clusters. Suggests you need to bump up some resource or the time limit

hii yes actually, earlier i was giving the 20 cpu but over two node i think, and i understand all tools are perform multi-threading but with single node. at currently with singel node and 40 cput with 100 hrs, its running fast and smooth. Thank you for your suggestion and help

I think this all depends on your cluster configuration tbh. nextflow should submit each process as a separate job to your scheduler, and the scheduler deals with where and how those get distributed to the nodes

I see, thats all. Thank you for your time and kind help.

That’s hard to answer without knowing the exact configuration you had before vs now. In general, the number of concurrent jobs Nextflow will submit are governed by the

executor.queueSize

attribute, so your config can include a block like this to run 8 jobs in parallel:

executor {
    queueSize = 8
}

It could be that your original configuration was defining this attribute somewhere, and you have now either overwritten it, or replaced it with a value your cluster configuration does not allow

Hii thank you for your response.

ok. I think the default queueSize is sth like

100

so that’s probably not it then. Do your jobs get queued, just not ran in parallel, or is nextflow queueing them one by one? Does your scheduler have a fair share system? The time limits on your jobs are quite large. Maybe once one job is running your get down-prioritised a lot because of the time limits, and hence no other jobs run? 🤔

Hi Thiseas, I am very sorry for the late response, as I was away from the lab. Yes, the job does not get queued and does not run in parallel. It's like right now bwa aln is running, but it's running each sample one by one, and when it gets over, then it moves to the next stage, and again that stage will run for one by one. When I was not using config, I remember the job was getting submitted independently, like BWA index, seq, and adapter removal. However, once this job is complete, I will try to run them without config and see. Thank you very much.

Are you still loading the profile/config for your cluster, on top of your new config with the resource tweaks?