@Daniel Straub: I'm finishing the long awaited 2nd generation phylo placement (https://github.com/nf-core/ampliseq/issues/787), which includes adding reference trees for archaea and bacteria to the sbdi-gtdb database. I think this might be the most popular way to use phylogenetic placement, but the bacterial tree is very large and two processes hence require much more memory than their

process_high

and

process_medium

respectively allocates to them. To avoid to many retries, I have at the moment set starting memory to them to 60 GiB, hardcoded in

modules.config

.

withName: 'EPANG_PLACE' {
        ext.args   = { "--model ${meta.model}" }
        publishDir = [
            path: { "${params.outdir}/pplace/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" },
            mode: params.publish_dir_mode,
            saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
        ]
        cpus   = { 20    * task.attempt }
        memory = { 60.GB * (1.5 ** (task.attempt - 1)) }
    }

(And similarly for the other process.) This will be overkill for certain cases (e.g. the archaeal tree which is much smaller). If there's a way of allocating memory based on the size of the tree that would be ideal but require a lot of testing. In the meantime this is the best I can come up with. I plan to document this in

usage.md

to make users aware that it might be possible for them to reduce memory requirement.

Good to hear about phylo placement updates!

To avoid to many retries, I have at the moment set starting memory to them to 60 GiB, hardcoded in

modules.config

.

Usually, resource allocations go to conf/base.config. I think it would be better to have all resource management at one place. Potentially you could use the label

process_medium_memory

with 50GB and update it to 60GB, or add a

withName

definition for that specific process.

If there's a way of allocating memory based on the size of the tree that would be ideal but require a lot of testing.

Afaik we have no resource allocation based on input files in nf-core (yet). A colleague and a master student are supposed to tackle dynamic resource allocation based on input files for some processes later this year. Maybe we can learn from that and add it here then. So for the time being we indeed have to go with the maximal settings.

also added a param,

--skip_pplace

set to

true

by default

That doesnt follow our usual pattern, but as long as we dont have a better idea, fine. The PR seems not to include your changes yet.

And third, it would be good to test this but it clearly takes too much resources for the normal CI tests. Should we change the full scale test to use SBDI-GTDB with phylogenetic placement?

Is there a way to add a minimal tree for testing only for test samples? If thats too much trouble, then full scale testing is better than noghting. But thats done so infrequently that any error would come up only very late.

Thanks! I will need to get back to this later; next week.