@Jim Downie @Nicolas Vannieuwkerke @Maxime Garcia @pontus let's hash out these two questions on the config specs in a nthread here: • https://github.com/nf-core/website/pull/4087#discussion_r3109041772 • https://github.com/nf-core/website/pull/4087#issuecomment-4280624252 🧵 👇

I think I'm talking mainly specifically about /how/ or /whether/ these dynamic includes can be used

In that case I can't say anything as I've not dealt with someothign so complex, so I will need you/VSC/UPPMAX people to come up with the guidance 😬

I think that the logic of what we are doing at Sanger is sound, most of the infrastructure is the same across the two cluster types and it's good to share logic. But currently we have made a specific and perhaps non-sanctioned approach with a

sanger.conf

that includes configs from a

sanger/

directory

I'll come back for your final decision in about 30m 😉 so please can someone write the final text for then 😉

I think this approach is fine and allowed, but would prefer not to specify this as a recommended way just now.

Is this also a hard requirement? As it has caused more confusion than anything else maybe we should just drop it?

hard requirement

you mean one config == one cluster/infra?

yes

No, which is why it's phrased as a 'SHOULD' . But I personally would rather try to discourage it ot prevent very very complicated configs which are a) harder to maintainer b) will be hard to mass update when we get to Nextflow DSL3 ;)\

Fair enough, I'll try and think of something 😛

I think I'm talking mainly specifically about /how/ or /whether/ these dynamic includes can be used

To be clear, I think there might be more use-cases for this than "supporting slightly different clusters" which is why I bring it up more generally - maybe enabling or disabling certain features via params?

Each cluster or type of infrastructure SHOULD only be represented by a single configuration file. Dependent on the infrastructure this can mean that there is one config per cluster on the system or one configuration file for the whole system.

we can always include configs in configs, so I would even go for must

Each cluster or type of infrastructure SHOULD only be represented by a single configuration file. Dependent on the infrastructure this can mean that there is one config per cluster on the system or one configuration file for the whole system.

I like that

we can always include configs in configs, so I would even go for must

I would keep it as a SHOULD since HPC systems can be very weird

FWIW Nextflow should really support system-level config files that are installed in

/etc/nextflow/

or something

> Each cluster or type of infrastructure SHOULD only be represented by a single configuration file. > > For HPC infrastructure: a single config MAY include multiple related- or linked-HPCs that are dynamically selected within the config. > If multiple HPCs are supported in a single config, sub-configs that are selected based on a condition in the main config SHOULD go in in sub directories and included in the main config with

includeConfig

That's good also. Would be good to state that the directory must have the same name as the config basename?

So it would be:

Yeah. Alternative would be to put /everything/ in a directory, and have:

sanger/
    sanger.conf
    hpc1.conf
    hpc2.conf

But that could be taken as a stronger push towards using this approach

Yeah yours is also very clear! Might need some updates to our config but it will make everything more clear

@pontus @Maxime Garcia, is that OK with you?