Matthias Zepper
03/10/2025, 3:56 PMIncludeConfig !System.getenv('NXF_OFFLINE') && params.custom_config_base ? "${params.custom_config_base}/nfcore_custom.config" : "/dev/null"
I am asking, because to me it seems that Seqera Platform can’t resolve this and thus won’t include the institutional profiles anymore. Because during my tests I had not set _NXF_OFFLINE_ but still could not not access the downloaded profiles on our offline HPC cluster. Since I reckon that certainly half of the clusters in the profiles are offline, this makes we wonder if pipelines with the 3.x.x templates still work on offline HPCs? (without specifying the profile explicitly with -c)
https://nfcore.slack.com/archives/CGRTMASKY/p1741368525283169Maxime Garcia
03/10/2025, 4:06 PMMatthias Zepper
03/10/2025, 4:14 PMVersion 24.1.7_1e80d58 . But I can also start an agent on my private Linux machine and try with the Seqera Cloud hosted version. But if anyone else had another cluster setup already at hand, that would of course have been be useful.Júlia Mir
03/11/2025, 2:33 PMMatthias Zepper
03/11/2025, 2:44 PMMatthias Zepper
03/11/2025, 3:07 PMNXF_OFFLINE=true . But then the negation is wrong, or?
• If NXF_OFFLINE is set, !System.getenv('NXF_OFFLINE') becomes false.
• If NXF_OFFLINE is not set (null/false), !System.getenv('NXF_OFFLINE') becomes true.
Because of the logical AND (&&), The condition !System.getenv('NXF_OFFLINE') && params.custom_config_base only evaluates to true if
• NXF_OFFLINE is not set (false).
• params.custom_config_base is set (true).
In all other cases, the config profiles are not included.
Ironically, the former case is exactly what I have and it still does not work with Seqera Platform. We have not set NXF_OFFLINE , even though our cluster is offline, because that caused other issues previously. But even if we would set it, it would not help, because of the wrong negation?
Or am I missing something?Phil Ewels
03/11/2025, 3:23 PMPhil Ewels
03/11/2025, 3:24 PMPhil Ewels
03/11/2025, 3:24 PMPhil Ewels
03/11/2025, 3:25 PMif (!System.getenv('NXF_OFFLINE') && params.custom_config_base) {
includeConfig = "${params.custom_config_base}/nfcore_custom.config"
} else {
includeConfig = "/dev/null"
}Phil Ewels
03/11/2025, 3:27 PMMatthias Zepper
03/12/2025, 1:36 PMnf-core pipelines download even has a feature to download them for offline use. Before, I could point my params.custom_config_base to a local copy of the configs and had a convenient possibility of using profiles.
> Note that the linked issue had nothing to do with Platform (that I know of)
All I know is, that it does not longer work to load profiles in Platform also when NXF_OFFLINE is not set, so that evaluation does not work.
Even if it worked as intended, Seqera Platform is known to be unable to consolidate configs (tw launch -c erases all pre-configuration provided with tw pipelines add -c). So any form of pipeline pre-configuration is anyway hard and just got even harder. From now on, one has to explicitly provide the full path to e.g. the uppmax profile at every single launch vs. just running -profile "uppmax" . I consider this a major regression.Phil Ewels
03/12/2025, 1:39 PMNXF_OFFLINE isn't set and the base is set, so why would that not evaluate to true?Phil Ewels
03/12/2025, 1:40 PMPhil Ewels
03/12/2025, 1:40 PMPhil Ewels
03/12/2025, 1:40 PMMatthias Zepper
03/12/2025, 3:32 PMI don’t understand why it’s not working for you though. You said thatThat is what I try to understand. I guess it has to do with the communication through the agent, but I do not know what requests are actually sent back and forth before the pipeline is eventually launched. For exampleisn’t set and the base is set, so why would that not evaluate to true?NXF_OFFLINE
params.custom_config_base is set in the GUI, but maybe only transmitted in the final ephemeral file when I click LAUNCH in the GUI and not yet when nextflow in run on the freshly cloned pipeline? So maybe it is params.custom_config_base for the Platform issue and NXF_OFFLINE for local installs?
But either way I don’t see why it’s no longer working for you now?Because we have just installed our first pipeline that uses the new template in Platform. We are usually several releases behind.
Happy to hear alternative suggestions.Maybe just check if the file to include exists?
def institutionalConfig = "${params.custom_config_base}/nfcore_custom.config"
if(file(institutionalConfig).exists()) {
includeConfig = institutionalConfig
} else {
if (!System.getenv('NXF_OFFLINE') && params.custom_config_base) {
System.err.println("WARNING: Could not load nf-core/config profiles: $institutionalConfig")
}
}
Quickly whipped up draft, I would need to test if it changes something with Platform.
If you could create an MRE to try to track down what’s causing that behaviour, that would be helpful. Currently I see no regression in the code alone, and no-one else has reported this problem yet.I will think about it, but a MRE for this is quite hard, because one would need a virtual machine with an agent connecting to Seqera Platform Cloud instance and put the pipelines up on there and add them to Platform as well in a preconfigured manner. I was hoping that you already have some sort of testing instance at Seqera.
Phil Ewels
03/12/2025, 3:53 PMMaybe just check if the file to include exists?Can't do this sorry, new config syntax doesn't allow
def and if at root level of config any more. That's why we had to make this change.Phil Ewels
03/12/2025, 3:54 PMincludeConfig ({
def institutionalConfig = "${params.custom_config_base}/nfcore_custom.config"
if(file(institutionalConfig).exists()) {
includeConfig = institutionalConfig
} else {
if (!System.getenv('NXF_OFFLINE') && params.custom_config_base) {
System.err.println("WARNING: Could not load nf-core/config profiles: $institutionalConfig")
}
}
}())Matthias Zepper
03/12/2025, 4:24 PMtry.Phil Ewels
03/13/2025, 9:43 AMPhil Ewels
03/13/2025, 9:44 AMPhil Ewels
03/13/2025, 9:44 AMPhil Ewels
03/13/2025, 9:45 AMPhil Ewels
03/13/2025, 9:47 AMPhil Ewels
03/13/2025, 9:48 AMincludeConfig statement? I don't see why that shouldn't fix things 🤔Matthias Zepper
03/13/2025, 10:20 AMMatthias Zepper
03/13/2025, 10:25 AMPhil Ewels
03/13/2025, 10:26 AMFelix Lenner
03/14/2025, 7:43 AMNXF_OFFLINE because the cluster is offline, and because I want to use the cluster profile I set custom_config_base to point to a local directory that was downloaded with nf-core download so I can do -profile uppmax. This worked before the template change.
I think it's right? If not offline (online) and base is set, include remote configs.
[4:24 PM]
If NXF_OFFLINE is set, or base is not set, use /dev/nullDon't we want to use /dev/null only if base is not set when NXF_OFFLINE is set? But also make sure that base is not then pointing to the default remote configs, but has been overwritten with local configs? Otherwise I don't see how you would be able to use e.g.
-profile uppmax offline.Felix Lenner
03/14/2025, 7:44 AMPhil Ewels
03/14/2025, 9:54 AMMatthias Zepper
03/14/2025, 10:06 AMPhil Ewels
03/14/2025, 10:16 AM