Repost from <#CGRTMASKY|configs>, because we figur...
# tools
m
Repost from #configs, because we figured that my template question is more appropriate for #tools : What was the rationale for making the inclusion of the institutional configs conditional?
Copy code
IncludeConfig !System.getenv('NXF_OFFLINE') && params.custom_config_base ? "${params.custom_config_base}/nfcore_custom.config" : "/dev/null"
I am asking, because to me it seems that Seqera Platform can’t resolve this and thus won’t include the institutional profiles anymore. Because during my tests I had not set _NXF_OFFLINE_ but still could not not access the downloaded profiles on our offline HPC cluster. Since I reckon that certainly half of the clusters in the profiles are offline, this makes we wonder if pipelines with the 3.x.x templates still work on offline HPCs? (without specifying the profile explicitly with
-c
) https://nfcore.slack.com/archives/CGRTMASKY/p1741368525283169
m
on which platform release have you tried that?
m
Version 24.1.7_1e80d58
. But I can also start an agent on my private Linux machine and try with the Seqera Cloud hosted version. But if anyone else had another cluster setup already at hand, that would of course have been be useful.
j
here is the issue describing these change https://github.com/nf-core/tools/issues/3132 in case you want to follow the whole story
m
Thanks for the pointer. I knew there must have been a rationale…
Ok, so from the discussion I get, that @Phil Ewels wanted the profiles to be included if people have specified
NXF_OFFLINE=true
. But then the negation is wrong, or? • If
NXF_OFFLINE
is set,
!System.getenv('NXF_OFFLINE')
becomes
false
. • If
NXF_OFFLINE
is not set (null/false),
!System.getenv('NXF_OFFLINE')
becomes
true
. Because of the logical AND (
&&
), The condition
!System.getenv('NXF_OFFLINE') && params.custom_config_base
only evaluates to
true
if •
NXF_OFFLINE
is not set (false). •
params.custom_config_base
is set (true). In all other cases, the config profiles are not included. Ironically, the former case is exactly what I have and it still does not work with Seqera Platform. We have not set
NXF_OFFLINE
, even though our cluster is offline, because that caused other issues previously. But even if we would set it, it would not help, because of the wrong negation? Or am I missing something?
p
So final code in the template is here
I think it's right? If not offline (online) and base is set, include remote configs.
If NXF_OFFLINE is set, or base is not set, use /dev/null
Copy code
if (!System.getenv('NXF_OFFLINE') && params.custom_config_base) {
    includeConfig = "${params.custom_config_base}/nfcore_custom.config"
} else {
    includeConfig = "/dev/null"
}
Note that the linked issue had nothing to do with Platform (that I know of), it was due to problems with the new language server / VSCode, and the upcoming stricter Nextflow syntax.
m
> I think it’s right? If not offline (online) and base is set, include remote configs. Yes, that is indeed what is implemented, but I can’t wrap my head around why this behavior is intentional? Why would the institutional configs matter only for online users?
nf-core pipelines download
even has a feature to download them for offline use. Before, I could point my
params.custom_config_base
to a local copy of the configs and had a convenient possibility of using profiles. > Note that the linked issue had nothing to do with Platform (that I know of) All I know is, that it does not longer work to load profiles in Platform also when
NXF_OFFLINE
is not set, so that evaluation does not work. Even if it worked as intended, Seqera Platform is known to be unable to consolidate configs (
tw launch -c
erases all pre-configuration provided with
tw pipelines add -c
). So any form of pipeline pre-configuration is anyway hard and just got even harder. From now on, one has to explicitly provide the full path to e.g. the
uppmax
profile at every single launch vs. just running
-profile "uppmax"
. I consider this a major regression.
p
I don't understand why it's not working for you though. You said that
NXF_OFFLINE
isn't set and the base is set, so why would that not evaluate to true?
Before we had a try / except, which we can no longer do. I was thinking that the catch was primarily to deal with people running offline, when folks don't have access to the shared config. But yeah maybe this is too simplistic and there's a better way to deal with both scenarios - happy to hear alternative suggestions.
But either way I don't see why it's no longer working for you now?
If you could create an MRE to try to track down what's causing that behaviour, that would be helpful. Currently I see no regression in the code alone, and no-one else has reported this problem yet.
m
I don’t understand why it’s not working for you though. You said that
NXF_OFFLINE
isn’t set and the base is set, so why would that not evaluate to true?
That is what I try to understand. I guess it has to do with the communication through the agent, but I do not know what requests are actually sent back and forth before the pipeline is eventually launched. For example
params.custom_config_base
is set in the GUI, but maybe only transmitted in the final ephemeral file when I click
LAUNCH
in the GUI and not yet when
nextflow
in run on the freshly cloned pipeline? So maybe it is
params.custom_config_base
for the Platform issue and
NXF_OFFLINE
for local installs?
But either way I don’t see why it’s no longer working for you now?
Because we have just installed our first pipeline that uses the new template in Platform. We are usually several releases behind.
Happy to hear alternative suggestions.
Maybe just check if the file to include exists?
Copy code
def institutionalConfig = "${params.custom_config_base}/nfcore_custom.config"
if(file(institutionalConfig).exists()) {
    includeConfig = institutionalConfig
} else {
    if (!System.getenv('NXF_OFFLINE') && params.custom_config_base) {
        System.err.println("WARNING: Could not load nf-core/config profiles: $institutionalConfig")
    }
}
Quickly whipped up draft, I would need to test if it changes something with Platform.
If you could create an MRE to try to track down what’s causing that behaviour, that would be helpful. Currently I see no regression in the code alone, and no-one else has reported this problem yet.
I will think about it, but a MRE for this is quite hard, because one would need a virtual machine with an agent connecting to Seqera Platform Cloud instance and put the pipelines up on there and add them to Platform as well in a preconfigured manner. I was hoping that you already have some sort of testing instance at Seqera.
p
Maybe just check if the file to include exists?
Can't do this sorry, new config syntax doesn't allow
def
and
if
at root level of config any more. That's why we had to make this change.
shakefist 2
The syntax in the docs above might work though, eg:
Copy code
includeConfig ({
    def institutionalConfig = "${params.custom_config_base}/nfcore_custom.config"
    if(file(institutionalConfig).exists()) {
         includeConfig = institutionalConfig
    } else {
        if (!System.getenv('NXF_OFFLINE') && params.custom_config_base) {
            System.err.println("WARNING: Could not load nf-core/config profiles: $institutionalConfig")
        }
    }
}())
🙏 1
m
Ah, I see. I thought it was only about the
try
.
p
Right, apparently my memory is appalling, but @Rike Hanssen managed to jog it. Yes we did come across this already. I don't think that it's specific to the Agent, I think it's probably restricted to older versions of Platform though.
Things bounced around a bit, but it should be properly fixed and compatible from Enterprise v24.2.4 onwards - see the changelog.
So I suspect that if you do a test run using Seqera Cloud then it should work as you'd expect
There was some talk of patching older versions of platform, so if you're unable to upgrade Enterprise but could upgrade to a patch release then please open a support ticket to request this.
I'm not sure that this is something that we can / should fix in nf-core - or at least I couldn't think of a good solution when we discussed this internally last year. It should only affect a pretty small number of people: running an old version of Enterprise and using the latest nf-core pipelines (a segment that you unfortunately fall into). Every alternative syntax that I could think of would not work with the language server and upcoming stricter syntax, so if we reverted we would just be causing ourselves more problems further down the line.
I wonder if you can do a workaround by using a custom config in Platform with the
includeConfig
statement? I don't see why that shouldn't fix things 🤔
m
Oh, if it is fixed in a newer platform release, that is actionable on our side, we can just turf that to KTH IT. 😄 Doing a test run on Sequera Cloud is unfortunately not possible, because Miarka is not allowed to connect to the big dangerous world of the world wide web 😉. Otherwise I would long have pushed for switching to the hosted service because of the lower maintenance overhead.
🤣 1
👍🏻 1
👍 1
Yes, I agree, if it is only a problem for this small group of users, there is no reason for changing the template. It is just hard for me to understand which of the many parts (agent, cli, platform release, nextflow version, pipeline) is to blame if things stop working. But thanks a lot for the help in figuring that out to you Rike, Phil, Maxime, Julia and James.
p
Yeah, there are many moving parts! We're trying to get better at testing all these things so that we can identify (and hopefully resolve) incompatibilities before stuff gets released. But it's not trivial..
f
Hi, I'm not running with Platform, I set
NXF_OFFLINE
because the cluster is offline, and because I want to use the cluster profile I set
custom_config_base
to point to a local directory that was downloaded with
nf-core download
so I can do
-profile uppmax
. This worked before the template change.
I think it's right? If not offline (online) and base is set, include remote configs.
[4:24 PM]
If NXF_OFFLINE is set, or base is not set, use /dev/null
Don't we want to use /dev/null only if base is not set when NXF_OFFLINE is set? But also make sure that base is not then pointing to the default remote configs, but has been overwritten with local configs? Otherwise I don't see how you would be able to use e.g.
-profile uppmax
offline.
p
Yup, I think we were a little hasty and didn't consider people running offline with a local copy of the configs. Apologies. Anyone fancy putting in a pull-request to the template with Ben's suggested syntax please? 🙏🏻
m
I can do.
🙏 1
p
Thank you!