I'm having trouble running nextflow on our HPC sys...
# configs
p
I'm having trouble running nextflow on our HPC system using the institutional config (
nextflow run nf-core/rnaseq -profile test,vsc_calcua,skylake_local --outdir test_output
). I get the following error message:
ERROR ~ Unknown config attribute host
.
host
is a local parameter defined in the config (https://github.com/nf-core/configs/blob/2842f014f342f177ad9417b284262bfa630261e3/conf/vsc_calcua.config#L13). Did anything change with regards to I'd contact the authors for help, but that'd be me 🄲 I've verified that removing all lines that reference host makes the config work again, but I haven't figured out why it suddenly started causing issues.
s
So you tried with a copy of the config as a file, and then specified using
-c
rather than
-profile
? What if you remove the
def
from the
def host = System.getenv("VSC_INSTITUTE")
line?
p
I had to use
--custom_config_base
pointing to a full clone of the nf-core/configs repo, rather than
-c
to a single file, but yep. Also, in case that wasn't clear from the start, I'm quite sure this config did work properly in the past... Anyway, what I've tried so far: • removing all lines containing
host
works • using an older version of nextflow doesn't work • defining
def host
, but not using the
host
variable at any later point works • changing references to
$host
doesn't work • removing
def
from the assignment line works! (your suggestion) But why and why now though? šŸ¤” I use a def in other places in the config too...and I borrowed the idea from another config even.
Pinging @Nicolas Vannieuwkerke and @Joon Klaps too, since I think I borrowed that
host
check from either of your UGent/KUL configs.
n
What version of Nextflow are you using? Can you share a log too? šŸ˜›
p
Sure! • Version = 24.04.4 • Full log in attachment, relevant lines are:
Copy code
Aug-07 10:00:00.139 [main] DEBUG nextflow.config.ConfigBuilder - In the following config snippet the attribute `host` is empty:

//
//
//

Aug-07 10:00:00.143 [main] ERROR nextflow.cli.Launcher - Unknown config attribute `host` -- check config file: /user/antwerpen/203/vsc20380/.nextflow/assets/nf-core/rnaseq/nextflow.config
nextflow.exception.ConfigParseException: Unknown config attribute `host` -- check config file: /user/antwerpen/203/vsc20380/.nextflow/assets/nf-core/rnaseq/nextflow.config
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.validate(ConfigBuilder.groovy:465)
	at nextflow.config.ConfigBuilder.validate(ConfigBuilder.groovy:452)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.merge0(ConfigBuilder.groovy:426)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.buildConfig0(ConfigBuilder.groovy:375)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.buildGivenFiles(ConfigBuilder.groovy:320)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.buildConfigObject(ConfigBuilder.groovy:814)
	at org.codehaus.groovy.vmplugin.v8.IndyInterface.fromCache(IndyInterface.java:321)
	at nextflow.config.ConfigBuilder.build(ConfigBuilder.groovy:827)
	at nextflow.cli.CmdRun.run(CmdRun.groovy:328)
	at nextflow.cli.Launcher.run(Launcher.groovy:503)
	at nextflow.cli.Launcher.main(Launcher.groovy:657)
Older sem-relevant github issue basically describes that functions should not be defined in config files, but these used to work and they are obviously used in some other institutional configs as well.
Lastly, I just noticed that the ugent config uses
hostname = ...
rather than
def hostname
, but it does use def elsewhere (e.g.,
def hostname = "doduo"
). Would anyone know what the difference is between defining variables with or without def in config files? Global vs local variables?
n
We define hostname 2 lines above that because we set a default šŸ™‚ That line you are referring to overwrites the default
It is possible something changed in the latest version of nextflow. I haven't tried that version on our HPC yet
p
Woosh. Sorry, still waking up apparently šŸ˜…
šŸ›Œ 1
n
Nah no problem! I'll update the version on our HPC and try it out there šŸ˜›
p
v 23.04.2 shows the same behaviour for what it's worth!
šŸ¤” 1
And the ugent config also seems to run on calcua. So it seems there's something specifically wrong with my calcua config.
n
Okay nice debugging šŸ˜› I'll take a closer look at your config!
p
šŸ™‡
n
host
is not defined in that function
A simple fix would be to add it as a function input šŸ˜›
p
Ooh, I can try that! Just to get this straight though: • If I use
def x = value
in the main body of the file, it will only be accessible in other top level sections, but not inside functions. • If I use
x = value
, it will also be accessible inside functions?
n
Yes correct! But I'd advise to always use
def
to prevent weird issues šŸ˜›
p
Gotcha, many thanks! I also noticed a few other issues, so I'll try to fix those while I'm at it šŸ™‚
n
Allright good luck! šŸ˜‰
p
Just an FYI @Nicolas Vannieuwkerke: I think the env scope you are using to define
APPTAINER_TMPDIR
and
APPTAINER_CACHEDIR
in the ugent config are being ignored. Based on some tests I've run and this warning in the docs. I've resorted to only setting the
NXF_APPTAINER_CACHEDIR
if it's missing (via
singularity.cacheDir
) and just showing warnings for the others (since I can't figure out how to override them inside the config).
n
Yes I know something's wrong there (I just didn't put the time in to try and fix it). Thanks for pointing it out though. The main issue here is that those env variables need to be set in the launch environment while
env
only sets the env variables for processes
p
Exactly my own conclusion, that's reassuring to hear. If you're interested, you can have a look at my (hacky) warn-the-user approach at a later time.
n
That's interesting! I'll put it on my todo, thank you!