It seems you can add anything at all to the nextfl...
# random
k
It seems you can add anything at all to the nextflow.config file and Nextflow doesn't complain? I added a
cpus
value to indicate 32 cores for the "HPC" profile, but it didn't seem to work. Then my workmate pointed out it should be
process.cpus
. I was surprised that Nextflow wouldn't complain about an obviously bad configuration, so I added
this.doesnt.exist = true
and, yep, Nextflow accepts this just fine. Can anyone explain why?
👌 2
nextflow pet 2
p
Great timing! Try this:
Copy code
NXF_EDGE=1 nextflow self-update
export NXF_SYNTAX_PARSER=v2
And run again.. You should now see a warning in the terminal:
Copy code
WARN: Unrecognized config option 'cpus'
Basically it's because Nextflow config files are just Groovy scripts. You can put pretty much anything in them, including functions and all kinds of other code.
As you've found, this has quite a lot of downsides, so a major effort of the past year has been to write a Nextflow language server which can properly parse Nextflow config files and scripts and do native script compilation.
This allows us to define a stricter syntax, which in turn allows us to give more useful error messages in cases like this.
This functionality has been available in the VSCode Nextflow extension since the Barcelona Summit in October last year
And we're now just starting to bring the language server into the Nextflow runtime itself. That's what the
NXF_SYNTAX_PARSER=v2
thing is above.
You'll be one of the first people to try it out, as the edge release only went out this week 😅
k
Why not just use JSON?
t
Use JSON for what? For the config files, JSON cannot do closures, etc., never mind that you'd have to come up with a syntax for clauses like
withName
, and that would likely be ugly and hard to read. And if something is a simple setting, you can just make a parameter for it, and
-params-file
already supports JSON.
p
There was a long discussion about whether to do this, or how to do it - see https://github.com/nextflow-io/nextflow/issues/2723
TLDR is that formats like YAML / JSON / TOML are not powerful enough for some key configuration syntax. To jam what we wanted in made the syntax very ugly.
Instead the approach has been to write the new config parser, which makes the config language much simpler and easier to validate.
Other additions such as https://github.com/nextflow-io/nextflow/pull/5399 allow configuration to be exported to formats like YAML and JSON (with some limitations, such as closures being converted into strings).
And as @Thomas Danhorn says - it's been possible to pass parameters as YAML / JSON with
-params-file
for a long time.
n
Mind that this will always create warnings for nf-core pipelines as they contain plugin config options. The plugins should be updated first before massive adoption of this feature 😁
p
You're handling that though, right? 😉
n
I'm trying 😁. Should be okay before the next major Nextflow release
p
And even then it'll likely still be opt in
Doubt this will be default until 25.10, would be my guess
n
My goal is to have it in 2.5.0 (not in the upcoming 2.4.0) and try and release it some time after the release of 25.04.0 🙂
Doubt this will be default until 25.10, would be my guess
I hope so 😅
k
So I did upgrade my NF and indicated the v2 parser, and I learned that I had put my
container
directive in my NF script (not the config) after the
output
declaration and it was therefore seen not as the Docker image to use but some invalid part of my output and so it was silently ignored and my process wasn't running inside my container. Wow, that's bad.
I guess the examples all show the indentation of the
input:
and
output:
blocks, leading me (a new NF user) to assume there's some sort of white-space parsing of block structure a la Python/Haskell, but that's not the case. Curious, why didn't these use
{}
blocks like elsewhere? Seems exceedingly dangerous.
Further, the new parser doesn't complain about the bogus config/profile setting if I am not using the given profile. This seems surprising to me. I would expect the entire config to be parsed and validated, but I guess it only evaluates the part that I need? This just feels like I'm waiting for runtime errors that ought to be compile-time errors?
I was also curious if this stricter parser might catch an error like
publishDir 'results', mode: 'xyz'
as a compilation error but it's still a runtime error with "No enum constant nextflow.processor.PublishDir.Mode.XYZ". Seems like that would be a good thing to parse ahead of time?
Here's another irritant. The rules for quoting values are not obvious, so this is correct:
Copy code
path 'out/components/component-*', emit: components
But putting the "emit" argument in quotes is not:
Copy code
path 'out/components/component-*', emit: 'components'
But the error message is "ERROR ~ sculu.nf at 10, 9: Unrecognized output
components
for process `make_components`" without telling me what the problem is.
An interesting side effect of not quoting the "emit" value shows up if I make the mistake of thinking this is a Boolean:
Copy code
path 'foo', emit: true
This is in my "make_components" process, and the error message I get is: "Process 'make_components' has been already used -- If you need to reuse the same component, include it with a different name or include it in a different workflow context". That makes no sense!
It really seems like there's a need to put things that will be evaluated as strings in quotes and things that are code values should be barewords!
Like
path 'foo', emit: foo, optional: yes
gives me the error "ERROR ~ sculu.nf at 24, 42:
yes
is not defined"
t
You are making some good points. Hopefully the new language server will help with a lot of them. I'm not sure what you were expecting for that last example, though -- you are using an uninitialized variable and get `... is not defined`; makes perfect sense to me. If you define
yes
as
true
or
false
, it should work.
k
Here's my point. In
path 'foo', emit: foo, optional: yes
, "foo" is treated as a string and "yes" is treated as a variable but there's no discernible difference for the user.
Or maybe "foo" is treated as a string? I'm not sure what happens to it. It's some sort of label? But the values are handled differently by NF and the syntax does nothing to separate them.
Actually, really interesting point you made. I can set
yes = true
in my code and then write
path 'foo', emit: yes, optional: yes
and I guess that means it will be an optional output that can be found in
process.out.true
. Yikes.
t
I think the way it works is that anything unquoted will be treated like a variable (unless it is already a keyword, like
true
), and anything quoted as a string. Since types are optional in Groovy and basically discouraged/forbidden in Nextflow, you can assign a boolean to a string without error, and the language will try to do the most sensible thing at its disposal, e.g. a string with value
'true'
. Of course if you use a bareword that turns out to be a variable, and you are unaware of its content, you might be in for a surprise. I think at the root of the issue is the lack of strong typing (in C assigning a string to number or boolean or vice versa just can't be done, except maybe with very explicit casting), but that is the very thing that makes scripting languages very powerful and convenient. (You probably would not be happy if you had to develop and compile a C program with variable declarations for every 10-line Python you whipped up to do something real quick.) Definitely a double-edged sword.
k
Mostly I'm just trying to share my frustrations as a new user to NF and how the tooling fails to give me the feedback I need to get my pipeline working. I have almost 30 years of programming experience, I know half a dozen languages including Perl/Python and Rust, and I've written HPC/cloud pipelines in bash and WDL. It seems your expectation is that all users will know how to configure their editors with LSPs to catch all the problems that the compiler won't. If I'm shelled into my uni HPC and all I have is vi, I'm (probably) not going to have any help beyond what the NF runtime gives me.
p
Lots of interesting examples here, thanks! Hopefully we can add to the parser to catch many of these, 'cc @Ben Sherman
Worth remembering that the language server is very new. Until only a few months ago, Nextflow code was just Groovy
The feedback that we got loud and clear was that it could be frustrating to only find these kinds of errors at runtime, which is why we have put a tonne of effort into writing a custom language parser / server.
There is a lot more coming, it's still early days for this evolution of the language - check some of the discussion between @Ben Sherman and me in the Nextflow podcast if you're curious: https://seqera.io/podcasts
We discuss typing and a lot more around the language in our various episodes (more coming in the weeks to come too)
The tooling is also continuing to extend - for example, support for vi hopefully! Also command line syntax checking / linting and integration of the language parser into Nextflow for immediate static compilation and error checking on launch.
But yeah - having concrete syntax examples like this is super helpful. So please keep posting them!
We have been saying to use the forum for this: https://community.seqera.io/vscode but this thread is also fine as you're here :)
b
@Ken Youens-Clark you are speaking my language 😆 the emit being allowed as a string or identifier has always bothered me. IMO the whole process input/output syntax needs to be torn out and replaced with something more normal about the container directive, are you saying it was silently ignored by the v2 parser? the v2 parser should give a compile-time error Anyway, I hope you can appreciate that it took me about a year just to get us to where we are now -- a decent parser in Nextflow as an opt-in feature. It will probably take another year to add types and type checking and fix the rest of these weird errors (hopefully in pieces rather than all at the end). We're gonna make Nextflow a proper language with proper tooling. I hope you can handle the wait in the meantime 😄
k
I had my
container
directive/declaration(?) after the
output:
and the v1 parser silently ignores it and runs the program in my current environment. I didn't realize the problem because I have the same program installed locally.
👍 1