I've twice now run into differences in how a proje...
# configs
d
I've twice now run into differences in how a project local config file and the repository
conf/modules.config
is interpreted. The below block works fine when distributed with the pipeline, but gives an error: `Unknown config attribute
process.withName:MEGAHIT_INTERLEAVED.params.publish_dir_mode
-- check config file` when I place it in a local config, either a
nextflow.config
in my work directory or a file passed with
-c
. Changing the
mode
line to
mode: { params.publish_dir_mode },
fixes the problem. Isn't this puzzling?
Copy code
withName: MEGAHIT_INTERLEAVED {
        publishDir = [
            path: { "${params.outdir}/megahit" },
            mode: params.publish_dir_mode,
            pattern: '**/{megahit_assembly.contigs.fa.gz,megahit_assembly.log}'
        ]
    }
p
I'd expect this is due to scoping and where various things are visible. I also believe there are slightly more dragons sleeping here (e.g. what happens if another file redefines
params.publish_dir_mode
). Generally I believe using a closure will be the right thing here.
m
Putting
mode: { params.publish_dir_mode }
is returning null and so you'll likely be getting the default setting - i.e.
symlink
( unless this has been changed recently ). The params used in your config need to also be defined there since Nextflow resolves configs separately. Using a closure does defer when something is resolved, but not everything actually takes a closure and I think
publishDir.mode
is one thing that doesn't.
p
I don't have time to play now, but I thought almost all things were supported as closures - but maybe in this case it would mean defining the
publishDir
itself as a closure - that then returns a map.
m
That's something I've never tried. Only so far the individual parts.
d
I do think the construct with the closure worked like expected, i.e. defaults from
conf/modules.conf
were overriden.
m
but what was the mode you wanted?
symlink
or something else?
d
I actually confused myself a bit, sorry for that. I was not intending to override the
mode
, but the
pattern
. When I didn't have the curly braces around
mode
I got the syntax error. When I did add them,
pattern
was overriden like it should (but then it's not using any
params
).
m
There's no inheritance between configs, so either define it in the same config so it picks up whatever you provide using the command-line, or
-params.file
, or explicitly set a value for
mode:
. I'm pretty sure the closure is just returning
null
, but allowing the interpreter to skip it because closures are valid objects.
p
Would you mind walking through why it would return
null
or some zero value?
m
because it's not defined in that specific config. So Nextflow reads it in, and then tries to evaluate the config and finds it's not defined so returns
null
.
p
I don't think I follow - there's no issue in the evaluation (mode is assigned the closure), and later on when used - if closures are supported for
mode
in
publishDir
it's evaluated (called), otherwise the value is the closure. And I don't think it would ever be a null unless the value from the closure is
null
- if there's an unsatisfied reference in a closure, that's an error.
m
Running a test:
Copy code
workflow {
    TASK(1)
}

process TASK {
    input:
    val it

    script:
    """
    echo "$it" > test.txt
    """

    output:
    path "test.txt"
}
`nextflow.config`:
Copy code
params.publish_dir_mode = 'copy'

process {
    withName: 'TASK' {
        publishDir = [
            path: 'results',
            mode: params.publish_dir_mode
        ]
    }
}
`test.config`:
Copy code
process {
    withName: 'TASK' {
        publishDir = [
            path: 'results',
            mode: { params.publish_dir_mode }
        ]
    }
}
terminal
:
Copy code
gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf 
N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [cheeky_gauss] DSL2 - revision: 6a919b49ed
executor >  local (1)
[81/15c034] process > TASK [100%] 1 of 1 ✔

gitpod /workspace/Nextflow_sandbox (main) $ rm -r results/
gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf -c test.config 
N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [marvelous_edison] DSL2 - revision: 6a919b49ed
executor >  local (1)
[bb/1400c8] process > TASK [  0%] 0 of 1
ERROR ~ Invalid method invocation `setMode` with arguments:  (_nf_config_68ada98a$_run_closure1$_closure2$_closure3) on PublishDir type

 -- Check '.nextflow.log' file for details
I wasn't expecting that error. I thought it wouldn't complain.
p
But there it looks like it doesn't like the closure at least, so it's odd if it works for Daniel.
💯 1
m
Might be a version thing. Which Nextflow version are you using Daniel?
d
23.10.1 build 5891. I was helping @Emma Strand in #ampliseq and @Thomas Mollica in #metatdenovo.
Give me 30' and I'll try to return with an MRE.
m
Re:
I don't think I follow - there's no issue in the evaluation (mode is assigned the closure), and later on when used - if closures are supported for
mode
in
publishDir
it's evaluated (called), otherwise the value is the closure.
And I don't think it would ever be a null unless the value from the closure is
null
- if there's an unsatisfied reference in a closure, that's an error.
You're right there. I'm getting stuff confused
d
I'm getting more confused. I tried #metatdenovo again, and when I had the closure I got: `ERROR ~ Invalid method invocation
setMode
with arguments: (_nf_config_9d9ad10e$_run_closure1$_closure2$_closure4) on PublishDir type`. I still get the original syntax error without the curly braces (also with 23.04.5). Removing the
mode
line entirely makes it work again, and the
publish_dir_mode
parameter is honoured. So, to replicate:
nextflow run nf-core/metatdenovo -r 1.0.0 --outdir results -profile docker,test
, with or without
-publish_dir_mode symlink
and the below
nextflow.config
works like I want it to. (I admit this is not very minimal, but at least replicable.)
Copy code
process {
    withName: MEGAHIT_INTERLEAVED {
        publishDir = [
            path: { "${params.outdir}/megahit" },
            pattern: '**/{megahit_assembly.contigs.fa.gz,megahit_assembly.log}'
        ]
    }
}
m
That makes sense, but then it uses the default
mode:
which is
symlink
.
I'm suprised you see that it's honoured. Let me check
d
Well, it is, I just checked again. Running the pipeline without this parameter copies.
m
O.o Confirmed. It inherits it.
Erm. wait.
d
Which is lucky for me, as this seems to be the only way of getting around that syntax error and I do want to copy.
m
I just tried :
nextflow run main.nf -c test.config --publish_dir_mode 'symlink'
and it
copied
So no inheritance
d
For me with metatdenovo it symlinks.
(I didn't have any quotes though.)
m
OK. Editor is not updating. Testing again
d
And it avoids the files we publish with the default pipeline config, so the local config does its job. `nextflow run nf-core/metatdenovo -r 1.0.0 --outdir results -resume -profile docker,test --publish_dir_mode symlink`:
Copy code
$ ll results/megahit/megahit_out/
total 8
lrwxrwxrwx 1 daluab daluab 89 mar  1 16:40 megahit_assembly.contigs.fa.gz -> /tmp/mt/work/54/cb2dde5ce132ead2025a66614ff9bf/megahit_out/megahit_assembly.contigs.fa.gz
lrwxrwxrwx 1 daluab daluab 79 mar  1 16:40 megahit_assembly.log -> /tmp/mt/work/54/cb2dde5ce132ead2025a66614ff9bf/megahit_out/megahit_assembly.log
m
Copy code
gitpod /workspace/Nextflow_sandbox (main) $ rm -rf results/
gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf -c test.config
N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [goofy_watson] DSL2 - revision: 6a919b49ed
executor >  local (1)
[b7/3fb34b] process > TASK [100%] 1 of 1 ✔

gitpod /workspace/Nextflow_sandbox (main) $ file results/test.txt 
results/test.txt: ASCII text
gitpod /workspace/Nextflow_sandbox (main) $ rm -rf results/
gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf -c test.config --publish_dir_mode 'symlink'
N E X T F L O W  ~  version 23.10.1
Launching `main.nf` [condescending_heisenberg] DSL2 - revision: 6a919b49ed
executor >  local (1)
[27/a3a328] process > TASK [100%] 1 of 1 ✔

gitpod /workspace/Nextflow_sandbox (main) $ file results/test.txt 
results/test.txt: symbolic link to /workspace/Nextflow_sandbox/work/27/a3a328b53442db46311e4c24dd2a07/test.txt
This is unexpected to me. I'm surprised
This was definitely not the behaviour over a year ago so I wonder when it changed
d
As I said in my first post, the syntax error in itself was quite unexpected to me and took me a while to understand.
m
I did a little more testing and it seems the inheritance works only if you use the same exact same label as it was defined in a previous config.