Daniel Lundin
03/01/2024, 11:10 AMconf/modules.config is interpreted. The below block works fine when distributed with the pipeline, but gives an error: `Unknown config attribute process.withName:MEGAHIT_INTERLEAVED.params.publish_dir_mode -- check config file` when I place it in a local config, either a nextflow.config in my work directory or a file passed with -c. Changing the mode line to mode: { params.publish_dir_mode }, fixes the problem. Isn't this puzzling?
withName: MEGAHIT_INTERLEAVED {
publishDir = [
path: { "${params.outdir}/megahit" },
mode: params.publish_dir_mode,
pattern: '**/{megahit_assembly.contigs.fa.gz,megahit_assembly.log}'
]
}pontus
03/01/2024, 12:21 PMparams.publish_dir_mode).
Generally I believe using a closure will be the right thing here.Mahesh Binzer-Panchal
03/01/2024, 1:05 PMmode: { params.publish_dir_mode } is returning null and so you'll likely be getting the default setting - i.e. symlink ( unless this has been changed recently ). The params used in your config need to also be defined there since Nextflow resolves configs separately. Using a closure does defer when something is resolved, but not everything actually takes a closure and I think publishDir.mode is one thing that doesn't.pontus
03/01/2024, 1:08 PMpublishDir itself as a closure - that then returns a map.Mahesh Binzer-Panchal
03/01/2024, 1:10 PMDaniel Lundin
03/01/2024, 2:18 PMconf/modules.conf were overriden.Mahesh Binzer-Panchal
03/01/2024, 2:19 PMMahesh Binzer-Panchal
03/01/2024, 2:20 PMsymlink or something else?Daniel Lundin
03/01/2024, 2:23 PMmode, but the pattern. When I didn't have the curly braces around mode I got the syntax error. When I did add them, pattern was overriden like it should (but then it's not using any params).Mahesh Binzer-Panchal
03/01/2024, 2:26 PM-params.file, or explicitly set a value for mode:. I'm pretty sure the closure is just returning null, but allowing the interpreter to skip it because closures are valid objects.pontus
03/01/2024, 2:27 PMnull or some zero value?Mahesh Binzer-Panchal
03/01/2024, 2:28 PMnull.pontus
03/01/2024, 2:32 PMmode in publishDir it's evaluated (called), otherwise the value is the closure.
And I don't think it would ever be a null unless the value from the closure is null - if there's an unsatisfied reference in a closure, that's an error.Mahesh Binzer-Panchal
03/01/2024, 2:37 PMworkflow {
TASK(1)
}
process TASK {
input:
val it
script:
"""
echo "$it" > test.txt
"""
output:
path "test.txt"
}
`nextflow.config`:
params.publish_dir_mode = 'copy'
process {
withName: 'TASK' {
publishDir = [
path: 'results',
mode: params.publish_dir_mode
]
}
}
`test.config`:
process {
withName: 'TASK' {
publishDir = [
path: 'results',
mode: { params.publish_dir_mode }
]
}
}
terminal :
gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf
N E X T F L O W ~ version 23.10.1
Launching `main.nf` [cheeky_gauss] DSL2 - revision: 6a919b49ed
executor > local (1)
[81/15c034] process > TASK [100%] 1 of 1 ✔
gitpod /workspace/Nextflow_sandbox (main) $ rm -r results/
gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf -c test.config
N E X T F L O W ~ version 23.10.1
Launching `main.nf` [marvelous_edison] DSL2 - revision: 6a919b49ed
executor > local (1)
[bb/1400c8] process > TASK [ 0%] 0 of 1
ERROR ~ Invalid method invocation `setMode` with arguments: (_nf_config_68ada98a$_run_closure1$_closure2$_closure3) on PublishDir type
-- Check '.nextflow.log' file for detailsMahesh Binzer-Panchal
03/01/2024, 2:37 PMpontus
03/01/2024, 2:38 PMMahesh Binzer-Panchal
03/01/2024, 2:39 PMDaniel Lundin
03/01/2024, 2:40 PMDaniel Lundin
03/01/2024, 2:42 PMMahesh Binzer-Panchal
03/01/2024, 2:47 PMI don't think I follow - there's no issue in the evaluation (mode is assigned the closure), and later on when used - if closures are supported forinmodeit's evaluated (called), otherwise the value is the closure.publishDir
And I don't think it would ever be a null unless the value from the closure isYou're right there. I'm getting stuff confused- if there's an unsatisfied reference in a closure, that's an error.null
Daniel Lundin
03/01/2024, 3:31 PMsetMode with arguments: (_nf_config_9d9ad10e$_run_closure1$_closure2$_closure4) on PublishDir type`. I still get the original syntax error without the curly braces (also with 23.04.5). Removing the mode line entirely makes it work again, and the publish_dir_mode parameter is honoured. So, to replicate: nextflow run nf-core/metatdenovo -r 1.0.0 --outdir results -profile docker,test , with or without -publish_dir_mode symlink and the below nextflow.config works like I want it to. (I admit this is not very minimal, but at least replicable.)
process {
withName: MEGAHIT_INTERLEAVED {
publishDir = [
path: { "${params.outdir}/megahit" },
pattern: '**/{megahit_assembly.contigs.fa.gz,megahit_assembly.log}'
]
}
}Mahesh Binzer-Panchal
03/01/2024, 3:32 PMmode: which is symlink.Mahesh Binzer-Panchal
03/01/2024, 3:33 PMDaniel Lundin
03/01/2024, 3:34 PMMahesh Binzer-Panchal
03/01/2024, 3:36 PMMahesh Binzer-Panchal
03/01/2024, 3:37 PMDaniel Lundin
03/01/2024, 3:37 PMMahesh Binzer-Panchal
03/01/2024, 3:38 PMnextflow run main.nf -c test.config --publish_dir_mode 'symlink' and it copiedMahesh Binzer-Panchal
03/01/2024, 3:39 PMDaniel Lundin
03/01/2024, 3:39 PMDaniel Lundin
03/01/2024, 3:39 PMMahesh Binzer-Panchal
03/01/2024, 3:40 PMDaniel Lundin
03/01/2024, 3:41 PM$ ll results/megahit/megahit_out/
total 8
lrwxrwxrwx 1 daluab daluab 89 mar 1 16:40 megahit_assembly.contigs.fa.gz -> /tmp/mt/work/54/cb2dde5ce132ead2025a66614ff9bf/megahit_out/megahit_assembly.contigs.fa.gz
lrwxrwxrwx 1 daluab daluab 79 mar 1 16:40 megahit_assembly.log -> /tmp/mt/work/54/cb2dde5ce132ead2025a66614ff9bf/megahit_out/megahit_assembly.logMahesh Binzer-Panchal
03/01/2024, 3:42 PMgitpod /workspace/Nextflow_sandbox (main) $ rm -rf results/
gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf -c test.config
N E X T F L O W ~ version 23.10.1
Launching `main.nf` [goofy_watson] DSL2 - revision: 6a919b49ed
executor > local (1)
[b7/3fb34b] process > TASK [100%] 1 of 1 ✔
gitpod /workspace/Nextflow_sandbox (main) $ file results/test.txt
results/test.txt: ASCII text
gitpod /workspace/Nextflow_sandbox (main) $ rm -rf results/
gitpod /workspace/Nextflow_sandbox (main) $ nextflow run main.nf -c test.config --publish_dir_mode 'symlink'
N E X T F L O W ~ version 23.10.1
Launching `main.nf` [condescending_heisenberg] DSL2 - revision: 6a919b49ed
executor > local (1)
[27/a3a328] process > TASK [100%] 1 of 1 ✔
gitpod /workspace/Nextflow_sandbox (main) $ file results/test.txt
results/test.txt: symbolic link to /workspace/Nextflow_sandbox/work/27/a3a328b53442db46311e4c24dd2a07/test.txt
This is unexpected to me. I'm surprisedMahesh Binzer-Panchal
03/01/2024, 3:43 PMDaniel Lundin
03/01/2024, 3:44 PMMahesh Binzer-Panchal
03/01/2024, 7:40 PM