https://nf-co.re logo
Join Slack
Powered by
publishDir does not allow lists of maps anymore, s...
# pipeline-maintainers
e
publishDir does not allow lists of maps anymore, so publishDir like hits will now fail:
Copy code
publishDir = [
            [
                path: { "${params.outdir}/remove_redundancy/merge_families/full_msa/raw/mgnifam/${meta.id}/" },
                mode: params.publish_dir_mode,
                pattern: "*/full_msa/*",
                saveAs: { filename -> filename.tokenize('/').last() }
            ],
            [
                path: { "${params.outdir}/remove_redundancy/merge_families/hmm/raw/mgnifam/${meta.id}/" },
                mode: params.publish_dir_mode,
                pattern: "*/hmm/*",
                saveAs: { filename -> filename.tokenize('/').last() }
            ],
            [
                path: { "${params.outdir}/remove_redundancy/merge_families/generate_families_iteratively/${meta.id}/" },
                mode: params.publish_dir_mode,
                enabled: params.save_iterative_family_metadata,
                pattern: "*/*.*"
            ]
        ]
Is this wanted behaviour or a bug? What is the suggested way of writing these now? I don't like what the bots are proposing and cannot see any alternative:
Copy code
publishDir = [
            path: { "${params.outdir}/remove_redundancy/merge_families" },
            mode: params.publish_dir_mode,
            saveAs: { filename ->
                def parts = filename.tokenize('/')
                if (parts.size() == 3 && parts[1] == 'full_msa') { return "full_msa/raw/mgnifam/${meta.id}/${parts[2]}" }
                if (parts.size() == 3 && parts[1] == 'hmm')      { return "hmm/raw/mgnifam/${meta.id}/${parts[2]}" }
                if (!params.save_iterative_family_metadata)      { return null }
                if (parts.size() == 2 && parts[1].contains('.')) { return "generate_families_iteratively/${meta.id}/${filename}" }
                return null
            }
        ]
p
Output directives? homer disappear
e
but but..before that becomes mandatory...? 😛
p
Pin the maximum Nextflow version? homer disappear
e
it's an nf-core tools problem
the check 
config -flat
changed to 
config -o json
. By Claude:
Copy code
-flat prints dotted key/value pairs and never serializes closures. -o json runs Groovy's JsonOutput over the config tree  — and that's where closures nested inside list elements (never wrapped as ConfigClosurePlaceholder, unlike map values) get invoked against a JsonDelegate, giving No such variable: meta.
n
Imo the bots suggestion is not bad, I would rather use some pattern matching that replicates what you did with 
pattern:
before on the filenames instead of tokenizing it?
m
What! That has to be a bug
this 2
e
that's what Claude says as well..:
Copy code
So the list-of-maps was always latently unserializable; nothing ever asked for JSON until 4.1.0. Nextflow's asymmetric closure wrapping is arguably the real bug — worth an issue on nextflow-io/nextflow if you want it fixed at the source rather than worked around in every pipeline.
👍🏻 1
There are quite a bunch of pipelines using [ [], [] ]...so I guess more same errors will pop soon
j
e
Thanks Jim...!
DOes latest relased Nextflow version fix that then? And should we use that in the latest tempalte?
I think I already tried the altest one and it also failed..
j
You can "fix" it temporarily by making the closures null-safe if evaluated without context:
Copy code
saveAs: { filename -> filename?.tokenize('/') }
❌ 1
👀 1
I think it will work on edge but not the latest release. Ben is backporting the fix to 26.04
But anyway, switch to workflow outputs, they're great
e
nah, same error with 
filename?
😞
j
The whole closure needs to return null if 
filename
is null:
Copy code
saveAs: { filename -> filename?.tokenize('/')?.last() }
⭐ 1
e
Right! This worked:
Copy code
publishDir = [
            [
                path: "${params.outdir}/full_msa/raw/mgnifam",
                mode: params.publish_dir_mode,
                pattern: "*/full_msa/*",
                saveAs: { filename -> filename ? "${meta.id}/${filename.tokenize('/').last()}" : null }
            ],
            [
                path: "${params.outdir}/hmm/raw/mgnifam",
                mode: params.publish_dir_mode,
                pattern: "*/hmm/*",
                saveAs: { filename -> filename ? "${meta.id}/${filename.tokenize('/').last()}" : null }
            ],
            [
                path: "${params.outdir}/generate_families_iteratively",
                mode: params.publish_dir_mode,
                enabled: params.save_iterative_family_metadata,
                pattern: "*/*.*",
                saveAs: { filename -> filename ? "${meta.id}/${filename}" : null }
            ],
            [
                path: "${params.outdir}/generate_families_iteratively",
                mode: params.publish_dir_mode,
                enabled: params.save_iterative_family_metadata,
                pattern: "*/rf/*",
                saveAs: { filename -> filename ? "${meta.id}/${filename}" : null }
            ]
        ]
Thanks again Jim!
5 Views
Open in Slack
PreviousNext
Powered by