https://nf-co.re logo
Join Slack
Powered by
I am struggling with downloading a recent version ...
# rnaseq
m
I am struggling with downloading a recent version of the rnaseq pipeline with nf-core tools and could use some inspiration how to fix it. I am using this Pixi environment 
pixi.toml
on Debian Linux 13:
Copy code
[workspace]
authors = ["Matthias Zepper <MatthiasZepper@users.noreply.github.com>"]
channels = ["conda-forge", "bioconda"]
name = "nf-core-tools"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[activation.env]
NXF_APPTAINER_CACHEDIR = "${PIXI_PROJECT_ROOT}/.apptainer-cache"
NXF_SINGULARITY_CACHEDIR = "${PIXI_PROJECT_ROOT}/.apptainer-cache"

[dependencies]
python = ">=3.14.7,<3.15"
nextflow = ">=26.04.6"
nf-core = ">=4.1.0,<5"
apptainer = ">=1.5.4,<2"
When I try downloading with 
pixi run nf-core pipelines download rnaseq -r "3.27.0" -r "3.26.0" -o "rnaseq" -s "singularity" -x "none"
I get:
Copy code
,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 4.1.0 - <https://nf-co.re>


WARNING  Could not find GitHub authentication token. Some API requests may fail.
INFO     Detected Nextflow version 26.04.6
WARNING  Deleting existing output directory: 'rnaseq'
INFO     Saving 'nf-core/rnaseq'
         Pipeline revision: '3.27.0, 3.26.0'
         Use containers: 'singularity'
         Container library: '<http://quay.io|quay.io>'
         Using NXF_SINGULARITY_CACHEDIR': /home/matthias/Coding/Bioinfo/nf-core-tools/.apptainer-cache'
         Output directory: 'rnaseq'
         Include default institutional configuration: 'False'
INFO     Downloading workflow
INFO     Downloading workflow files from GitHub
INFO     Fetching container names for workflow revision 3.27.0 using nextflow inspect. This might take a while.
INFO     Processing workflow revision 3.27.0, found 49 container images in total.
ERROR    Could not parse Nextflow config for 'rnaseq/3_27_0'. Possibly because of a Nextflow version mismatch?
         Command 'nextflow config -o json rnaseq/3_27_0' failed with exit code 1
         WARN: Unrecognized config option 'manifest.diagram'
         ERROR ~ No such variable: task
Indeed, running nextflow config manually gives the same error 
pixi run nextflow config -o json rnaseq/3_27_0
:
Copy code
WARN: Unrecognized config option 'manifest.diagram'
ERROR ~ No such variable: task
This does not happen when I am running it directly against the GitHub version 
pixi run nextflow config -o json nf-core/rnaseq -r "3.27.0"
, which prints a nicely formatted JSON output. For pipeline version 3.26.0, the warnings differ, but the issue persists:
Copy code
ERROR    Could not parse Nextflow config for 'rnaseq/3_26_0'. Possibly because of a Nextflow version mismatch?
         Command 'nextflow config -o json rnaseq/3_26_0' failed with exit code 1
         WARN: Unrecognized config option 'validation.defaultIgnoreParams'
         WARN: Unrecognized config option 'validation.monochromeLogs'
         ERROR ~ No such variable: task
Strangely, the CI test didn't catch this problem, even though the Nextflow version 26.04.6 is the same. nf-core tools is admittedly still 4.0.3 in the test, but since it seems to be an issue of 
nextflow config
, I reckon the tools version is irrelevant here. Pinning Nextflow at 25.10.04 resolves the issue, but gives other issues with recent pipelines such as plugins that require more recent versions. Any idea what the underlying reason could be?
m
the WARN: Unrecognized config option 'manifest.diagram' is definitively not an issue and just a red herring
something is definitively weird with the ERROR ~ No such variable: task
j
Hm, strange, I'll look into it
🙏 1
With apologies for the AI summary (got a bit going on): Looked into the 
nf-core pipelines download
failure on rnaseq 3.26/3.27 (
nextflow config -o json
-> 
No such variable: task
). It's a Nextflow bug, not an rnaseq or nf-core/tools one. Cause: Nextflow 26.04.0 to 26.04.6 evaluates closures nested inside a list of maps when rendering 
config -o json
. Our modules configs use 
publishDir = [ [ path: { ...task... } ] ]
, so it blows up. A single flat map is fine, and so are 
-flat
and plain 
nextflow config
. Tracked upstream in: - github.com/nextflow-io/nextflow/issues/7341 - github.com/nextflow-io/nextflow/issues/7474 Fix status: Fixed in github.com/nextflow-io/nextflow/pull/7375 (merged to master Jul 22) and backported to the STABLE-26.04.x branch on Aug 27, but there's no 26.04.7 release yet. 26.04.6 (Jul 9) predates it: github.com/nextflow-io/nextflow/…/…. I tested the full 3.27.0 config: 25.10.4 passes, 26.04.6 and 26.07.0-edge fail, 26.09.1-edge passes (github.com/nextflow-io/nextflow/…/…). Workaround: Use 26.09.1-edge for the download, or wait for 26.04.7. Pinning 25.10.4 also works but causes the plugin version problems. Why tools CI missed it: CI only tests Nextflow 25.10.4 on PRs. The tests that run Nextflow mock out the config fetch (
fetch_wf_config
returns 
{}
). The one test that calls it for real uses a fresh template pipeline, which has a single-map 
publishDir
. The real-pipeline download tests only fetch methylseq 1.6 and never parse its config. - github.com/nf-core/tools/blob/…/test_download.py - github.com/nf-core/tools/blob/…/pytest.yml Suggested tools follow-ups: - Add a fixture with list-form 
publishDir
and a 
task
closure, run through the real 
fetch_wf_config
. - Include the Nextflow version in the config cache key (it currently hashes only 
nextflow.config
and 
<http://main.nf|main.nf>
). - Add 26.04.x and edge to the download test matrix. Open: I couldn't explain why the reporter's direct 
nextflow config -o json nf-core/rnaseq -r 3.27.0
worked. No change needed in rnaseq.
💡 1
m
No need to apologize - the AI solved it, so thanks for assigning your coworker on it.
blob np 1
Open in Slack
PreviousNext
Powered by