A stupid question (it truly is). When updating the template to a new version, how do you solve the conflict for

ro-crate-metadata.json

? I mean, how do you update just what's necessary, but keep the description text? I don't know how to do it easily as the description is just a wall of text. Thanks!

Admittedly newly returning to nextflow after a long stretch, I'm confusing myself a bit when reading the configuration of profiles and resourceLimits for multiple queues. I'm trying to learn from the explanations and the examples here: nf-co.re/docs/developing/institutional-profiles/configuration#… I understand that

process {
  queue = { task.time <= 24.h ? 'general' : 'long' }
}

will switch queue based on the job duration, while profiles can be used to pick specific partitions to submit jobs to (as explained in this ARCC example)

profiles {
  bigmem {
    process {
      resourceLimits = [
        cpus: 64,
        memory: 4.TB,
        time: 168.h
      ]
      queue = 'bigmem'
    }
  }
}

My question is whether profiles (i.e. resourceLimits) kick in when the queue switches at the process scope, or whether the relevant profiles must be selected at launch, i.e.

nextflow -profile general,long,bigmem

Hello all, I am attempting to use the retry strategy as in example below

withLabel: ld_annotation {
        cpus          = 2
        memory        = { [8.GB, 24.GB, 48.GB][Math.min(task.attempt - 1, 2)] }
        time          = 60.m
        maxRetries    = 2
        errorStrategy = { task.exitStatus in [137, 50002] ? 'retry' : 'terminate' }
    }

The ld_annotation job is running over the google cloud batch and it sometimes needs retries to get a machine with bigger memory. When that happens (retry kicks in) and later in the pipeline I encounter another failure on that branch, if I resume, the cache is invalid and the whole branch restarts from the ld_annotation yet again. Is it something obvious that could explain the cache invalidation on retry? I have never stamped into this problem before, but can see a few folks have lately similar issues github.com/nextflow-io/nextflow/issues/6884 @Evangelos Karatzas FYI

is there any issues with using mpi processes in nf-core modules/pipelines?

Is it normal that linting raises a warning for

process_standard_label: Conflicting process labels found: process_high,process_long

I thought process_long was specifically intended to be stacked up with another process label?

Dear all, I would like to make a fix some minor issues in a sentieon module. While I can test locally using a license (by setting the

SENTIEON_LICSRVR_IP

env. variable) I'm unable to replicate the successful tests in the CI. What am I missing?

I am unable to retrieve raw.githubusercontent.com/nf-core/…/genomeassembler.config (400 bad request) but e.g. raw.githubusercontent.com/nf-core/…/rnaseq.config is fine. Is this a problem on my end, or on githubs end?

Can someone help with this: github.com/nf-core/proteinfamilies/…/100247688583?pr=189 ? I reverted the #C07URFZF074 Nextflow and nf-schema versions to match the template ones, but I get this error on the GitHub runner:

WARN: Unrecognized config option 'validation.defaultIgnoreParams'
         WARN: Unrecognized config option 'validation.monochromeLogs'
         ERROR ~ No such variable: meta

Dear all, are there nextflow tools which, starting from a pipeline's generic nextflow.config, provide the ability of creating a custom nextflow.config file via web form? I was wondering if the information provided in the nextflow_schema.json can be reused to render the help text and organize the interactive options of the form. The question comes from the fact that, often times, the pipelines that I write are used by technical staff that doesn't necessarily feel comfortable checking a nextflow.config or a nextflow_schema.json. The idea would be to let them configure a pipeline starting from the defaults and help text already set in the nextflow.config/nextflow_schema.json

Hi all, is there a way to run only specific processes/subworkflows of an nf-core pipeline without executing the full pipeline? Context: I ran

nf-core/rnaseq v3.11.1

on a combined sample cohort on Seqera. I now want to re-run just the merge/report step without re-doing alignment/quantification (the most time consuming + cost consuming step) as the cohort came from 7 different datasets, so original 1 merged metrics doesn't work. I tried

-resume

the run with the the updated sample sheet, but looks like it still run from scratch instead of hitting cache. Alternatively, can I divided merged.gene_counts.tsv into different dataset? Is there a recommended approach for this kind of partial re-run? Thanks!

Looking at https://nf-co.re/docs/developing/components/gpu-modules, I became unsure - what is the correct way for a user/institutional config to flag the lack of GPUs?

This might not be exactly an nf-core question but it's somewhat related. I noticed that modules using Wave containers (community.wave.seqera.io) seems to fail when using Singularity with

ext.singularity_pull_docker_container = true

It looks like the

PATH

environment variable is supposed to be updated by

shell-hook.sh

which is called in the

ENTRYPOINT

section in the Dockerfile, but

singularity exec

doesn't invoke the

ENTRYPOINT

section which is translated to

%runscript

when converting from OCI image to SIF. The native Singularity images don't have this issue because the

shell-hook.sh

script is invoked in the

%environment

section. While this is not a critical issue, it would be nice to be consistent with where and when we update the environment variables and to allow users to run OCI-converted Wave containers with Singularity.

when nextflow is

staging foreign file

, is there any way to tell when it is done with that? As far as I am aware it does not report when it finished, but maybe I am missing something

Dear all, I started to update a few sentieon modules, then I thought of updating them all. Besides a flaky

sentieon/rsemcalculateexpression

test, updating

sentieon/rsemcalculateexpression

opened up a bigger question: the linter raised an

Invalid 'meta' keys detected: meta.strandedness

error, so I tried to make

strandedness

a parameter. I updated the tests accordingly. However, this module is used in a nf-core subworkflow, which now (understandably) complains about the change in number of parameters. Is a

strandedness

value channel even desirable? How would the community approach these issues?

Hi there. New to Nextflow and trying to understand how to best approach it, cause had already a couple of first failed attempts. First experiment started with a workflow that would handle things string by string, one string at a time (kind of). And the workflow alone looked like this:

workflow {
    main:
    printPythonVersions()

    def repo_root = projectDir.parent

    def cells_channel = channel.fromPath(params.lastpass_export, checkIfExists: true)
        .splitCsv()
        .flatten()
        // .take(params.take_only) //TODO REMOVE DEMO
        .view { cell -> "Processing cell: ${cell}" }

    def base_tokens = tokenizeLine(params.tokenize_bases_script, repo_root, cells_channel)
        .splitText()
        .map { line -> line.trim() } // makes the stdout into a channel of lines
        .filter { token -> !token.isEmpty() }
        .unique()

    def case_variants_tokens = case_variants(params.tokenize_cases_script, repo_root, base_tokens)
        .splitText().map { line -> line.trim() }

    def one_leet_variants_tokens = one_leet_variants(params.one_leet_variants_script, repo_root, base_tokens)
        .splitText().map { line -> line.trim() }

    def commonAffixesChannel = channel.fromList(get_common_affixes())

    def prefixed_tokens = base_tokens.combine(commonAffixesChannel).map { token, affix -> "${affix}${token}" }
    def suffixed_tokens = base_tokens.combine(commonAffixesChannel).map { token, affix -> "${token}${affix}" }

    def all_candidates_file = base_tokens
        .mix(case_variants_tokens)
        .mix(one_leet_variants_tokens)
        .mix(prefixed_tokens)
        .mix(suffixed_tokens)
        .filter { candidate -> max_length_filter(candidate) } // Example: filter out candidates longer than 128 characters
        .unique()
        .collectFile(name: "all_candidates_from_nf.txt", newLine: true)

    publish:
    all_candidates = all_candidates_file
 }

The result of this first experiment would be to burn all of the laptop's resources for 4 hours and then, by spinning hundrends of thousands of tasks, etc. leading to an out of memory. In a second experiment I tried to redesign the whole thing to work in batches:

workflow {
    main:
    printPythonVersions()

    def repo_root = projectDir.parent

    def cells_channel = channel.fromPath(params.lastpass_export, checkIfExists: true)
        .splitCsv()
        .flatten()
        // .take(params.take_only) //TODO REMOVE DEMO
        // .view { cell -> "Processing cell: ${cell}" } //too slow in full run

    def base_tokens = tokenizeLine(params.tokenize_bases_script, repo_root, cells_channel)
        .splitText()
        .map { line -> line.trim() } // makes the stdout into a channel of lines
        .filter { token -> !token.isEmpty() }
        .unique()

    def base_batches = base_tokens
        .collate(params.batch_size)
        .map { batch -> batch.join('\n') + '\n' }

    def base_parts = write_batch_parts(base_batches)
    // def base_file = base_parts.collectFile(name: 'base_parts_from_nf.txt', newLine: false)

    def case_variants_parts = case_variants_batches(params.tokenize_cases_script, repo_root, base_batches)
    // def case_variants_file = case_variants_parts.collectFile(name: 'case_variants.txt', newLine: false)

    def one_leet_variants_parts = one_leet_variants_batches(params.one_leet_variants_script, repo_root, base_batches)
    // one_leet_variants_parts
    //     .count()
    //     .view { cnt -> "one_leet_variants_parts count: ${cnt}" }
    // def one_leet_variants_file = one_leet_variants_parts.collectFile(name: 'one_leet_variants.txt', newLine: false)

    def common_affixes = get_common_affixes()

    def prefixed_parts = prefixed_batches(base_batches, common_affixes, true)
    // def prefixed_file = prefixed_parts.collectFile(name: 'prefixed_variants.txt', newLine: false)

    def suffixed_parts = suffixed_batches(base_batches, common_affixes, false)
    // def suffixed_file = suffixed_parts.collectFile(name: 'suffixed_variants.txt', newLine: false)

    def all_parts = base_parts
        .mix(case_variants_parts)
        .mix(one_leet_variants_parts)
        .mix(prefixed_parts)
        .mix(suffixed_parts)

    def all_file = all_parts.collectFile(name: 'all_candidates_from_nf.txt', newLine: false)

    all_final = filter_by_length_and_make_unique(all_file)

    // Unlike mix(), concat() preserves source order: all items from file1, then all items from file2.
    def files =
        // base_file
        // .concat(case_variants_file)
        // .concat(one_leet_variants_file)
        // .concat(prefixed_file)
        // .concat(suffixed_file)
        // .concat(all_file)
        all_file
        .concat(all_final)
        .map { file -> tuple(file.baseName, file) }

    count_lines(files)
        .view { tpl -> "file name ${tpl[0]} with count: ${tpl[1].trim()}" }

    publish:
    all_candidates = all_file
    pure_candidates = all_final
    // basic = base_file
    // case_variants = case_variants_file
    // one_leet_variants = one_leet_variants_file
    // prefixed = prefixed_file
    // suffixed = suffixed_file
 }

This time it executed for several hours but completed successfully without any issues with batches of count 1000. This was more of a toy-experiment as the whole thing could have been coded in a single Python file, which made it as fast as complete execution in under 30 minutes and the python was single threaded .... So the genuine questions from this first experience: 1. Why Nextflow does not manage itself the load that it gets and requires me to engineer it in batches ? 2. In what kind of scenarios to think of nextflow to actually take advantage its capacity to run in parallel ? I understand that splitting the python in mini python scripts might not have been the best idea of all times 🙂

Does the v2 parser have a full knowledge of all valid (but undocumented) nextflow configs? e.g.

WARN: Unrecognized config option 'threadPool.PublishDir.maxThreads'

that used to be valid and I'm not sure if it's ignored now

I'm not sure if I'm just engaging in a cargo cult like mentality with a nextflow hang I've been experiencing haha

also, is there a list of all valid configs? I know many are no longer recommended but having hidden config options you need to track down from random git issues is suboptimal

github.com/nextflow-io/nextflow/blob/…/ThreadPoolManager.groovy#… suggests that this config might still be present?

Good morning! I am trying to use Phylophoenix and the files produced in the output folder do not seem to match the readme. I do see a newick file produced, but I do not know how the distances were computed, as the snp matrix i get in SNVPhyl_GRiPHin_Summary.xlsx is on a completely different scale. Can someone point me to documentation or tell me how the tree in the newick file was computed?

I'm developing a pipeline with several nf-core modules feeding in to several local modules wrapping custom R scripts using nextflow.enable.moduleBinaries = true. It seems like whenever I add a module with a new script or change an existing script and run with resume, it doesn't pick up the cache and runs the whole pipeline. Is there a way to prevent this from happening by modifying something at either the pipeline or module level? Any suggestions are appreciated!

I want to rewrite a decent nextflow pipeline into the nf-core framework, any advice/hints/short cuts? Seems like a good job for an LLM, which do folks recommend?

So a confession first, I spent two months building a tool and only now thought to ask whether anyone needs it in bioinformatics. Classic engineer right? Measure never, cut twice. Background. I come from regulated software industry, where traceability is not a hobby, it is a compliance requirement. Every change points to a requirement, every requirement points to a test, and when a bug is found you must say exactly which product type it reached. Auditors visit. They do not laugh istg! So I assumed bioinformatics was the same, and built github.com/QuietFlare/clew. It reads the Nextflow lineage store and answers for ex: these two samples were swapped at intake, what did they touch, and what do I do with each output, and can seal that answer as a report. Now the stupid question. Is that a real need? When a reference input or container turns out faulty after a run, does anyone in clinical or regulated settings have to produce an impact record an auditor would ask and accept? Or does everyone just type -resume, say a small prayer, and move on? I would honestly like to know which world I built this for.

Is there a better recommended way of highlighting issues on the Seqera web pages than the contact us form linked from the page footer? (This being docs.seqera.io but it probably holds for all pages.)

Hi all! Me and my team have been working on adapting a pipeline to the NF framework, and we are curious about some of the conditional flow best practices. We have a setup/pre-flight script that generates a run configuration file that very simply assigns a true/false value as to whether or not a particular step needs to run. For example, if the file is zipped, we would have a parameter like "run_unzip_process" set to true and force that process to run on the input. From the workflow scope, we thought maybe we could create some helper command to read the config, extract that param for a particular process, and then use a basic if else statement to run that process or forward the original input to the next process (pseudocode in the comments). We think this might become cumbersome and repetitive since we will need this for a lot of our processes. Has anyone done something similar for their own pipelines or have any insight into a more robust way to implement this?

Dear all, I am new to nextflow, but am enthusiatically writing my own modules for modular custom pipelines. When it comes to bulk and single-cell multiomics, tools are divided between R and Python, raising the question of input file formats. For example, nf-core modules exist for differential expression with limma, edgeR and DESeq2. These consume tsv-files and are therefore language agnostic. Then the decoupler-module does the same. But I use decoupler with single-cell datasets, which are large, sparse, and shouldnt be converted to tsv or csv files. Processing for multiomics may require the aforementioned differential expression in R, but processing before and after is preferred (by me) in muon, and intermediate processing steps may again be used from R. I let muon modules emit h5mu or zarr files, which then require conversion or extraction steps for several tsv/csv files (expression, metadata), or back from several files to anndata/mudata, which works for bulk data, but not large single-cell matrices. How may I handle this better, and in a way that will hold up modularity with other existing (limma, deseq2) and future nf-core modules, i.e. , muon, sva, RUVSeq, and more? (looking at the mlomix proposal using some of these) Modules and subworkflows for common conversions adn extraction like nf-core/anndata?

Can a process resource limits somehow be changed to increase by the number of inputs?

Are you supposed to push template branch updates to github?

Can we use python scripts in a subworkflow when submitting a subworkflow pr? From a quick glance, I dont see existing nf-core subworkflows that have them.

Who are the admins on this slack? There’s a suspiciously too good to be true job advert in #C02TLUQ1W2H which ticks a lot of the hallmarks of a scam (ai art, very high pay, rando gmail address, new poster)