nf-core #rubber-duck

James Fellows Yates

06/20/2024, 8:08 AM

OK apparently I had to explicitly wrap the

split

in a list

Copy code

ch_dbs = databases
        .dump(tag: 'pre-map')
        .map{
            meta_db, db ->
            [ [meta_db.db_type.split(";")].flatten(), meta_db.subMap( meta_db.keySet() - 'db_type' ), db]
        }
        .dump(tag: "pre-transpose")
        .transpose(by: 0)
        .dump(tag: "post-transpose")
        .map{
            type, meta_db, db ->
            [meta_db + [db_type: type], db]
        }
        .dump(tag: 'post-map')

James Fellows Yates

06/20/2024, 8:08 AM

I don't know why that works though

Maxime Garcia

06/20/2024, 8:09 AM

I'd guess you needed to ensure it was a list so should access it properly otherwise it's a string, no?

James Fellows Yates

06/20/2024, 8:10 AM

Well th esplit dump (as you see above) is at listed printed a sa list

James Fellows Yates

06/20/2024, 8:10 AM

[DUMP: pre-transpose] [['short', 'long'],

Maxime Garcia

06/20/2024, 8:17 AM

did you try to flatten it directly?

Maxime Garcia

07/04/2024, 8:00 AM

<!here> Office hours is going to start again in ~ 1 hour. Come to our #C02R19TRGRE office and talk with @James Fellows Yates and me if you have any questions. We'll be happy to chat about code, pipelines, configs, life, the universe and everything

Simon Pearce

07/19/2024, 11:16 AM

Help me 🦆 s I am trying to understand why fastquorum doesn't seem to have igenomes working at the moment. When I try

Copy code

nextflow run main.nf \
--input <https://raw.githubusercontent.com/nf-core/test-datasets/fastquorum/testdata/samplesheets/samplesheet.tiny.csv> \
--genome GRCh38 \
--outdir test \
-profile docker

It won't align anything. It seems to have all the required boilerplate in https://github.com/nf-core/fastquorum/blob/da1148fed20864d5e041d3201560fde74705bcb3/main.nf#L33

Copy code

params.fasta     = getGenomeAttribute('fasta')
params.fasta_fai = getGenomeAttribute('fasta_fai')
params.dict      = getGenomeAttribute('dict')
params.bwa       = getGenomeAttribute('bwa')

// Initialize fasta file with meta map:
fasta = params.fasta ? Channel.fromPath(params.fasta).map{ it -> [ [id:it.baseName], it ] }.collect() : Channel.empty()

and when I try with some

println

action

getGenomeAttribute

seems to be working absolutely fine. But setting params.fasta just doesn't do anything:

Copy code

params.fasta = getGenomeAttribute('fasta')
println params.fasta

gives

null

, but

Copy code

params.fasta2 = getGenomeAttribute('fasta')
println params.fasta2

gives

<s3://ngi-igenomes/igenomes//Homo_sapiens/NCBI/GRCh38/Sequence/WholeGenomeFasta/genome.fa>

What is silently stopping

params.fasta

from being reassigned here? It seems to stop the other

fasta_fai

dict

and

bwa

parameters on the next lines too.

Sateesh_Peri

09/08/2024, 3:37 PM

I have been staring at this long but, can't seem to spot what is off I am trying to write this bismark subworkflow test and it seems to be complaining about the input meta not being a path type input specification of bismark align module:

Copy code

input:
    tuple val(meta), path(reads)

test input:

Copy code

input[0] = Channel.of([
                            [ id:'test', single_end:true ],
                            file(params.pipelines_testdata_base_path + 'testdata/SRR389222_sub1.fastq.gz', checkIfExists: true)
                ])

error:

Copy code

ERROR ~ Error executing process > 'BISMARK:BISMARK_ALIGN (1)'
  
  Caused by:
    Not a valid path value type: java.util.LinkedHashMap ([id:test, single_end:true])

i can't see why its trying to cast the meta to a path instead of val ? ...would appreciate some help from the rubber duck

Simon Pearce

09/16/2024, 8:45 AM

I'm sure that I'm doing something dumb here but I can't figure out what the issue is. I have a channel that I have constructed with

groupTuple

, that looks like:

[meta, [[file1a, file1b], [file2a, file2b], ... ]]

The second element here is an

ArrayBag

, in case that matters. I'm trying to return:

[meta, [file1a, file1b, file2a, file2b, ... ]]

I thought that

.map{meta, counts -> [meta, counts.collect()]}

would do that, but it doesn't seem to do anything

Sateesh_Peri

10/03/2024, 1:57 PM

I’m going in circles, so I wish to summon the rubber duck! In methylseq, Bismark generates the reference index from the FASTA file if one is not provided. When

params.fasta

is set via the config (like in

test.config

, pulled from test-datasets)

BISMARK_ALIGN

is able to find the

genome.fa

files, and it works as expected:

Copy code

lrwxrwxrwx 1 gitpod gitpod 112 Oct 3 09:10 genome.fa -> /workspace/methylseq/work/stage-5bb8a185-99f1-4ef1-8483-b3b1b5e9d899/f8/feea95df595ec597193ef59e555b56/genome.fa

However, if

params.fasta

is set via the CLI (downloaded local copy),

BISMARK_ALIGN

complains that it can’t find the

genome.fa

file:

Copy code

lrwxrwxrwx 1 gitpod gitpod  30 Oct 3 09:12 genome.fa -> /workspace/methylseq/genome.fa

I tested Bismark locally with both work directories, and Bismark doesn’t have any issues in that environment. Something about the Nextflow logic is causing Bismark to think the symlinked file is missing. Hey rubber duck, why is that?

chase mateusiak

10/14/2024, 7:42 PM

Is there anyone here, maybe from the plant world, who frequently does bulk segregant analysis? I have been for the last year or so, and I am wondering if it is worth proposing a nf-core workflow. There isn't a great end analysis tool -- that has been part of the project I've been on -- and there isn't much consensus in the literature that I'm familiar with around how to generate the counts, especially in haploid or ploidy > 2 organisms. If anyone does BSA, or has done BSA, I'd be curious to know your thoughts about the value of a workflow.

Slackbot

10/22/2024, 6:01 AM

This message was deleted.

Sateesh_Peri

10/25/2024, 12:09 PM

@Maxime Garcia this conditional linking is still giving me some errors when using an S3 workDir sad cowboy https://github.com/nf-core/methylseq/blob/aac172df2c1c6558683885d3d52157bed3b6bbab/modules/nf-core/bwameth/align/main.nf#L32-L34

Copy code

touch -c BwamethIndex/*
  
  if [ ! -f BwamethIndex/genome.fa ]; then
      ln -s $(readlink genome.fa) BwamethIndex/genome.fa;
  fi
  
  bwameth.py \
       \
       \
      -t 4 \
      --reference BwamethIndex/genome.fa \
      SRR389222_sub2_trimmed.fq.gz \
      | samtools view  -@ 4 -bhS -o SRR389222_sub2.bam -

error:

Copy code

Command error:
  ln: genome.fa: File exists

shakefist 1

Kobe Lavaerts

10/31/2024, 8:48 PM

1000007957.jpg

❤️ 3

nf core duck 16

big red rubber duck 1

Margherita Mutarelli

11/01/2024, 3:12 PM

Seasonal ducks in ALE-HOP

🤣 4

Sateesh_Peri

11/19/2024, 4:11 PM

hi Duckies.....talking out loud so it will help me process: I am having issue with #CP3RJSMF0

v2.7.1

where the bismark align step is not getting cached and is being re-run upon resume. I ran the

test,debug

profile, and noticed that the hashes for the bismark Index folder input seems to vary, which I believe is invalidating the cache and causing a re-run. Q is - why is the bismark index folder getting a diff hash each time even with the path to the stage folder being the same ?

run-1

hash

Copy code

e0e6778c9d458895209ee2555180d687 [nextflow.util.ArrayBag] [FileHolder(sourceObj:/workspace/methylseq/work/48/07893206ffcec48f5d0061c928db23/BismarkIndex, storePath:/workspace/methylseq/work/48/07893206ffcec48f5d0061c928db23/BismarkIndex, stageName:BismarkIndex)]

resume-1

hash

Copy code

2d2d4fcf2bffcae17ace858866c308ee [nextflow.util.ArrayBag] [FileHolder(sourceObj:/workspace/methylseq/work/48/07893206ffcec48f5d0061c928db23/BismarkIndex, storePath:/workspace/methylseq/work/48/07893206ffcec48f5d0061c928db23/BismarkIndex, stageName:BismarkIndex)]

resume-2

hash

Copy code

b6b7a8001ea430230fd4d49bddab8521 [nextflow.util.ArrayBag] [FileHolder(sourceObj:/workspace/methylseq/work/48/07893206ffcec48f5d0061c928db23/BismarkIndex, storePath:/workspace/methylseq/work/48/07893206ffcec48f5d0061c928db23/BismarkIndex, stageName:BismarkIndex)]

anybody have ideas on where should I be looking to pinpoint ? one change we introduced in

2.7.1

was this force soft-linking of fasta file into the index folder to avoid the issues with nextflow nested-symlinks

Copy code

ln -sf \$(readlink $fasta) $index/$fasta

but, still that doesn't explain why the

BismarkIndex

is getting a diff hash each time 🤷

Edmund

11/22/2024, 3:53 PM

New thread https://nfcore.slack.com/archives/C04PPJLT1T3/p1732257847172429?thread_ts=1732032677.595229&channel=C04PPJLT1T3&message_ts=1732257847.172429

James Fellows Yates

11/28/2024, 11:25 AM

Am I being an idiot?

James Fellows Yates

11/28/2024, 11:26 AM

image.png

James Fellows Yates

11/28/2024, 11:26 AM

Can I no longer count?

Matthias De Smet

12/10/2024, 4:46 AM

Are there any `groupTuple`/`groupKey` magicians around? I’ve got a channel of

[meta, bam]

of which I want to gather and join the elements by

meta.samplename

. The catch is the channel elements can have a different amount of chuncks per

meta.samplename

. I’m looking for a non blocking way to count all the elements in channel per

meta.samplename

so I can make a

groupKey

Current code is in https://github.com/nf-cmgg/preprocessing/blob/dev/subworkflows/local/fastq_to_aligned_cram/main.nf#L68-L98. Here I first try to join on the amount of scattered

meta.chunks

and then on the amount of replicates in

meta.count

Simon Pearce

05/26/2025, 8:35 AM

I'm trying to migrate the

fasta_index_dna

subworkflow to nf-test. This is proving trickier that I'd expect. @Matthias De Smet, you originally implemented this module by the looks of it. This subworkflow takes the fasta, but also an extra altliftover channel:

Copy code

ch_fasta        // channel: [mandatory] [ val(meta), path(fasta) ]
        ch_altliftover  // channel: [mandatory, if aligner is bwamem or bwamem2 or snap] [ val(meta), path(altliftover) ]

The issue is in this section of the code:

Copy code

BWAMEM1_INDEX.out.index
                .join(ch_altliftover)
                .map { meta, index, alt -> [meta, index + alt] }

which I believe is trying to add the altliftover file inside the folder that is output by bwamem. But I don't think we can use

to do this, because it is a file not a meta map, and what we get from nf-test begins (condensing and truncating for space):

Copy code

"0": [[{"id": "test"},
                        ["/workspace/modules/.nf-test/tests/fcebacc24247d7e952390921d1be834a/workspace",
                         "/workspace/modules/.nf-test/tests/fcebacc24247d7e952390921d1be834a/modules",
                         "/workspace/modules/.nf-test/tests/fcebacc24247d7e952390921d1be834a/.nf-test",
                         "/workspace/modules/.nf-test/tests/fcebacc24247d7e952390921d1be834a/tests",
                         "/workspace/modules/.nf-test/tests/fcebacc24247d7e952390921d1be834a/fcebacc24247d7e952390921d1be834a",
                            [[[
                               ".command.begin:md5,d41d8cd98f00b204e9800998ecf8427e",

i.e. it returns a list of all the files and folders with full paths. If I actually give a file to ch_altliftover, it still gives full paths but only the new file not the original bwa folder at all. I don't know how to pass this file inside the folder, without adding an extra process.

James Fellows Yates

04/23/2026, 1:05 PM

Why would the args here:

Copy code

withName: KMCP_COMPUTE {
        ext.prefix = { "${meta.id}-kmcp-compute" }
        ext.args   = {
            [
                params.kmcp_compute_byfile ? "" : "--by-seq",
                params.kmcp_compute_options,
            ].join(' ').trim()
        }
        publishDir = [
            enabled: false
        ]
    }

in a

modules.conf

Not reach this module: https://github.com/nf-core/modules/blob/master/modules/nf-core/kmcp/compute/main.nf HELP

Maxime Garcia

05/04/2026, 9:04 AM

I'm going deep #C073GFZ68PR

Maxime Garcia

05/04/2026, 9:11 AM

So it seems my issue is actually @Mahesh Binzer-Panchal 🤯

Maxime Garcia

05/04/2026, 9:12 AM

somehow, this entry in my contributors list make the lint fails for the list on the last one if it's not the last one

Copy code

[
            name: 'Mahesh Binzer-Panchal',
            affiliation: 'National Bioinformatics Infrastructure Sweden',
            github: 'mahesh-panchal',
            contribution: ['contributor'],
            orcid: '0000-0003-1675-0677',
        ],

Louis Le Nézet

06/10/2026, 1:46 PM

I'm running the same test (

test_all

), from the same git repository

nf-core/phaseimpute

, from the same commit

fa9fac8cd

. I'm running it on 4 different machine (PC windows, PC linux, github action, HPC). The result from a

SHAPEIT5_phase_common

process differ ever so slightly but throw off all my nf-test assertion... This test use the same file as input (same variants md5sum), use

--thread 1

and

--seed 1

. This wasn't happening before and I can't figure out what changed since then. The two PC aggree on the same variants hash

a27ecf59af876aed33316ca37189dd9f

while GA and HPC found

797b0fd345ce761987d4b3bddac4236a

. These md5 are stable accross multiples runs for each devices. This happen only for chromosome 21, while chr22 is stable in the 4 conditions.

Igor Trujnara

07/14/2026, 2:14 PM

I have a module that uses a Python script to fetch data from an API via a wrapper (

omadb

). I have a test case with 2 reasonably long test sequences (~400bp). When I run the script inside the Nextflow module, it tends to fail with

400 Bad request ("sequence too short")

. When I run substantially identical code in a Python REPL inside a venv, everything works fine and I get a response. Versions are not identical (Python 3.11 vs 3.14), but nothing changed in these updates should change the outcome. I have added a debug print to the script to check if the sequence is read in correctly, and it is.

Louis Le Nézet

09/29/2026, 10:31 AM

I have the

VCF_PHASE_SHAPEIT5

. It takes as input: • ch_vcf (meta[id, optional(chr)], vcf, index): mandatory target variants to be phased (can be multi/single sample, multi/single chr) • ch_chunks (meta[panel_id, chr], regionout): optional list of successive regions in a chr to perform the phasing on (multiple / chr), obtained from chunking the panel so linked to panel_id • ch_ref (meta[panel_id, chr], vcf, index): optional chromosome level VCF of a reference population haplotypes (one / chr) • ch_scaffold (meta[panel_id, chr], vcf, index): optional chromosome level VCF of some haplotypes that overlap ch_vcf position and samples (one / chr) • ch_scaffold (meta[chr], region, map): optional region and geneticmap to perform chunking and phasing (one / chr) Here are the use case: • work with only input vcfs (multi-sample, multi/single chr), + optionally chunks (multi / chr) and map (one / chr) • work with input vcf (multi/single sample + multi/single chr) + panel/scaffold (one / chr) + optionally chunks (multi / chr) and map (one / chr) I need to properly combine the different channel based on the data provided. Currently the sbwf do a join on all channel and let the user in charge of doing the whole meta modification before using it. This is no ideal and add a lot of meta map manipulation for the second use case. My idea was either to: • use a boolean to ask if ch_vcf needs to be joined or simply combined (this works for most cases but not if ch_vcf is split by chr) • use an array of common meta to perform the join on (should work but need a way when ch_vcf is not split by chr)