I’m going in circles, so I wish to summon the rubber duck! In methylseq, Bismark generates the reference index from the FASTA file if one is not provided. When

params.fasta

is set via the config (like in

test.config

, pulled from test-datasets)

BISMARK_ALIGN

is able to find the

genome.fa

files, and it works as expected:

lrwxrwxrwx 1 gitpod gitpod 112 Oct 3 09:10 genome.fa -> /workspace/methylseq/work/stage-5bb8a185-99f1-4ef1-8483-b3b1b5e9d899/f8/feea95df595ec597193ef59e555b56/genome.fa

However, if

params.fasta

is set via the CLI (downloaded local copy),

BISMARK_ALIGN

complains that it can’t find the

genome.fa

file:

lrwxrwxrwx 1 gitpod gitpod  30 Oct 3 09:12 genome.fa -> /workspace/methylseq/genome.fa

I tested Bismark locally with both work directories, and Bismark doesn’t have any issues in that environment. Something about the Nextflow logic is causing Bismark to think the symlinked file is missing. Hey rubber duck, why is that?

curent dev?

yup

do you see the same file if you do a view on the fasta channel?

what do you mean by same file ? I have dump tags added that can be seen with

-dump-channels

I meant correct file, sorry

i did a diff of the file downloaded by nextflow for staging and the one i provide locally....no diff

ok

yup

ok so managed to reproduce

how is nextflow staging a file diff from a local copy ?

I assume the file is local, so just making a symlink to it is enough, so no staging

> I assume the file is local, so just making a symlink to it is enough, so no staging ya so what does staging do to the file that its seen by nextflow and not the symlink to local copy ?

I see the same amount of symlink

also not sure if this is a bismark issue....as if you see the `.command.log`s the paths are all correct

I tried

nextflow run . -profile test,docker --outdir results --fasta /home/max/workspace/methylseq/work/stage-cbce4bf6-570f-4892-a970-a0331e256283/ec/96d70fa57743bbf529718869c43adf/genome
.fa

with absolute path u mean ?

and absolute path from a fasta that was staged

trying

sorry if I managed to add confusion

no it makes sense

I don't see any weird stuff in the .command.run

328   │ nxf_stage() {
 329   │     true
 330   │     # stage input files
 331   │     rm -f BismarkIndex/genome.fa
 332   │     mkdir -p BismarkIndex && ln -s /home/max/workspace/methylseq/genome.fa BismarkIndex/genome.fa
 333   │ }

328   │ nxf_stage() {
 329   │     true
 330   │     # stage input files
 331   │     rm -f BismarkIndex/genome.fa
 332   │     mkdir -p BismarkIndex && ln -s /home/max/workspace/methylseq/work/stage-cbce4bf6-570f-4892-a970-a0331e256283/ec/96d70fa57743bbf529718869c43adf/genome.fa BismarkIndex/genome.fa
 333   │ }

any suggestions on how to resolve this @Maxime Garcia @Phil Ewels

I don't understand why the full path to a stage fasta is working, but the full path to a local fasta is not.

m thinking should I ask in nextflow slack probably Ben or Rob ?

Yeah, that sounds like a good idea

hi @Rob Syme @Ben Sherman sorry to have added you to this channel without notice but, welcome to the nf core rubber duck channel We were wondering if you could help shed some light on some puzzling nextflow behavior...its this thread but I will summarize: • When

params.fasta

is set via the config (as in

test.config

, pulled from test-datasets),

BISMARK_ALIGN

is able to locate the

genome.fa

file, and it works as expected:

lrwxrwxrwx 1 gitpod gitpod 112 Oct 3 09:10 genome.fa -> /workspace/methylseq/work/stage-5bb8a185-99f1-4ef1-8483-b3b1b5e9d899/f8/feea95df595ec597193ef59e555b56/genome.fa

• However, if

params.fasta

is set via the CLI (using a downloaded local copy),

BISMARK_ALIGN

throws an error, stating it can’t find the

genome.fa

file:

lrwxrwxrwx 1 gitpod gitpod 30 Oct 3 09:12 genome.fa -> /workspace/methylseq/genome.fa

• Setting process.stageInMode to copy works. • Using the staged version of the file also works. So, essentially, we would like to know:

is there something specific that Nextflow does during staging that allows it to work, as opposed to simply downloading and using a local copy of the file?

when the input file is remote, it is staged into the work directory as shown a above. if it's local then it should just be symlinked into the task directory so if it says it can't find the file but the symlink is there, could be a gitpod issue

I have the same issue locally (not on gitpod)

does the symlink work if you use it manually?

Maybe it's a container mounting issue?

Oh I see

This would be the case for any pipeline using any local reference files though, right? Why is this not breaking all over the place?

bismark is weird

Hmm. You're right. Nextflow is being more intelligent than I thought. Given a dummy workflow

params.input = "data/foo.txt"

workflow {
    Channel.fromPath(params.input) | Example
}

process Example {
    debug true
    input: path(input)
    script: "head $input"
}

Using the default

input

parameter, the

.command.run

docker run command mounts the grandparent directory of

foo.txt

-v /Users/robsyme/dev/scratch/20241004_symlink_local_files_in_container:/Users/robsyme/dev/scratch/20241004_symlink_local_files_in_container

If I change to

--input /Users/robsyme/tmpdata/data/foo.txt

, Nextflow mounts the great-grandparent directory of

foo.txt

-v /Users/robsyme:/Users/robsyme

Either way, Nextflow doesn't need any extra

containerOptions

to do the right thing. Sateesh or Maxime - are your local

genome.fa

references on the same filesytem? I know that symlinks don't work across heterogenous filesystems.

i downloaded my

genome.fa

to gitpod so, yes same fs

😕

to reproduce: open methylseq dev in gitpod remote-file:

nextflow run . -profile test,docker --outdir results

local-file:

wget <https://github.com/nf-core/test-datasets/raw/methylseq/reference/genome.fa>
nextflow run . -profile test,docker --outdir results --fasta genome.fa

yeah, just a normal thinkpad laptop with ubuntu mint

Testing it myself in Gitpod...

ooh, I see

The task workdir looks like:

BismarkIndex
├── Bisulfite_Genome
│   ├── CT_conversion
│   │   ├── BS_CT.1.bt2
│   │   ├── BS_CT.2.bt2
│   │   ├── BS_CT.3.bt2
│   │   ├── BS_CT.4.bt2
│   │   ├── BS_CT.rev.1.bt2
│   │   ├── BS_CT.rev.2.bt2
│   │   └── genome_mfa.CT_conversion.fa
│   └── GA_conversion
│       ├── BS_GA.1.bt2
│       ├── BS_GA.2.bt2
│       ├── BS_GA.3.bt2
│       ├── BS_GA.4.bt2
│       ├── BS_GA.rev.1.bt2
│       ├── BS_GA.rev.2.bt2
│       └── genome_mfa.GA_conversion.fa
└── genome.fa -> /workspace/methylseq/genome.fa

aaaah i thought abt this earlier but, brushed it aside facepalm

So I guess we need to patch the bismarkindex process so that it include only the index, and pass in the fasta to the bismarkalign process

Maybe we should avoid "carrying forward" task inputs like this.

agree...will try providing fasta separately

It is very convenient, though... I'll need to think about it a bit. thinker

I tend to discourage directory inputs/outputs because they have edge cases like this. although I recognize that it is more convenient when starting out I've been thinking we could represent these directories quite nicely as a record type:

record BismarkIndex {
  Path genome
  // ...
}

and you could have nested record types within that. then you pass the record type into a process and it can see from the record type all of the relevant files and handle them properly

maybe you guys can help me concretize this idea. how would you model this bismark directory with record types? here is my naive sketch based on the filenames:

record BismarkIndex {
  BisulfiteGenome genome
  Path fa
}

record BisulfiteGenome {
  ConversionThingy ct_conversion
  ConversionThingy ga_conversion
}

record ConversionThingy {
  List<Path> bt2
  Path fa
}

I have no idea what any of these parts mean, but I think I've captured the basic structure. Maybe one of you can come up with something more specific and meaningful

for a task workdir like:

BismarkIndex
├── Bisulfite_Genome
│   ├── CT_conversion
│   │   ├── BS_CT.1.bt2
│   │   ├── BS_CT.2.bt2
│   │   ├── BS_CT.3.bt2
│   │   ├── BS_CT.4.bt2
│   │   ├── BS_CT.rev.1.bt2
│   │   ├── BS_CT.rev.2.bt2
│   │   └── genome_mfa.CT_conversion.fa
│   └── GA_conversion
│       ├── BS_GA.1.bt2
│       ├── BS_GA.2.bt2
│       ├── BS_GA.3.bt2
│       ├── BS_GA.4.bt2
│       ├── BS_GA.rev.1.bt2
│       ├── BS_GA.rev.2.bt2
│       └── genome_mfa.GA_conversion.fa
└── genome.fa -> /workspace/methylseq/genome.fa

something like this

record BismarkIndex {
    Path genome
    BisulfiteGenome bisulfite_genome
}

record BisulfiteGenome {
    CTConversion ct_conversion
    GAConversion ga_conversion
}

record CTConversion {
    Path bt2_1
    Path bt2_2
    Path bt2_3
    Path bt2_4
    Path bt2_rev_1
    Path bt2_rev_2
    Path genome_mfa_ct_conversion_fa
}

record GAConversion {
    Path bt2_1
    Path bt2_2
    Path bt2_3
    Path bt2_4
    Path bt2_rev_1
    Path bt2_rev_2
    Path genome_mfa_ga_conversion_fa
}

One potential challenge is that not all task outputs will have a known number of files. Ben, do you anticipate we'd be able to handle collections?

record GAConversion {
    Path[] btfiles
    Path genome_mfa_ga_conversion_fa
}

indeed, that is why I did

List<Path> bt2

because I didn't know if there are always four files + 2 rev files or if it's variable. but you can do either one, enumerate them as individual fields or have a collection

ya ur right they have the same structure and its not variable so a single record will do

record BismarkIndex {
    Path genome
    BisulfiteGenome bisulfite_genome
}

record BisulfiteGenome {
    Conversion ct_conversion
    Conversion ga_conversion
}

record Conversion {
    List<Path> bt2
    Path genome_mfa_conversion_fa
}

Just FYI I've come across the exact same problem with a module for kraken2-build, it requires a directory input because it needs to take a directory output from an upstream command/module

kraken2-add

. I've not had a chance to test the stageAs: copy workaround, but reading the discussion it sounds almost like the exact behaviour as @Sateesh_Peri has. I'm not familiar with the record types yet so I can't say if that would help, but I do know the kraken2 database directory is a) quite complicated and very variable in the number of files it may have in there (and iirc with naming), so if this solution requires prior definition/specification of every object in the directory coming through a channel, this might be very difficult

I would love to see an example to test out my theory. With enums / records / collections you should be able to model just about any directory structure, but it might be overkill in some cases

But also a bit surprised this is coming up now, metagenomics tools use directories all over the place 😅

I only need a directory listing, but feel free to make it... maximal... to test the limits of the record syntax

$ tree

.
├── accmap_file.tmp
├── library
│   └── added
│       ├── dfArnMPK6W.fna
│       ├── dfArnMPK6W.fna.masked
│       ├── prelim_map_LPFBAbnMNN.txt
│       └── prelim_map.txt
├── seqid2taxid_acc.tmp
├── seqid2taxid.map.tmp
└── taxonomy
    ├── names.dmp -> /home/james/git/nf-core/createtaxdb/tutorial/names_reduced.dmp
    ├── nodes.dmp -> /home/james/git/nf-core/createtaxdb/tutorial/nodes_reduced.dmp
    ├── nucl_gb.accession2taxid -> /home/james/git/nf-core/createtaxdb/tutorial/nucl_gb.accession2taxid
    └── prelim_map.txt

This is a very simply example, but basically the

<randomhash.fna>

corresponds to different input genomes into building the database, the more genomes there are (normally hundreds of thousands), the more random hash files

record Root {
    Path accmap_file_tmp
    Library library
    Path seqid2taxid_acc_tmp
    Path seqid2taxid_map_tmp
    Taxonomy taxonomy
}

record Library {
    Added added
}

record Added {
    Path dfArnMPK6W_fna
    Path dfArnMPK6W_fna_masked
    Path prelim_map_LPFBAbnMNN_txt
    Path prelim_map_txt
}

record Taxonomy {
    Path names_dmp
    Path nodes_dmp
    Path nucl_gb_accession2taxid
    Path prelim_map_txt
}

correct Ben ?

But do I have define everything in

Added

?

i don't think u need to define

(and would this work already? Again: I basically know 0 how to use records or exactly what they are, just half listened to the podcast making them sound very useful... but this issue is big problem for me so if it's a solution I could implement next week that would be great 😬 )

I will let @Ben Sherman answer this 😬

So I'm going in this #C073GFZ68PR again as well

Why is this coming up now though?!

bcoz methylseq decided it was time 😛

The files that the docker/singulariy image couldn't access was the names.dmp, nodes.dmp, and nucl_gb.accession2taxid

I'm guessing it's because they are symlinks into directories that are not mounted into the container. And no I don't know why it only started breaking recently 🤷

I literally brought up a record type in my meeting with @Maxime Garcia earlier. I'm glad I'm on the right wavelength!

anyhow, I have an improvment on bismark/genomepreparation: https://github.com/nf-core/modules/pull/6807

I think you will have to declare each input file that is a symlink so that it is mounted correctly

So in BismarckIndex, I have two folders that contains multiple files...

I really want to understand why this is an issue now 😭. I wonder if there is an option in docker/singularity to 'follow symlinks' to follow those too or something

and I would like to know why cloud/fusion is having issues with the

mv

fasta workaround I have for this, like why is using

mv

a bad idea ?

So little Neo, I’m going to tell you a secret. There is no

mv

.

there is no spoon

On a fuse based system like Fusion or gcsfuse this can cause problems as stuff gets out of sync.

folks this is becoming a blocker for me to release methylseq. I would appreciate all the help to find a fix for this problem asap

@Sateesh_Peri - just re-read the thread. Record types seem good going forward, but I'd like to identify if there has been a breaking change in Nextflow's behaviour.

(FYI I just tried to see if I could do that with createtaxdb but because no release yet, don't have older working versions that e.g. still have checkmax etc - previous commits have different bugs😅 )

ya im trying to run an older commit of methylseq dev and getting

ERROR ~ Failed to load the meta schema

. lemme see if I can find another reproducible example

Make a 2 step pipeline without nf-schema

ok I couldn't think of a way to reproduce the setup of bismark index folder in time so, thought of modules which atleast reproduces the issue for me to test with diff NXF versions but, looks like i'm not able to go beyond 22.04.4 To reproduce: • Open @Maxime Garcia’s PR trying to fix bismark in gitpod •

wget <https://raw.githubusercontent.com/nf-core/test-datasets/methylseq/reference/genome.fa>

• first run the test with current NXF_VER=

24.04.4

nf-test test /workspace/nf-core_modules/modules/nf-core/bismark/genomepreparation/tests/main.nf.test --profile docker --verbose

• Then in the

main.nf.test

change the output to local file:

file('$baseDir/genome.fasta', checkIfExists: true)

• Run, the test and observe failure to find

genome.fa

N E X T F L O W  ~  version 24.04.4
  Launching `/workspace/nf-core_modules/.nf-test-6a5e80b0f824a0ed737bdfd9876957b1.nf` [festering_engelbart] DSL2 - revision: f33aedfc4e
  ERROR ~ No such file or directory: /workspace/nf-core_modules/genome.fasta

Next, change NXF_VER

export NXF_VER=23.04.4

same failure:

N E X T F L O W  ~  version 23.04.4
  Launching `/workspace/nf-core_modules/.nf-test-6a5e80b0f824a0ed737bdfd9876957b1.nf` [friendly_hoover] DSL2 - revision: f33aedfc4e
  ERROR ~ No such file or directory: /workspace/nf-core_modules/genome.fasta

22.04.0

N E X T F L O W  ~  version 22.04.4
  Launching `/workspace/nf-core_modules/.nf-test-6a5e80b0f824a0ed737bdfd9876957b1.nf` [desperate_bartik] DSL2 - revision: f33aedfc4e
  Nextflow stderr:

NXF_VER=20.04.1

Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.vmplugin.v7.Java7
        at org.codehaus.groovy.vmplugin.VMPluginFactory.<clinit>(VMPluginFactory.java:43)
        at org.codehaus.groovy.reflection.GroovyClassValueFactory.<clinit>(GroovyClassValueFactory.java:35)

😱

what's the rest of the error for 22.04?

nothing...thats was it

@Ben Sherman is that an acceptable solution for now: https://github.com/nf-core/modules/pull/6807

and also here: https://github.com/nf-core/modules/pull/6815

does it work?

yes

I'd say it's fairly similar to what we've been doing in FASTQC: https://github.com/nf-core/modules/blob/master/modules/nf-core/fastqc/main.nf#L37-L39