https://nf-co.re logo
Join Slack
Powered by
Dear all, I have a nf-test that fails with a "file...
# nf-test
a
Dear all, I have a nf-test that fails with a "file not found" error. The file exists, however, it seems that nf-test cannot stage it properly. I use nf-test 
0.9.2
, and nextflow 
24.04.2
. Further details in the thread.
My test:
Copy code
# tests/mini.nf.test
nextflow_pipeline {

    name "Test main.nf workflow"
    script "main.nf"

    test("Should run the main workflow tests against a minimal set of input data.") {

        config "${projectDir}/tests/configs/mini/nextflow.config"
        when {
            params {
                outdir = "${projectDir}/results/mini"
            }
        }
        then {
            assert workflow.success
        }
    }
}
The config file that is used in the above test:
Copy code
params {
    input     = "${projectDir}/tests/inputs/mini/samplesheet.csv"
    input_dir = "${projectDir}/tests/inputs/mini"
    ...
}
The file 
samplesheet.csv
that is referenced in the above config file:
Copy code
samplename,description,batch,phenotype,sex,variant_regions,coverage
Biosample-15lines,Biosample-15lines,,,,./tests/configs/chr21_enrichment_kit_10.bed,./tests/configs/par_regions_hg38_giab_v3.bed
For portability, the paths in the above samplesheet are relative. Both files exist:
Copy code
$ ls ./tests/configs/chr21_enrichment_kit_10.bed
./tests/configs/chr21_enrichment_kit_10.bed
$ ls ./tests/configs/par_regions_hg38_giab_v3.bed
./tests/configs/par_regions_hg38_giab_v3.bed
When I execute the test, 
chr21_enrichment_kit_10.bed
is not found by the BEDTOOLS_SPLIT process:
Copy code
nf-test test tests/mini.nf.test --profile test,docker --clean-snapshot   --update-snapshot
...
  Assertion failed:

  assert workflow.success
         |        |
         workflow false

  Nextflow stdout:

  ERROR ~ Error executing process > 'NFCORE_NFONTVC:NFONTVC:BED_SCATTER_BEDTOOLS:BEDTOOLS_SPLIT (Biosample-15lines)'

  Caused by:
    Process `NFCORE_NFONTVC:NFONTVC:BED_SCATTER_BEDTOOLS:BEDTOOLS_SPLIT (Biosample-15lines)` terminated with an error exit status (1)


  Command executed:

    bedtools \
        split \
         \
        -n 4 \
        -i chr21_enrichment_kit_10.bed \
        -p Biosample-15lines

    cat <<-END_VERSIONS > versions.yml
    "NFCORE_NFONTVC:NFONTVC:BED_SCATTER_BEDTOOLS:BEDTOOLS_SPLIT":
        bedtools: $(bedtools --version | sed -e "s/bedtools v//g")
    END_VERSIONS

  Command exit status:
    1

  Command output:
    (empty)

  Command error:
    Error: The requested file (chr21_enrichment_kit_10.bed) could not be opened. Error message: (No such file or directory). Exiting!

  Work dir:
    /home/user/src/nf-ont-vc/.nf-test/tests/eb83213b17e329329c7a66fe535eecef/work/01/b3cea0c24d11d1525971bde80a7d40
The workdir contains a broken symlink, hence the "file not found error":
Copy code
$ ls -l /home/user/src/nf-ont-vc/.nf-test/tests/eb83213b17e329329c7a66fe535eecef/work/01/b3cea0c24d11d1525971bde80a7d40/chr21_enrichment_kit_10.bed
home/user/src/nf-ont-vc/.nf-test/tests/eb83213b17e329329c7a66fe535eecef/work/01/b3cea0c24d11d1525971bde80a7d40/chr21_enrichment_kit_10.bed -> /home/user/src/nf-ont-vc/.nf-test/tests/eb83213b17e329329c7a66fe535eecef/tests/configs/chr21_enrichment_kit_10.bed
m
relative pass are messy with nf-test
a
I modified the BEDTOOLS_SPLIT process by prepending 3 lines that print out the process' input and context:
Copy code
"""
    echo "PWD: \$(pwd)"
    echo "BED: ${bed}"
    ls -l ${bed}

    bedtools \\
        split \\
        $args \\
        -n ${count} \\
        -i ${bed} \\
        -p ${prefix}
    ...
    """
Now, by running nf-test as before, I can see what bedtools is trying to use:
Copy code
...
  Command error:
    PWD: /home/user/src/nf-ont-vc/.nf-test/tests/eb83213b17e329329c7a66fe535eecef/work/9a/8ed5d8421da61688923b485139b38a
    BED: chr21_enrichment_kit_10.bed
    lrwxrwxrwx    1 1000     1000           115 Nov 18 08:55 chr21_enrichment_kit_10.bed -> /home/user/src/nf-ont-vc/.nf-test/tests/eb83213b17e329329c7a66fe535eecef/tests/configs/chr21_enrichment_kit_10.bed
    Error: The requested file (chr21_enrichment_kit_10.bed) could not be opened. Error message: (No such file or directory). Exiting!
...
If I 
view
the the input channel of BEDTOOLS_SPLIT, I see that the provided input is a path to the existing file: 
BED file: [[id:Biosample-15lines], /home/user/src/nf-ont-vc/tests/configs/chr21_enrichment_kit_10.bed]
How can I properly stage 
./tests/configs/chr21_enrichment_kit_10.bed
?
@Maxime Garcia, yes, but I would use absolute paths as a last option
a
I'm with Maxime here that relative paths in samplesheets are a generally bad idea. To solve this issue I have a helper script which I run before my tests to setup my samplesheets and any stub files I use so that they can be easily run anywhere.
a
@Maxime Garcia, @Arthur, thanks a lot 🙂 I will therefore align the testing directory structure with my colleagues, or have a pre-testing script to do that as suggested
TIL I can use 
export NXF_FILE_ROOT=$PWD
to resolve relative paths during nf-test. When I run the above line and launch my above nf-test, the test succeeds.
🙌 1
a
Huh. TIL.
m
Nice
a
My only concern is, does this effectively break any use of files in the 
projectDir
(since its often done to access e.g. 
.nextflow_schema.json
or 
assets/input_schema.json
) if you set it to some random location?
a
@Arthur, sorry, I don't follow; what is often done to access e.g. 
assets/input_schema.json
? If I 
export NXF_FILE_ROOT=/data
(where 
/data
an existing path on my machine), and re-run the test, then I get the error:
Copy code
...
Command error:
    PWD: /home/user/src/nf-ont-vc/.nf-test/tests/eb83213b17e329329c7a66fe535eecef/work/0d/eac534b674f837a4e80d81e83ade17
    BED: chr21_enrichment_kit_10.bed
    lrwxrwxrwx    1 1000     1000            47 Nov 18 16:03 chr21_enrichment_kit_10.bed -> /data/tests/configs/chr21_enrichment_kit_10.bed
    Error: The requested file (chr21_enrichment_kit_10.bed) could not be opened. Error message: (No such file or directory). Exiting!
...
If I also copy the file to 
/data/tests/configs
and re-run the test, then the pipeline fails trying to access an illegal storage location (to retrieve the file 
par_regions_hg38_giab_v3.bed
, which is supposed to exist right next to 
chr21_enrichment_kit_10.bed
):
Copy code
$ mkdir -p /data/tests/configs
$ cp tests/configs/chr21_enrichment_kit_10.bed /data/tests/configs/
$ ls /data/tests/configs/
chr21_enrichment_kit_10.bed

$ nf-test test tests/mini.nf.test --profile test,docker --clean-snapshot --update-snapshot
...
  Command error:
    [E::hts_open_format] Failed to open file "par_regions_hg38_giab_v3.bed" : No such file or directory
    SIGSEGV: Illegal storage access. (Attempt to read from nil?)
    .command.sh: line 8:    32 Segmentation fault      (core dumped) mosdepth --threads 4 --by par_regions_hg38_giab_v3.bed -F 1804 -Q 1 Biosample-chr21 Biosample-chr21-ready.bam
...
If I also copy the missing 
par_regions_hg38_giab_v3.bed
file, then the test works again:
Copy code
$ cp tests/configs/par_regions_hg38_giab_v3.bed /data/tests/configs/
$ ls /data/tests/configs/
chr21_enrichment_kit_10.bed  par_regions_hg38_giab_v3.bed

$ echo $NXF_FILE_ROOT
/data

$ nf-test test tests/02.mini.main.nf.test --profile test,docker --clean-snapshot   --update-snapshot
...
  Test [eb83213b] 'Should run the main workflow tests against a minimal set of input data.' PASSED (220.477s)
...
SUCCESS: Executed 1 tests in 220.514s
a
nf-core
pipelines have used the fact that relative paths are resolved against the 
projectDir
by default when using those paths is what I mean, and so I'm not sure if modifying this ENV var might break that. Not necessarily directly related to your stuff
👍 1
Open in Slack
PreviousNext
Powered by