Hi. I'm trying to run the tests for the glimpse2/p...
# modules
k
Hi. I'm trying to run the tests for the glimpse2/phase module and getting the following error:
Copy code
Test Process GLIMPSE2_PHASE

  Test [c6e28d6d] 'Should run with vcf' 
    > Nextflow 24.10.5 is available - Please consider updating your version to it
    > N E X T F L O W  ~  version 24.10.3
    > Launching `/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test-c6e28d6de8abe0a1a87b7a131dfa913c.nf` [furious_engelbart] DSL2 - revision: 6f81f626f7
    > ERROR ~ Error executing process > 'GLIMPSE2_PHASE (input)'
    > 
    > Caused by:
    >   Cannot invoke method replace() on null object -- Check script '/ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf' at line: 35
It seems to be referencing this line:
Copy code
def prefix = task.ext.prefix ?: "${meta.id}_${region}"
Seems like fairly innocuous boilerplate, and I can run any number of other tests that include similar references to
task.ext.prefix
with no issues. It's only this module's tests that are failing in this way. If I change the line to the following, the error goes away:
Copy code
def prefix = "test"
a
Can you please confirm: • the exact command you are running to run this test • the exact contents of your
main.nf
for the module under test
k
Sure. Here is the command:
Copy code
nf-test test --profile docker modules/nf-core/glimpse2/phase --verbose
And the code is just as I pulled it from the nf-core glimpse2/phase module. https://github.com/nf-core/modules/blob/master/modules/nf-core/glimpse2/phase/main.nf
a
I cannot reproduce this issue with the current master code locally
k
Interesting. It's failing on our GitHub CI action too. I can only imagine it must have something to do with our nf-test.config file then?
a
What is in the
nf-test.config
?
k
Copy code
config {
    // location for all nf-tests
    testsDir "."

    // nf-test directory including temporary files for each test
    workDir System.getenv("NFT_WORKDIR") ?: ".nf-test"

    // location of an optional nextflow.config file specific for executing tests
    configFile "tests/conf/nf-test.config"

    // run all test with defined profile(s) from the main nextflow.config
    profile ""
    triggers 'nextflow.config', 'nf-test.config', 'conf/test.config', 'conf/test_full.config'

    // Include plugins
    plugins {
        load "nft-bam@0.4.0"
        load "nft-utils@0.0.3"
    }
}
This is the referenced
tests/conf/nf-test.config
file:
Copy code
/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Nextflow config file for running minimal tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Defines input files and everything required to run a fast and simple pipeline test.

    Use as follows:
        nextflow run astreaforensics/wopr -profile test,<docker/singularity> --outdir <OUTDIR>

----------------------------------------------------------------------------------------
*/

process {
    resourceLimits = [
        cpus: 4,
        memory: '15.GB',
        time: '1.h'
    ]
}

params {
    config_profile_name               = 'nf-test profile'
    config_profile_description        = 'Minimal test dataset to check pipeline function'
    publish_dir_mode                  = "copy"
    singularity_pull_docker_container = false
    saveunmapped = true

    // Base directory for nf-core/modules test data
    modules_testdata_base_path = 'https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/'

    // Input data
    input = "${projectDir}/tests/csv/fastq_pair.csv"

    sequencing_qc_tool = 'falco'
}

profiles {
    conda {
        conda.enabled           = true
        conda.channels          = ['conda-forge', 'bioconda']
        apptainer.enabled       = false
    }
    docker {
        docker.enabled          = true
        docker.runOptions       = '-u $(id -u):$(id -g) --platform=linux/amd64'
    }
    singularity {
        singularity.enabled     = true
        singularity.autoMounts  = true
    }
    apptainer {
        apptainer.enabled       = true
        apptainer.autoMounts    = true
    }
}

// Load test_data.config containing paths to test data
includeConfig 'test_data.config'
a
None of that really tracks as the problem to me. For starters, line 35 in current
master
doesn’t contain
.replace
, but line 33 does. Can you sanity check by dumping
/ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
?
k
Sure. I will say though that after a lot of debugging, I believe the replace() on line 33 is an (insane) red herring. It points to line 35, and that does indeed seem to be the issue, as when I remove the reference to task.ext.prefix from that line, the error disappears.
Here it is:
Copy code
process GLIMPSE2_PHASE {
    tag "$meta.id"
    label 'process_medium'

    beforeScript  """
    if cat /proc/cpuinfo | grep avx2 -q
    then
        echo "Feature AVX2 present on host"
    else
        echo "Feature AVX2 not present on host"
        exit 1
    fi
    """

    conda "${moduleDir}/environment.yml"
    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        'https://depot.galaxyproject.org/singularity/glimpse-bio:2.0.1--h46b9e50_1':
        'biocontainers/glimpse-bio:2.0.1--h46b9e50_1' }"

    input:
        tuple val(meta) , path(input, arity: '1..*'), path(input_index), path(bamlist), path(samples_file), val(input_region), val(output_region), path(reference), path(reference_index), path(map)
        tuple val(meta2), path(fasta_reference), path(fasta_reference_index)

    output:
        tuple val(meta), path("*.{vcf,vcf.gz,bcf,bgen}"), emit: phased_variants
        tuple val(meta), path("*.txt.gz")               , emit: stats_coverage, optional: true
        path "versions.yml"                             , emit: versions

    when:
        task.ext.when == null || task.ext.when

    script:
    def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    def args   = task.ext.args   ?: ""
    def prefix = task.ext.prefix ?: "${meta.id}_${region}"
    def suffix = task.ext.suffix ?: "bcf"

    def map_command           = map                 ? "--map $map"                    : ""
    def samples_file_command  = samples_file        ? "--samples-file $samples_file"  : ""
    def fasta_command         = fasta_reference     ? "--fasta $fasta_reference"      : ""
    def input_region_cmd      = input_region        ? "--input-region $input_region"  : ""
    def output_region_cmd     = output_region       ? "--output-region $output_region": ""

    def input_type            = input.collect{
        it.toString().endsWithAny("cram", "bam") ? "bam" :
        it.toString().endsWithAny("vcf", "bcf", "vcf.gz") ? "gl" :
        it.getExtension()
    }.unique()

    if (input_type.size() > 1 | !(input_type.contains("gl") | input_type.contains("bam"))) {
        error "Input files must be of the same type and either .bam/.cram or .vcf/.vcf.gz/.bcf format. Found: ${input_type}"
    } else {
        input_type = input_type[0]
    }
    if (input_type == "gl" & input.size() > 1) {
        error "Only one input .vcf/.vcf.gz/.bcf file can be provided"
    }
    def input_list = input.size() > 1

    """
    if [ -n "$bamlist" ] ;
    then
        input_command="--bam-list $bamlist"
    elif $input_list ;
    then
        ls -1 | grep '\\.cram\$\\|\\.bam\$' > all_bam.txt
        input_command="--bam-list all_bam.txt"
    else
        if [ "$input_type" == "bam" ];
        then
            input_command="--bam-file $input"
        elif [ "$input_type" == "gl" ];
        then
            input_command="--input-gl $input"
        else
            echo "Input file type not recognised"
            echo "$input_type"
            exit 1
        fi
    fi

    GLIMPSE2_phase \\
        $args \\
        \$input_command \\
        --reference $reference \\
        $map_command \\
        $fasta_command \\
        $samples_file_command \\
        $input_region_cmd \\
        $output_region_cmd \\
        --thread $task.cpus \\
        --output ${prefix}.${suffix}

    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        glimpse2: "\$(GLIMPSE2_phase --help | sed -nr '/Version/p' | grep -o -E '([0-9]+.){1,2}[0-9]' | head -1)"
    END_VERSIONS
    """

    stub:
    def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    def args   = task.ext.args   ?: ""
    def prefix = task.ext.prefix ?: "${meta.id}_${region}"
    def suffix = task.ext.suffix ?: "bcf"
    """
    touch ${prefix}.${suffix}

    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        glimpse2: "\$(GLIMPSE2_phase --help | sed -nr '/Version/p' | grep -o -E '([0-9]+.){1,2}[0-9]' | head -1)"
    END_VERSIONS
    """
}
a
Ok, can you isolate the test being run using
Copy code
nf-test test modules/nf-core/glimpse2/phase --verbose --profile docker --tag "Should run with vcf"
k
I spent multiple hours looking at line 33 because I figured it must have had to be that due to the replace() 🫠
Sure. The others fail too due to the same issue. Let me try running just that one though.
Here's the full output:
Copy code
nf-test test modules/nf-core/glimpse2/phase --verbose --profile docker --tag "Should run with vcf"

🚀 nf-test 0.9.2
https://www.nf-test.com
(c) 2021 - 2024 Lukas Forer and Sebastian Schoenherr

Load .nf-test/plugins/nft-bam/0.4.0/nft-bam-0.4.0.jar
Load .nf-test/plugins/nft-utils/0.0.3/nft-utils-0.0.3.jar

Test Process GLIMPSE2_PHASE

  Test [c6e28d6d] 'Should run with vcf' 
    > Nextflow 24.10.5 is available - Please consider updating your version to it
    > N E X T F L O W  ~  version 24.10.3
    > Launching `/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test-c6e28d6de8abe0a1a87b7a131dfa913c.nf` [dreamy_sax] DSL2 - revision: 6f81f626f7
    > ERROR ~ Error executing process > 'GLIMPSE2_PHASE (input)'
    > 
    > Caused by:
    >   Cannot invoke method replace() on null object -- Check script '/ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf' at line: 35
    > 
    > 
    > Source block:
    >   def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    >   def args   = task.ext.args   ?: ""
    >   def prefix = task.ext.prefix ?: "${meta.id}_${region}"
    >   def suffix = task.ext.suffix ?: "bcf"
    >   def map_command           = map                 ? "--map $map"                    : ""
    >   def samples_file_command  = samples_file        ? "--samples-file $samples_file"  : ""
    >   def fasta_command         = fasta_reference     ? "--fasta $fasta_reference"      : ""
    >   def input_region_cmd      = input_region        ? "--input-region $input_region"  : ""
    >   def output_region_cmd     = output_region       ? "--output-region $output_region": ""
    >   def input_type            = input.collect{
    >       it.toString().endsWithAny("cram", "bam") ? "bam" :
    >       it.toString().endsWithAny("vcf", "bcf", "vcf.gz") ? "gl" :
    >       it.getExtension()
    >   }.unique()
    >   if (input_type.size() > 1 | !(input_type.contains("gl") | input_type.contains("bam"))) {
    >       error "Input files must be of the same type and either .bam/.cram or .vcf/.vcf.gz/.bcf format. Found: ${input_type}"
    >   } else {
    >       input_type = input_type[0]
    >   }
    >   if (input_type == "gl" & input.size() > 1) {
    >       error "Only one input .vcf/.vcf.gz/.bcf file can be provided"
    >   }
    >   def input_list = input.size() > 1
    >   """
    >   if [ -n "$bamlist" ] ;
    >   then
    >       input_command="--bam-list $bamlist"
    >   elif $input_list ;
    >   then
    >       ls -1 | grep '\\.cram\$\\|\\.bam\$' > all_bam.txt
    >       input_command="--bam-list all_bam.txt"
    >   else
    >       if [ "$input_type" == "bam" ];
    >       then
    >           input_command="--bam-file $input"
    >       elif [ "$input_type" == "gl" ];
    >       then
    >           input_command="--input-gl $input"
    >       else
    >           echo "Input file type not recognised"
    >           echo "$input_type"
    >           exit 1
    >       fi
    >   fi
    >   
    >   GLIMPSE2_phase \\
    >       $args \\
    >       \$input_command \\
    >       --reference $reference \\
    >       $map_command \\
    >       $fasta_command \\
    >       $samples_file_command \\
    >       $input_region_cmd \\
    >       $output_region_cmd \\
    >       --thread $task.cpus \\
    >       --output ${prefix}.${suffix}
    >   
    >   cat <<-END_VERSIONS > versions.yml
    >   "${task.process}":
    >       glimpse2: "\$(GLIMPSE2_phase --help | sed -nr '/Version/p' | grep -o -E '([0-9]+.){1,2}[0-9]' | head -1)"
    >   END_VERSIONS
    >   """
    > 
    > Container:
    >   quay.io/biocontainers/glimpse-bio:2.0.1--h46b9e50_1
    > 
    > Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
    > 
    >  -- Check '/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test/tests/c6e28d6de8abe0a1a87b7a131dfa913c/meta/nextflow.log' file for details
    > Execution cancelled -- Finishing pending tasks before exit
    Assertion failed: 

assert process.success
       |       |
       |       false
       GLIMPSE2_PHASE

java.lang.NullPointerException: Cannot invoke method getAt() on null object
java.lang.RuntimeException: Different Snapshot:
[                                                                                                       [
    [                                                                                                       [
        "versions.yml:md5,c68de03046a6503cdbcf3a1495fc512f"                                        |            
    ]                                                                                                       ]
]                                                                                                       ]

FAILED (9.813s)

  Assertion failed: 
  
  3 of 3 assertions failed
  
  Nextflow stdout:
  
  N E X T F L O W  ~  version 24.10.3
  Launching `/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test-c6e28d6de8abe0a1a87b7a131dfa913c.nf` [dreamy_sax] DSL2 - revision: 6f81f626f7
  ERROR ~ Error executing process > 'GLIMPSE2_PHASE (input)'
  
  Caused by:
    Cannot invoke method replace() on null object -- Check script '/ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf' at line: 35
  
  
  Source block:
    def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    def args   = task.ext.args   ?: ""
    def prefix = task.ext.prefix ?: "${meta.id}_${region}"
    def suffix = task.ext.suffix ?: "bcf"
    def map_command           = map                 ? "--map $map"                    : ""
    def samples_file_command  = samples_file        ? "--samples-file $samples_file"  : ""
    def fasta_command         = fasta_reference     ? "--fasta $fasta_reference"      : ""
    def input_region_cmd      = input_region        ? "--input-region $input_region"  : ""
    def output_region_cmd     = output_region       ? "--output-region $output_region": ""
    def input_type            = input.collect{
        it.toString().endsWithAny("cram", "bam") ? "bam" :
        it.toString().endsWithAny("vcf", "bcf", "vcf.gz") ? "gl" :
        it.getExtension()
    }.unique()
    if (input_type.size() > 1 | !(input_type.contains("gl") | input_type.contains("bam"))) {
        error "Input files must be of the same type and either .bam/.cram or .vcf/.vcf.gz/.bcf format. Found: ${input_type}"
    } else {
        input_type = input_type[0]
    }
    if (input_type == "gl" & input.size() > 1) {
        error "Only one input .vcf/.vcf.gz/.bcf file can be provided"
    }
    def input_list = input.size() > 1
    """
    if [ -n "$bamlist" ] ;
    then
        input_command="--bam-list $bamlist"
    elif $input_list ;
    then
        ls -1 | grep '\\.cram\$\\|\\.bam\$' > all_bam.txt
        input_command="--bam-list all_bam.txt"
    else
        if [ "$input_type" == "bam" ];
        then
            input_command="--bam-file $input"
        elif [ "$input_type" == "gl" ];
        then
            input_command="--input-gl $input"
        else
            echo "Input file type not recognised"
            echo "$input_type"
            exit 1
        fi
    fi
    
    GLIMPSE2_phase \\
        $args \\
        \$input_command \\
        --reference $reference \\
        $map_command \\
        $fasta_command \\
        $samples_file_command \\
        $input_region_cmd \\
        $output_region_cmd \\
        --thread $task.cpus \\
        --output ${prefix}.${suffix}
    
    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        glimpse2: "\$(GLIMPSE2_phase --help | sed -nr '/Version/p' | grep -o -E '([0-9]+.){1,2}[0-9]' | head -1)"
    END_VERSIONS
    """
  
  Container:
    quay.io/biocontainers/glimpse-bio:2.0.1--h46b9e50_1
  
  Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
  
   -- Check '/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test/tests/c6e28d6de8abe0a1a87b7a131dfa913c/meta/nextflow.log' file for details
  Execution cancelled -- Finishing pending tasks before exit
  Nextflow stderr:
  
  Nextflow 24.10.5 is available - Please consider updating your version to it
  

  Snapshots:
    Obsolete snapshots can only be checked if all tests of a file are executed successful.


FAILURE: Executed 1 tests in 9.826s (1 failed)
a
Can you humour me for a moment and add these lines to your module main.nf script section (at the start) and run again?
Copy code
println output_region
println input_region
k
Sure. I tried that too. Having println statements there does not seem to work.
Copy code
🚀 nf-test 0.9.2
https://www.nf-test.com
(c) 2021 - 2024 Lukas Forer and Sebastian Schoenherr

Load .nf-test/plugins/nft-bam/0.4.0/nft-bam-0.4.0.jar
Load .nf-test/plugins/nft-utils/0.0.3/nft-utils-0.0.3.jar

Test Process GLIMPSE2_PHASE

  Test [c6e28d6d] 'Should run with vcf' 
    > Nextflow 24.10.5 is available - Please consider updating your version to it
    > N E X T F L O W  ~  version 24.10.3
    > Launching `/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test-c6e28d6de8abe0a1a87b7a131dfa913c.nf` [maniac_celsius] DSL2 - revision: 6f81f626f7
    > ERROR ~ Module compilation error
    > - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
    > - cause: Variable `input_region` already defined in the process scope @ line 35, column 18.
    >        def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    >                     ^
    > 
    > Module compilation error
    > - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
    > - cause: Variable `output_region` already defined in the process scope @ line 35, column 39.
    >        def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    >                                          ^
    > 
    > Module compilation error
    > - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
    > - cause: Variable `input_region` already defined in the process scope @ line 43, column 33.
    >        def input_region_cmd      = input_region        ? "--input-region $input_region"  : ""
    >                                    ^
    > 
    > Module compilation error
    > - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
    > - cause: Variable `input_region` already defined in the process scope @ line 43, column 72.
    >    ion        ? "--input-region $input_regi
    >                                  ^
    > 
    > Module compilation error
    > - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
    > - cause: Variable `output_region` already defined in the process scope @ line 44, column 33.
    >        def output_region_cmd     = output_region       ? "--output-region $output_region": ""
    >                                    ^
    > 
    > Module compilation error
    > - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
    > - cause: Variable `output_region` already defined in the process scope @ line 44, column 73.
    >    ion       ? "--output-region $output_reg
    >                                  ^
    > 
    > 6 errors
    > 
    > 
    >  -- Check script '/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test-c6e28d6de8abe0a1a87b7a131dfa913c.nf' at line: 13 or see '/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test/tests/c6e28d6de8abe0a1a87b7a131dfa913c/meta/nextflow.log' file for more details
    Assertion failed: 

assert process.success
       |       |
       |       false
       GLIMPSE2_PHASE

java.lang.RuntimeException: Process has no output channels. process.out can not be used.
java.lang.RuntimeException: Process has no output channels. process.out can not be used.
FAILED (7.26s)

  Assertion failed: 
  
  3 of 3 assertions failed
  
  Nextflow stdout:
  
  N E X T F L O W  ~  version 24.10.3
  Launching `/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test-c6e28d6de8abe0a1a87b7a131dfa913c.nf` [maniac_celsius] DSL2 - revision: 6f81f626f7
  ERROR ~ Module compilation error
  - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
  - cause: Variable `input_region` already defined in the process scope @ line 35, column 18.
         def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
                      ^
  
  Module compilation error
  - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
  - cause: Variable `output_region` already defined in the process scope @ line 35, column 39.
         def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
                                           ^
  
  Module compilation error
  - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
  - cause: Variable `input_region` already defined in the process scope @ line 43, column 33.
         def input_region_cmd      = input_region        ? "--input-region $input_region"  : ""
                                     ^
  
  Module compilation error
  - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
  - cause: Variable `input_region` already defined in the process scope @ line 43, column 72.
     ion        ? "--input-region $input_regi
                                   ^
  
  Module compilation error
  - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
  - cause: Variable `output_region` already defined in the process scope @ line 44, column 33.
         def output_region_cmd     = output_region       ? "--output-region $output_region": ""
                                     ^
  
  Module compilation error
  - file : /ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf
  - cause: Variable `output_region` already defined in the process scope @ line 44, column 73.
     ion       ? "--output-region $output_reg
                                   ^
  
  6 errors
  
  
   -- Check script '/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test-c6e28d6de8abe0a1a87b7a131dfa913c.nf' at line: 13 or see '/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test/tests/c6e28d6de8abe0a1a87b7a131dfa913c/meta/nextflow.log' file for more details
  Nextflow stderr:
  
  Nextflow 24.10.5 is available - Please consider updating your version to it
  



FAILURE: Executed 1 tests in 7.278s (1 failed)
a
Like this? That error seems weird
Copy code
script:
println output_region
k
Yes. Here is the whole revised file.
Copy code
process GLIMPSE2_PHASE {
    tag "$meta.id"
    label 'process_medium'

    beforeScript  """
    if cat /proc/cpuinfo | grep avx2 -q
    then
        echo "Feature AVX2 present on host"
    else
        echo "Feature AVX2 not present on host"
        exit 1
    fi
    """

    conda "${moduleDir}/environment.yml"
    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        'https://depot.galaxyproject.org/singularity/glimpse-bio:2.0.1--h46b9e50_1':
        'biocontainers/glimpse-bio:2.0.1--h46b9e50_1' }"

    input:
        tuple val(meta) , path(input, arity: '1..*'), path(input_index), path(bamlist), path(samples_file), val(input_region), val(output_region), path(reference), path(reference_index), path(map)
        tuple val(meta2), path(fasta_reference), path(fasta_reference_index)

    output:
        tuple val(meta), path("*.{vcf,vcf.gz,bcf,bgen}"), emit: phased_variants
        tuple val(meta), path("*.txt.gz")               , emit: stats_coverage, optional: true
        path "versions.yml"                             , emit: versions

    when:
        task.ext.when == null || task.ext.when

    script:
    println output_region
    println input_region
    def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    def args   = task.ext.args   ?: ""
    def prefix = task.ext.prefix ?: "${meta.id}_${region}"
    def suffix = task.ext.suffix ?: "bcf"

    def map_command           = map                 ? "--map $map"                    : ""
    def samples_file_command  = samples_file        ? "--samples-file $samples_file"  : ""
    def fasta_command         = fasta_reference     ? "--fasta $fasta_reference"      : ""
    def input_region_cmd      = input_region        ? "--input-region $input_region"  : ""
    def output_region_cmd     = output_region       ? "--output-region $output_region": ""

    def input_type            = input.collect{
        it.toString().endsWithAny("cram", "bam") ? "bam" :
        it.toString().endsWithAny("vcf", "bcf", "vcf.gz") ? "gl" :
        it.getExtension()
    }.unique()

    if (input_type.size() > 1 | !(input_type.contains("gl") | input_type.contains("bam"))) {
        error "Input files must be of the same type and either .bam/.cram or .vcf/.vcf.gz/.bcf format. Found: ${input_type}"
    } else {
        input_type = input_type[0]
    }
    if (input_type == "gl" & input.size() > 1) {
        error "Only one input .vcf/.vcf.gz/.bcf file can be provided"
    }
    def input_list = input.size() > 1

    """
    if [ -n "$bamlist" ] ;
    then
        input_command="--bam-list $bamlist"
    elif $input_list ;
    then
        ls -1 | grep '\\.cram\$\\|\\.bam\$' > all_bam.txt
        input_command="--bam-list all_bam.txt"
    else
        if [ "$input_type" == "bam" ];
        then
            input_command="--bam-file $input"
        elif [ "$input_type" == "gl" ];
        then
            input_command="--input-gl $input"
        else
            echo "Input file type not recognised"
            echo "$input_type"
            exit 1
        fi
    fi

    GLIMPSE2_phase \\
        $args \\
        \$input_command \\
        --reference $reference \\
        $map_command \\
        $fasta_command \\
        $samples_file_command \\
        $input_region_cmd \\
        $output_region_cmd \\
        --thread $task.cpus \\
        --output ${prefix}.${suffix}

    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        glimpse2: "\$(GLIMPSE2_phase --help | sed -nr '/Version/p' | grep -o -E '([0-9]+.){1,2}[0-9]' | head -1)"
    END_VERSIONS
    """

    stub:
    def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    def args   = task.ext.args   ?: ""
    def prefix = task.ext.prefix ?: "${meta.id}_${region}"
    def suffix = task.ext.suffix ?: "bcf"
    """
    touch ${prefix}.${suffix}

    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        glimpse2: "\$(GLIMPSE2_phase --help | sed -nr '/Version/p' | grep -o -E '([0-9]+.){1,2}[0-9]' | head -1)"
    END_VERSIONS
    """
}
a
Ohhhh its the super unintuitive
def
behaviour 🫠
k
It's bit me before!
a
Copy code
def x = "${output_region} ${input_region}"
println(x)
k
Not null!
Copy code
🚀 nf-test 0.9.2
https://www.nf-test.com
(c) 2021 - 2024 Lukas Forer and Sebastian Schoenherr

Load .nf-test/plugins/nft-bam/0.4.0/nft-bam-0.4.0.jar
Load .nf-test/plugins/nft-utils/0.0.3/nft-utils-0.0.3.jar

Test Process GLIMPSE2_PHASE

  Test [c6e28d6d] 'Should run with vcf' 
    > Nextflow 24.10.5 is available - Please consider updating your version to it
    > N E X T F L O W  ~  version 24.10.3
    > Launching `/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test-c6e28d6de8abe0a1a87b7a131dfa913c.nf` [marvelous_jones] DSL2 - revision: 6f81f626f7
    > chr21:16650000-16750000 chr21:16600000-16800000
    > ERROR ~ Error executing process > 'GLIMPSE2_PHASE (input)'
    > 
    > Caused by:
    >   Cannot invoke method replace() on null object -- Check script '/ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf' at line: 37
    > 
    > 
    > Source block:
    >   def x = "${output_region} ${input_region}"
    >   println(x)
    >   def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    >   def args   = task.ext.args   ?: ""
    >   def prefix = task.ext.prefix ?: "${meta.id}_${region}"
    >   def suffix = task.ext.suffix ?: "bcf"
    >   def map_command           = map                 ? "--map $map"                    : ""
    >   def samples_file_command  = samples_file        ? "--samples-file $samples_file"  : ""
    >   def fasta_command         = fasta_reference     ? "--fasta $fasta_reference"      : ""
    >   def input_region_cmd      = input_region        ? "--input-region $input_region"  : ""
    >   def output_region_cmd     = output_region       ? "--output-region $output_region": ""
    >   def input_type            = input.collect{
    >       it.toString().endsWithAny("cram", "bam") ? "bam" :
    >       it.toString().endsWithAny("vcf", "bcf", "vcf.gz") ? "gl" :
    >       it.getExtension()
    >   }.unique()
    >   if (input_type.size() > 1 | !(input_type.contains("gl") | input_type.contains("bam"))) {
    >       error "Input files must be of the same type and either .bam/.cram or .vcf/.vcf.gz/.bcf format. Found: ${input_type}"
    >   } else {
    >       input_type = input_type[0]
    >   }
    >   if (input_type == "gl" & input.size() > 1) {
    >       error "Only one input .vcf/.vcf.gz/.bcf file can be provided"
    >   }
    >   def input_list = input.size() > 1
    >   """
    >   if [ -n "$bamlist" ] ;
    >   then
    >       input_command="--bam-list $bamlist"
    >   elif $input_list ;
    >   then
    >       ls -1 | grep '\\.cram\$\\|\\.bam\$' > all_bam.txt
    >       input_command="--bam-list all_bam.txt"
    >   else
    >       if [ "$input_type" == "bam" ];
    >       then
    >           input_command="--bam-file $input"
    >       elif [ "$input_type" == "gl" ];
    >       then
    >           input_command="--input-gl $input"
    >       else
    >           echo "Input file type not recognised"
    >           echo "$input_type"
    >           exit 1
    >       fi
    >   fi
    >   
    >   GLIMPSE2_phase \\
    >       $args \\
    >       \$input_command \\
    >       --reference $reference \\
    >       $map_command \\
    >       $fasta_command \\
    >       $samples_file_command \\
    >       $input_region_cmd \\
    >       $output_region_cmd \\
    >       --thread $task.cpus \\
    >       --output ${prefix}.${suffix}
    >   
    >   cat <<-END_VERSIONS > versions.yml
    >   "${task.process}":
    >       glimpse2: "\$(GLIMPSE2_phase --help | sed -nr '/Version/p' | grep -o -E '([0-9]+.){1,2}[0-9]' | head -1)"
    >   END_VERSIONS
    >   """
    > 
    > Container:
    >   quay.io/biocontainers/glimpse-bio:2.0.1--h46b9e50_1
    > 
    > Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
    > 
    >  -- Check '/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test/tests/c6e28d6de8abe0a1a87b7a131dfa913c/meta/nextflow.log' file for details
    > Execution cancelled -- Finishing pending tasks before exit
    Assertion failed: 

assert process.success
       |       |
       |       false
       GLIMPSE2_PHASE

java.lang.NullPointerException: Cannot invoke method getAt() on null object
java.lang.RuntimeException: Different Snapshot:
[                                                                                                       [
    [                                                                                                       [
        "versions.yml:md5,c68de03046a6503cdbcf3a1495fc512f"                                        |            
    ]                                                                                                       ]
]                                                                                                       ]

FAILED (9.773s)

  Assertion failed: 
  
  3 of 3 assertions failed
  
  Nextflow stdout:
  
  N E X T F L O W  ~  version 24.10.3
  Launching `/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test-c6e28d6de8abe0a1a87b7a131dfa913c.nf` [marvelous_jones] DSL2 - revision: 6f81f626f7
  chr21:16650000-16750000 chr21:16600000-16800000
  ERROR ~ Error executing process > 'GLIMPSE2_PHASE (input)'
  
  Caused by:
    Cannot invoke method replace() on null object -- Check script '/ballpit/sandbox/kevin/code/astreaforensics-wopr/modules/nf-core/glimpse2/phase/tests/../main.nf' at line: 37
  
  
  Source block:
    def x = "${output_region} ${input_region}"
    println(x)
    def region = input_region    ? "${output_region.replace(":","_")}" : "${reference}"
    def args   = task.ext.args   ?: ""
    def prefix = task.ext.prefix ?: "${meta.id}_${region}"
    def suffix = task.ext.suffix ?: "bcf"
    def map_command           = map                 ? "--map $map"                    : ""
    def samples_file_command  = samples_file        ? "--samples-file $samples_file"  : ""
    def fasta_command         = fasta_reference     ? "--fasta $fasta_reference"      : ""
    def input_region_cmd      = input_region        ? "--input-region $input_region"  : ""
    def output_region_cmd     = output_region       ? "--output-region $output_region": ""
    def input_type            = input.collect{
        it.toString().endsWithAny("cram", "bam") ? "bam" :
        it.toString().endsWithAny("vcf", "bcf", "vcf.gz") ? "gl" :
        it.getExtension()
    }.unique()
    if (input_type.size() > 1 | !(input_type.contains("gl") | input_type.contains("bam"))) {
        error "Input files must be of the same type and either .bam/.cram or .vcf/.vcf.gz/.bcf format. Found: ${input_type}"
    } else {
        input_type = input_type[0]
    }
    if (input_type == "gl" & input.size() > 1) {
        error "Only one input .vcf/.vcf.gz/.bcf file can be provided"
    }
    def input_list = input.size() > 1
    """
    if [ -n "$bamlist" ] ;
    then
        input_command="--bam-list $bamlist"
    elif $input_list ;
    then
        ls -1 | grep '\\.cram\$\\|\\.bam\$' > all_bam.txt
        input_command="--bam-list all_bam.txt"
    else
        if [ "$input_type" == "bam" ];
        then
            input_command="--bam-file $input"
        elif [ "$input_type" == "gl" ];
        then
            input_command="--input-gl $input"
        else
            echo "Input file type not recognised"
            echo "$input_type"
            exit 1
        fi
    fi
    
    GLIMPSE2_phase \\
        $args \\
        \$input_command \\
        --reference $reference \\
        $map_command \\
        $fasta_command \\
        $samples_file_command \\
        $input_region_cmd \\
        $output_region_cmd \\
        --thread $task.cpus \\
        --output ${prefix}.${suffix}
    
    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        glimpse2: "\$(GLIMPSE2_phase --help | sed -nr '/Version/p' | grep -o -E '([0-9]+.){1,2}[0-9]' | head -1)"
    END_VERSIONS
    """
  
  Container:
    quay.io/biocontainers/glimpse-bio:2.0.1--h46b9e50_1
  
  Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`
  
   -- Check '/ballpit/sandbox/kevin/code/astreaforensics-wopr/.nf-test/tests/c6e28d6de8abe0a1a87b7a131dfa913c/meta/nextflow.log' file for details
  Execution cancelled -- Finishing pending tasks before exit
  Nextflow stderr:
  
  Nextflow 24.10.5 is available - Please consider updating your version to it
  

  Snapshots:
    Obsolete snapshots can only be checked if all tests of a file are executed successful.


FAILURE: Executed 1 tests in 9.79s (1 failed)
I spent many hours earlier thinking it really was output_region being null and that replace() being the issue
a
Wait…. hold up, please tell me you don’t have some
withName
config or custom config for this module in your pipeline?
I can smell a
ext.prefix = <something with .replace >
hiding in a wider pipeline config file
And if that’s the case I bet that whatever is being used there is
null
when under test
k
I do have a custom module config, but I didn't think it would be included. Does nf-test include the main pipeline nextflow.config as well?
Seems probable, as the ext.prefix does have a replace in it
a
Yes in the this context it’s pulling in your top-level
nextflow.config
k
Aghhh, I'm sure that must be it then
a
Which I presume then includes the custom config
k
Yes. Ugh, thank you
a
For the record I don’t believe it is
nf-test
that is controlling your code being run with your top-level nextflow config. It simply is running the nextflow command and then the default nextflow behaviour takes over
k
Makes sense. Is there a standard pattern for dealing with that?
(different module configs under a real run vs nf-test)
a
I don’t actually know. I have a feeling the only way might be to mask your config temporarily
Alternatively if you step into the
modules
subdir to run the test then it won’t pick it up as it won’t be at the same level as you launch from
k
Makes sense. Alright, well I really appreciate your help!
a
I would suggest asking further in #nf-test or creating an issue on the github (for nf-test)
k
Cool, will do
a
It’s a little bit of a faf (because you have to bascially re-duplicate the module config in reverse), but could you try creating a `nf-test-modules-override.config`:
Copy code
process {
withName: GLIMPSE2_PHASE {
 ext.prefix = null
}
}
And then add it as an include in your
tests/nextflow.config
file? The caveat here is that you will need to set
withName: <NAME>
to be at least as selective as whatever your pipeline config uses (i.e. if that is
withName: PIPELINE_NAME:MODULE_NAME
you would need the same in this)
k
Yeah, that does indeed work. Curious to see if any others have suggestions in #nf-test, but that will definitely be the backup plan!