Hi Team, I am seeking help to run oncoanalyser for my WES data. I have 14 tumor samples to analyze. Just to get familiar with the pipeline and the processes, I first ran in

wgts

mode, the pipeline completed successfully, generated all expected outputs including detailed orange reports.

Now, I am trying to run in

target

mode. the initial

panel_resource_creation

step completed successfully. I used

hg38_exome_v2.0.<http://2_targets_sorted_validated.re|2_targets_sorted_validated.re>_annotated.bed

as —`target_regions_bed` (from Twist bioscience as the sequencing library was prepared using this kit) and

DriverGenePanel.38.tsv

The pipeline log, however says,

pipeline completed with errors

, only producing

alignments

,

amber

,

cobalt

,

logs

, and

sage_calling

directories within each sample folder

For a quick peek, the last part of the

.out

files is as below:

```executor > slurm (197)

[96/816da1] NFC… (Patient_57_Patient_57_Y) | 14 of 14 ✔

[81/ae6953] NFC…ient_57_Patient_57_Y_0004) | 150 of 150 ✔

[- ] NFC…D_ALIGNMENT_RNA:STAR_ALIGN -

[- ] NFC…LIGNMENT_RNA:SAMTOOLS_SORT -

[- ] NFC…IGNMENT_RNA:SAMBAMBA_MERGE -

[- ] NFC…T_RNA:GATK4_MARKDUPLICATES -

[16/a02aaa] NFC… (Patient_50_Patient_50_Y) | 14 of 14 ✔

[- ] NFC…OFOX_QUANTIFICATION:ISOFOX -

[7f/ddf3c6] NFC…OFILING:AMBER (Patient_60) | 3 of 3 ✔

[78/c54d58] NFC…FILING:COBALT (Patient_60) | 3 of 3

[0e/199c80] NFC…CALLING:ESVEE (Patient_60) | 3 of 3

[- ] NFC…SAGE_CALLING:SAGE_GERMLINE -

[c0/130d97] NFC…:SAGE_SOMATIC (Patient_60) | 3 of 3 ✔

[- ] NFC…E_ANNOTATION:PAVE_GERMLINE -

[73/228706] NFC…:PAVE_SOMATIC (Patient_36) | 1 of 1

[- ] NFC…ETEDPURPLE CALLINGPURPLE -

[- ] NFC…PPEND:SAGE_APPEND_GERMLINE -

[- ] NFC…APPEND:SAGE_APPEND_SOMATIC -

[- ] NFC…X_ANNOTATION:LINX_GERMLINE -

[- ] NFC…NX_ANNOTATION:LINX_SOMATIC -

[f7/d38db5] NFC… (Patient_60_Patient_60_N) | 3 of 3 ✔

[f8/247dae] NFC… (Patient_60_Patient_60_N) | 3 of 3 ✔

Plus 5 more processes waiting for tasks…

-[nf-core/oncoanalyser] Pipeline completed with errors-

ERROR ~ Execution aborted due to an unexpected error

-- Check script '/n/home12/upendrabhattarai/.nextflow/assets/nf-core/oncoanalyser/modules/local/purple/main.nf' at line: 6 or see '.nextflow.log' file for more details

ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

-- Check '.nextflow.log' file for details```

I can forward other log files and info if any can be helpful.

In one of the GitHub issue says, there might be some retrained resources for some commercial exome kits, I am wondering if there are any for our kit. Additionally, I also noticed that some of our genes of interest are not included in the driver gene panel. Based on a GitHub issue discussion, it seems that genes that are not present in this driver gene panel will not be included in the analysis report, so the `—-mode target`` is not recommended in those cases.

Given that, would you recommend proceeding with the

—mode wgts

output for the whole exome sequencing data interpretation? Are there specific caveats, for example, potentially inaccurate coverage statistics and other, that we should be aware of when using WES data with the whole-genome mode?

Thank you

Hi there, thanks for the detailed explanation. I'll have a look at resource files readily made for your kit but in the mean time, you should be able to run with the panel resource creation mode. These logs are not super informative, if you could show which process failed, I can take a closer look 🙂

Hi Rayan, Thank you for your reply. attached is the .nextflow.log from this run. It is reporting

esvee/Samplename.esvee.germline.vcf.gz.tbi

unable to find. inside the esvee directory I see

Samplename.esvee.somatic.vcf.gz.tbi

and esvee.unfiltered.vcf.gz.tbi not germline.vcf file. another error reported is

ERROR nextflow.processor.TaskProcessor Execution aborted due to an unexpected error java.lang.NullPointerException: Cannot get property containerEngine' on null object

esvee was run like:

#!/usr/bin/env bash -C -e -u -o pipefail
mkdir -p esvee/

esvee \
    -Xmx73443940762 \
     \
    -tumor Patient_6_N \
    -tumor_bam Patient_6_N.redux.bam \
     \
     \
    -esvee_prep_dir esvee/ \
    -ref_genome GRCh38_masked_exclusions_alts_hlas.fasta \
    -ref_genome_version 38 \
    -known_hotspot_file known_fusions.38.bedpe \
    -pon_sgl_file sgl_pon.38.bed.gz \
    -pon_sv_file sv_pon.38.bedpe.gz \
    -repeat_mask_file repeat_mask_data.38.fa.gz \
    -unmap_regions unmap_regions.38.tsv \
    -bamtool $(which sambamba) \
    -write_types 'PREP_JUNCTION;PREP_BAM;FRAGMENT_LENGTH_DIST;JUNC_ASSEMBLY;PHASED_ASSEMBLY;ALIGNMENT;BREAKEND;VCF' \
    -threads 12 \
    -log_level DEBUG \
    -output_dir esvee/

cat <<-END_VERSIONS > versions.yml
"NFCORE_ONCOANALYSER:TARGETED:ESVEE_CALLING:ESVEE":
    esvee: $(java -jar ${ESVEE_JAR} -version | sed 's/^.*Esvee version: //')
END_VERSIONS

and .command.err shows:

/usr/local/bin/esvee: line 6: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
/n/netscratch/hsph_bioinfo/Lab/nfcore_oncoanalyser_targeted/work/57/43153a8391d9127ec6be9c378f7c51/.command.sh: line 25: ESVEE_JAR: unbound variable

can you find any esvee files yourself ? or are they not generated altogether ? The

unbound variable

error is quite odd and would explain ESVEE not running

Hi, It looks like the esvee job completed, the unbound variable was just a warning. attached is the command.out for a esvee job. Inside the work/...../.. directory. I can find esvee directory and below are the contents from one of the esvee directory:

Patient_6_N.esvee.alignment.tsv        Patient_6_N.esvee.prep.disc_stats.tsv	   Patient_6_N.esvee.ref_depth.vcf.gz.tbi
Patient_6_N.esvee.assembly.tsv	       Patient_6_N.esvee.prep.fragment_length.tsv  Patient_6_N.esvee.somatic.vcf.gz
Patient_6_N.esvee.breakend.tsv	       Patient_6_N.esvee.prep.junction.tsv	   Patient_6_N.esvee.somatic.vcf.gz.tbi
Patient_6_N.esvee.phased_assembly.tsv  Patient_6_N.esvee.raw.vcf.gz		   Patient_6_N.esvee.unfiltered.vcf.gz
Patient_6_N.esvee.prep.bam	       Patient_6_N.esvee.raw.vcf.gz.tbi		   Patient_6_N.esvee.unfiltered.vcf.gz.tbi
Patient_6_N.esvee.prep.bam.bai	       Patient_6_N.esvee.ref_depth.vcf.gz

Below are my scripts: For panel resource creation:

nextflow run nf-core/oncoanalyser \
       -revision 2.2.0 \
       -profile singularity \
       -config path/to/reference_data.config \
       --mode panel_resource_creation \
       --genome GRCh38_hmf \
       --target_regions_bed path/to/hg38_exome_v2.0.2_targets_sorted_validated.re_annotated.bed \
       --driver_gene_panel path/toDriverGenePanel.38.tsv \
       --input path/to/metadata.csv \
       -c path/to/fas.config \
       --outdir path/to/nfcore_target_out

and script for the nfcore run in targeted mode:

nextflow run nf-core/oncoanalyser \
        -revision 2.2.0 \
        -profile singularity \
        -config path/to/panel.config \
        -config path/to/reference_data.config \
        --mode targeted \
        --panel wes_panel \
        --genome GRCh38_hmf \
        --force_panel \
        --input path/to/metadata.csv \
        -c path/to/fas.config \
        --outdir path/to/nfcore_target_out \
        -resume

Do you see anything needs fixing here? anyother file I can provide to help troubleshoot? Thank you for your help.

Hi Upen! Do you have a separate config with anything singularity related by any chance ? I don't see anything wrong (esvee completed successfully and you see the output) that would point towards oncoanalyser or the tools as the culprit(s). I would lean towards something more HPC related, like singularity, etc but tbd based on what you have

Hi Rayan, Thanks for looking into it. I do have config file for our cluster. Please find my config files and script in the attachment. I really appreciate your time and help.

could you try adding these directives in your config ?

singularity {
    enabled    = true
    autoMounts = true
    runOptions = "--bind /local/:/local"
}

Hi Rayan, I updated fas.config as:

executor.$slurm.queueSize = 30
process {
    executor = 'slurm'
    queue    = 'shared'
}

profiles {

        singularity {
                enabled    = true
                autoMounts = true
                runOptions = "--bind /local/:/local"
        }
}

job failed with

tail job.out

as:

-- Check '.nextflow.log' file for details

executor >  slurm (4)
[ed/76ae66] NFC… (Patient_60_Patient_60_N) | 14 of 14, cached: 14 ✔
[23/128fe9] NFC…ient_57_Patient_57_Y_0002) | 120 of 120, cached: 120
[-        ] NFC…D_ALIGNMENT_RNA:STAR_ALIGN -
[-        ] NFC…LIGNMENT_RNA:SAMTOOLS_SORT -
[-        ] NFC…IGNMENT_RNA:SAMBAMBA_MERGE -
[-        ] NFC…T_RNA:GATK4_MARKDUPLICATES -
[a6/53726e] NFC…UX (Patient_7_Patient_7_N) | 9 of 9, cached: 9
[-        ] NFC…OFOX_QUANTIFICATION:ISOFOX -
[7a/89882d] NFC…OFILING:AMBER (Patient_13) | 2 of 8, cached: 1
[3f/5bd208] NFC…FILING:COBALT (Patient_13) | 3 of 8, cached: 1
[f6/3cf878] NFC…CALLING:ESVEE (Patient_36) | 1 of 8, cached: 1
[-        ] NFC…SAGE_CALLING:SAGE_GERMLINE -
[fa/2c3d56] NFC…:SAGE_SOMATIC (Patient_36) | 1 of 8, cached: 1
[-        ] NFC…E_ANNOTATION:PAVE_GERMLINE -
[e9/2ca115] NFC…:PAVE_SOMATIC (Patient_36) | 1 of 1, cached: 1 ✔
[-        ] NFC…ETED:PURPLE_CALLING:PURPLE -
[-        ] NFC…PPEND:SAGE_APPEND_GERMLINE -
[-        ] NFC…APPEND:SAGE_APPEND_SOMATIC -
[-        ] NFC…X_ANNOTATION:LINX_GERMLINE -
[-        ] NFC…NX_ANNOTATION:LINX_SOMATIC -
[cf/554441] NFC… (Patient_36_Patient_36_N) | 1 of 8, cached: 1
[a8/db30f1] NFC… (Patient_13_Patient_13_N) | 2 of 8, cached: 1
Plus 5 more processes waiting for tasks…
-[nf-core/oncoanalyser] Pipeline completed with errors-
ERROR ~ Execution aborted due to an unexpected error

 -- Check script '/n/home12/upendrabhattarai/.nextflow/assets/nf-core/oncoanalyser/modules/local/purple/main.nf' at line: 6 or see '.nextflow.log' file for more details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details

Different error’s an improvement ! 😅 - could you share the nextflow.log of the run and command.log of the failed purple process ?

Some progress 😄. I have attached nextflow.log below. It looks like PURPLE did not start atall. I couldn't find the purple directory to look for command.log. The complain on the .out life above "the line 6 at /purple/main.nf" is about singularity. I am pasting the purple.main.nf below. maybe that can provide some hint for the fix? Thank you

process PURPLE {
    tag "${meta.id}"
    label 'process_medium'

    conda "${moduleDir}/environment.yml"
    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        '<https://depot.galaxyproject.org/singularity/hmftools-purple:4.2--hdfd78af_0>' :
        'biocontainers/hmftools-purple:4.2--hdfd78af_0' }"

    input:
    tuple val(meta), path(amber_dir), path(cobalt_dir), path(sv_tumor_vcf), path(sv_tumor_tbi), path(sv_normal_vcf), path(sv_normal_tbi), path(smlv_tumor_vcf), path(smlv_normal_vcf)
    path genome_fasta
    val genome_ver
    path genome_fai
    path genome_dict
    path gc_profile
    path sage_known_hotspots_somatic
    path sage_known_hotspots_germline
    path driver_gene_panel
    path ensembl_data_resources
    path germline_del
    path target_region_bed
    path target_region_ratios
    path target_region_msi_indels

    output:
    tuple val(meta), path('purple/'), emit: purple_dir
    path 'versions.yml'             , emit: versions
    path '.command.*'               , emit: command_files

    when:
    task.ext.when == null || task.ext.when

    script:
    def args = task.ext.args ?: ''

    def xmx_mod = task.ext.xmx_mod ?: 0.75

    def log_level_arg = task.ext.log_level ? "-log_level ${task.ext.log_level}" : ''

    def reference_arg = meta.containsKey('normal_id') ? "-reference ${meta.normal_id}" : ''

    def sv_tumor_vcf_arg = sv_tumor_vcf ? "-somatic_sv_vcf ${sv_tumor_vcf}" : ''
    def sv_normal_vcf_arg = sv_normal_vcf ? "-germline_sv_vcf ${sv_normal_vcf}" : ''

    def smlv_tumor_vcf_arg = smlv_tumor_vcf ? "-somatic_vcf ${smlv_tumor_vcf}" : ''
    def smlv_normal_vcf_arg = smlv_normal_vcf ? "-germline_vcf ${smlv_normal_vcf}" : ''

    def sage_known_hotspots_germline_arg = sage_known_hotspots_germline ? "-germline_hotspots ${sage_known_hotspots_germline}" : ''
    def germline_del_arg = germline_del ? "-germline_del_freq_file ${germline_del}" : ''

    def target_region_bed_arg = target_region_bed ? "-target_regions_bed ${target_region_bed}" : ''
    def target_region_ratios_arg = target_region_ratios ? "-target_regions_ratios ${target_region_ratios}" : ''
    def target_region_msi_indels_arg = target_region_msi_indels ? "-target_regions_msi_indels ${target_region_msi_indels}" : ''

    """
    purple \\
        -Xmx${Math.round(task.memory.bytes * xmx_mod)} \\
        ${args} \\
        -tumor ${meta.tumor_id} \\
        ${reference_arg} \\
        -amber ${amber_dir} \\
        -cobalt ${cobalt_dir} \\
        ${sv_tumor_vcf_arg} \\
        ${sv_normal_vcf_arg} \\
        ${smlv_tumor_vcf_arg} \\
        ${smlv_normal_vcf_arg} \\
        -ref_genome ${genome_fasta} \\
        -ref_genome_version ${genome_ver} \\
        -driver_gene_panel ${driver_gene_panel} \\
        -ensembl_data_dir ${ensembl_data_resources} \\
        -somatic_hotspots ${sage_known_hotspots_somatic} \\
        ${sage_known_hotspots_germline_arg} \\
        ${target_region_bed_arg} \\
        ${target_region_ratios_arg} \\
        ${target_region_msi_indels_arg} \\
        ${germline_del_arg} \\
        -gc_profile ${gc_profile} \\
        -circos \$(which circos) \\
        -threads ${task.cpus} \\
        ${log_level_arg} \\
        -output_dir purple/

    cat <<-END_VERSIONS > versions.yml
    "${task.process}":
        purple: \$(purple -version | sed -n '/^Purple version / { s/^.* //p }')
    END_VERSIONS
    """

    stub:
    """
    mkdir purple/

    touch purple/${meta.tumor_id}.purple.cnv.gene.tsv
    touch purple/${meta.tumor_id}.purple.cnv.somatic.tsv
    touch purple/${meta.tumor_id}.purple.driver.catalog.germline.tsv
    touch purple/${meta.tumor_id}.purple.driver.catalog.somatic.tsv
    touch purple/${meta.tumor_id}.purple.germline.vcf.gz
    touch purple/${meta.tumor_id}.purple.germline.vcf.gz
    touch purple/${meta.tumor_id}.purple.purity.tsv
    touch purple/${meta.tumor_id}.purple.qc
    touch purple/${meta.tumor_id}.purple.somatic.vcf.gz
    touch purple/${meta.tumor_id}.purple.sv.germline.vcf.gz
    touch purple/${meta.tumor_id}.purple.sv.vcf.gz

    echo -e '${task.process}:\\n  stub: noversions\\n' > versions.yml
    """
}

is there a specific reason you're defining the singularity-related configs under the profile scope ? That could cause issues/clashes with the actual singularity profile. Could you try it exactly like I suggested and we can see?

singularity {
    enabled    = true
    autoMounts = true
    runOptions = "--bind /local/:/local"
    cacheDir   = '/local/path/to/apptainer/cache/.singularity'
    // Apptainer cache dir (or singularity depending on system) - to reuse previously stored images should you want to 
}


executor {
    name            = 'slurm'
    perTaskReserve  = false
    perJobMemLimit  = true
    submitRateLimit = '10 sec' // if you'd like to toggle the rate at which samples are submitted 
    queueSize = 30 // your previous queueSize
}

process {
	queue = 'shared' // your previous queue specification
}

Hi Rayan, thank you for correcting me. No specific reason to include singularity config under profile, it was just my dumb logic as singularity was passed with -profile flag in the script. I corrected my config file as:

singularity {
    enabled    = true
    autoMounts = true
    runOptions = "--bind $PWD"
    cacheDir   = '/n/netscratch/hsph_bioinfo/Lab/sarah_hill/nfcore_oncoanalyser_targeted/workdir/tmp'
    // Apptainer cache dir (or singularity depending on system) - to reuse previously stored images should you want to 
}


executor {
    name            = 'slurm'
    perTaskReserve  = false
    perJobMemLimit  = true
    submitRateLimit = '10 sec' // if you'd like to toggle the rate at which samples are submitted 
    queueSize = 30 // your previous queueSize
}

process {
        queue = 'shared' // your previous queue specification
}

and ran the job again. This time I got more outputs, in the output folders for some sample there new folders created but it is not consistent across all the sample output folder, and it looks like this:

tree -L 2
.
├── panel_resources
│   ├── cobalt.region_normalisation.38.tsv
│   ├── DriverGenePanel.38.tsv
│   ├── hg38_exome_v2.0.2_targets_sorted_validated.re_annotated.bed
│   ├── logs
│   ├── pave.somatic_artefacts.38.tsv
│   └── target_region_ratios.tsv
├── Patient_10
│   ├── alignments
│   ├── amber
│   ├── bamtools
│   ├── cider
│   ├── cobalt
│   ├── esvee
│   ├── logs
│   └── sage_calling
├── Patient_12
│   ├── alignments
│   ├── amber
│   ├── cider
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_13
│   ├── alignments
│   ├── amber
│   ├── cider
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_23
│   ├── alignments
│   ├── amber
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_25
│   ├── alignments
│   ├── amber
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_34
│   ├── alignments
│   ├── amber
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_35
│   ├── alignments
│   ├── amber
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_36
│   ├── alignments
│   ├── amber
│   ├── bamtools
│   ├── cider
│   ├── cobalt
│   ├── esvee
│   ├── logs
│   ├── pave
│   └── sage_calling
├── Patient_48
│   ├── alignments
│   ├── amber
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_50
│   ├── alignments
│   ├── amber
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_57
│   ├── alignments
│   ├── amber
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_6
│   ├── alignments
│   ├── amber
│   ├── bamtools
│   ├── cider
│   ├── cobalt
│   ├── esvee
│   ├── logs
│   ├── pave
│   └── sage_calling
├── Patient_60
│   ├── alignments
│   ├── amber
│   ├── cobalt
│   ├── logs
│   └── sage_calling
├── Patient_7
│   ├── alignments
│   ├── amber
│   ├── cobalt
│   ├── logs
│   └── sage_calling
└── pipeline_info

attached is the .nextflow.log and the job.out still showing the same error the

line 6 in purple/main.nf

as before. I can't figure it out, whats going on. Thank you for your help.

Could you try a “clean” run with this config ? Clean work directory, nextflow cache (the .nextflow directory), etc. Ideally not using the singularity cache either

Hi Rayan, I will try that today and update you on progress.

A quick question, if I am interested in knowing just the gain and loss of some genes from my Whole exome sequencing data, can I trust the result from running WES data in whole genome mode, or that will in inaccurate? Thank you

Hi Upen, no worries - I would be quite cautious actually, without any sort of proper normalization, these results wouldn't necessarily be trustworthy

Hi Rayan, thank you for your insights, thats very helpful. My oncoanalyzer run with exome panel is still getting error while starting purple. I have attached all my logs, and config files below. If you have any suggestions for troubleshooting, please let me know. I would be very thankful.

Did ESVEE successfully run ? Does the test profile actually work in your environment with your version of Nextflow ?

Hi Rayan, thank you for trying to troubleshoot the errors. Test profile worked and ESVEE ran successfully, I started with fresh installation of next flow and remind all the cache and everything. I think the error is from how it is configured in my local system, the container. Anyway. I appreciate your time and I think I will go with different approach/pipeline for this dataset. Thank you.

Hi Upen, did you use that config when running the test profile? It indeed it looks like a container issue and I'm leaning towards Nextflow not being able to resolve the

$PWD

in your runOptions directive and that's why it's complaining about not being able to find files even though you've said that the processes upstream were successful. This is not part of oncoanalyser itself and you'll run into this issue with other pipelines as well.