Hello! I'm wondering if anyone else has experience in running variant calling on WGS samples via Sarek. I'm building a workflow, but run into a lot of issues. Particularly with GATK4_MARKDUPLICATES taking +12Hrs to run. Eventually everything just times out and I have to

-resume

. I have had lots of success with WES variant calling with Sarek, but WGS seems to be a totally different challenge. I believe my primary issue(s) are: 1. Not allocating resources properly, or not enough resources. 2. Not allocating enough time. 3. Slurm/Nextflow interaction shenanigans. Some notes: • This is done via an HPC with no internet access. The HPC has a limit of 128 CPU Cores and 500Gb of RAM. I do have GPU node that I have not tested yet. • I have run the test profile successfully (via downloading the required test files via github). • The primary input is Illumina FASTQs are ~30Gb each with tumor-normal pairs. I also have Ultima CRAMs at ~230Gbs each, but that has the same issue as well. Could someone sanity check my config and launch command? If these are not the issue, some advice? Is it normal for WGS to take over 3 days? Config:

params {
    wes                    = false 
    split_fastq            = 25000000
    nucleotides_per_second = 50000
    max_cpus               = 64
    max_memory             = 192.GB
    max_time               = 96.h
}

singularity {
  enabled = true
  autoMounts = true
}

process {
  executor = 'slurm'
  scratch = true
  
  resourceLimits = [
    cpus: 64,
    memory: 192.GB, 
    time: 96.h
  ]

  withLabel:process_single {
    cpus   = { 1 * task.attempt }
    memory = { 4.GB * task.attempt }
    time   = { 24.h * task.attempt }
  }

  withLabel:process_low {
    cpus   = { 2 * task.attempt }
    memory = { 8.GB * task.attempt }
    time   = { 24.h * task.attempt }
  }

  withLabel:process_medium {
    cpus   = { 8  * task.attempt } 
    memory = { 28.GB * task.attempt }
    time   = { 24.h * task.attempt }
  }

  withLabel:process_high {
    cpus   = { 16 * task.attempt } 
    memory = { 56.GB * task.attempt }
    time   = { 36.h * task.attempt }
  }

  withLabel:process_long {
    cpus   = { 16 * task.attempt }
    memory = { 56.GB * task.attempt }
    time   = { 48.h }
  }

  withName: '.*GATK4_MARKDUPLICATES.*' {
    cpus   = { 16 * task.attempt }
    memory = { 80.GB * task.attempt }
    time   = { 48.h * task.attempt }
  }
  
  withName: '.*GATK4_BASERECALIBRATOR.*' {
    cpus   = { 16 * task.attempt }
    memory = { 80.GB * task.attempt }
    time   = { 48.h * task.attempt }
  }
}

Launch Command:

nextflow run "$SAREK_DIR" \
    -profile singularity \
    -c "$CONFIG" \
    -work-dir "$WORK_DIR" \
    --input "$INPUT" \
    --outdir "$OUTPUT" \
    --genome GATK.GRCh38 \
    --igenomes_base "$iGENOMES" \
    --aligner bwa-mem2 \
    --tools ngscheckmate,mutect2,cnvkit,vep \
    --vep_cache_version "115" \
    --vep_genome "GRCh38" \
    --vep_species "homo_sapiens" \
    --vep_cache "$VEP_CACHE" \
    --vep_custom_args "--everything --per_gene --total_length --offline"

Nothing seems out of place to me. If I remember well, MarkDuplicates can behave weirdly and takes very long time if it has too much memory

I think you just have to sit it out, I ran some WGS analyses a couple years ago (with v3.3.2) and it took close to a week (with a few restarts). Interestingly I didn't need to allocate more time to MarkDuplicates, but

fastp

timed out, and a single VEP process could run for several days. It all depends on your read depth and number of samples, etc., but the default config of sarek is more geared toward WES.