Hi all, I maintain Parabricks nextflow workflows on NVIDIA's GitHub, however there's a lot of overlap with the nextflow workflows on the nf-core github. I've heard that there is a way to have the nf-core modules pull directly from GitHub so that there aren't two versions of these workflows floating around, but I am struggling to find documentation on how to do that. Can someone point me in the right direction? Or maybe I am misunderstanding how this could work and we can have a discussion about what makes the most sense. Thank you!

hey I am looking to incorporate a new parabricks module, and was wondering, how do you test the module ?

My guess is locally to start with, and then I guess we'll figure at the hackathon if we can get GPUS on the runners

hi parabricks folks (@Gary Burnett / @Ben Siranosian /others), I need some clarification: • I am testing the fq2bam_meth tool and i'm testing it with a

g4dn.12xlarge 48Core 192GB

instance • my run command:

pbrun fq2bam_meth --low-memory \
--ref /reference-genomes/hg38_with_spikeins/hg38_with_spikeins.fa \
--in-fq /raw-data/21_R1_001.fastq.gz /raw-data/21_R2_001.fastq.gz \
--out-bam /gputmp/out.bam \
--tmp-dir /gputmp \
--gpusort \
--gpuwrite

• now it starts of well and all the 4 GPUs are utilized nicely but, after a while the GPU utilization drops to 0 and looks like there is some minimal constant CPU usage • The

stderr

part of the log also seems to be stuck at the following pool:

[PB Info 2024-Oct-08 14:02:08] # 100 15 100 100  0   0 pool:  5 76093418204 bases/GPU/minute: 0.0 
[PB Info 2024-Oct-08 14:02:18] # 100 15 100 100  0   0 pool:  5 76093418204 bases/GPU/minute: 0.0 
[PB Info 2024-Oct-08 14:02:28] # 100 15 100 100  0   0 pool:  5 76093418204 bases/GPU/minute: 0.0 
[PB Info 2024-Oct-08 14:02:38] # 100 15 100 100  0   0 pool:  5 76093418204 bases/GPU/minute: 0.0

I even tried the

--align-only

mode but, still same issue. It seems to be stuck at this state for 3+ hrs now. Is this normal ? Will it complete eventually or something wrong with the tool/algorithm ? Would appreciate any insights here 🙏

hi @Gary Burnett does parabricks have any initiatives to support open-source orgs like nf-core with some free GPUs to test and integrate the modules ?

hi can i get a review for this fq2bam_meth module. I wasn't able to add any tests atm but, will do so right after we get some testing infra in place. Thanks

Getting this error testing the fq2bamameth module....any body have an idea on what could be wrong ?

[Parabricks Options Mesg]: Checking argument compatibility
    [Parabricks Options Mesg]: Automatically generating ID prefix
    [Parabricks Options Mesg]: Read group created for /home/ubuntu/nf-modules/.nf-test/tests/37c84f77c462e64864e9f897f5d7958
    6/work/stage-0263a36d-abba-4028-9020-189edeaaf67c/0e/c0c18db2598c8f93a571f41429c776/test.methylated_1.fastq.gz
    [Parabricks Options Mesg]: @RG\tID:1_MT192765.1.1\tLB:lib1\tPL:bar\tSM:test\tPU:1_MT192765.1.1
    [PB Info 2024-Oct-25 11:15:28] ------------------------------------------------------------------------------
    [PB Info 2024-Oct-25 11:15:28] ||                 Parabricks accelerated Genomics Pipeline                 ||
    [PB Info 2024-Oct-25 11:15:28] ||                              Version 4.3.2-1                             ||
    [PB Info 2024-Oct-25 11:15:28] ||                      GPU-PBBWA mem, Sorting Phase-I                      ||
    [PB Info 2024-Oct-25 11:15:28] ------------------------------------------------------------------------------
    [PB Info 2024-Oct-25 11:15:28] Mode = single-ended-gpu
    [PB Info 2024-Oct-25 11:15:28] Running with 1 GPU(s), using 4 stream(s) per device with 16 worker threads per GPU
    [PB Error 2024-Oct-25 11:15:28][-unknown-:0] Received signal: 11
    For technical support visit <https://docs.nvidia.com/clara/index.html#parabricks>, exiting.
    For technical support visit <https://docs.nvidia.com/clara/index.html#parabricks>
    Exiting...
    Please visit <https://docs.nvidia.com/clara/#parabricks> for detailed documentation

Hi everyone. I am trying to migrate the tests of parabricks haplotypecaller to nf-test. When I launch the command

nf-core modules create parabricks/haplotypecaller --migrate-pytest

I keep getting this error:

FileNotFoundError: [Errno 2] No such file or directory: 'modules/nf-core/parabricks/haplotypecaller_old/environment.yml'

Any idea why? I am using the latest versions of nf-core tools, nextflow and nf-test.

│     [PB Info 2024-Nov-12 12:38:19] Checking if the index file exists for the input compressed                                                 │
│ file:/home/ubuntu/modules/.nf-test/tests/f6910383a92212b3689aeb11b1d6c493/work/stage-b7e018ed-73a8-4eef-8c47-d5cde5ac77c4/fd/0f4ae853e6eabf54 │
│ 9c6df43e24f04d/dbsnp_146.hg38.vcf.gz for better performance                                                                                   │
│     [PB Warning 2024-Nov-12 12:38:19][src/PBLocalFile.cpp:54] Could not find index file                                                       │
│ /home/ubuntu/modules/.nf-test/tests/f6910383a92212b3689aeb11b1d6c493/work/stage-b7e018ed-73a8-4eef-8c47-d5cde5ac77c4/fd/0f4ae853e6eabf549c6df │
│ 43e24f04d/dbsnp_146.hg38.vcf.gz.tbi                                                                                                           │
│     [PB Warning 2024-Nov-12 12:38:19][src/PBLocalFile.cpp:54] Could not find index file                                                       │
│ /home/ubuntu/modules/.nf-test/tests/f6910383a92212b3689aeb11b1d6c493/work/stage-b7e018ed-73a8-4eef-8c47-d5cde5ac77c4/fd/0f4ae853e6eabf549c6df │
│ 43e24f04d/dbsnp_146.hg38.vcf.tbi                                                                                                              │
│     [PB Warning 2024-Nov-12 12:38:19][src/PBLocalFile.cpp:54] Could not find index file                                                       │
│ /home/ubuntu/modules/.nf-test/tests/f6910383a92212b3689aeb11b1d6c493/work/stage-b7e018ed-73a8-4eef-8c47-d5cde5ac77c4/fd/0f4ae853e6eabf549c6df │
│ 43e24f04d/dbsnp_146.hg38.vcf.gz.csi                                                                                                           │
│     [PB Warning 2024-Nov-12 12:38:19][src/PBLocalFile.cpp:54] Could not find index file                                                       │
│ /home/ubuntu/modules/.nf-test/tests/f6910383a92212b3689aeb11b1d6c493/work/stage-b7e018ed-73a8-4eef-8c47-d5cde5ac77c4/fd/0f4ae853e6eabf549c6df │
│ 43e24f04d/dbsnp_146.hg38.vcf.csi                                                                                                              │
│     [PB Warning 2024-Nov-12 12:38:19][src/PBLocalFile.cpp:54] Could not find index file                                                       │
│ /home/ubuntu/modules/.nf-test/tests/f6910383a92212b3689aeb11b1d6c493/work/stage-b7e018ed-73a8-4eef-8c47-d5cde5ac77c4/fd/0f4ae853e6eabf549c6df │
│ 43e24f04d/dbsnp_146.hg38.vcf.gz.tbi                                                                                                           │
│     [PB Error 2024-Nov-12 12:38:19][src/dbsnp_vcf.cpp:223] Cannot open file                                                                   │
│ "/home/ubuntu/modules/.nf-test/tests/f6910383a92212b3689aeb11b1d6c493/work/stage-b7e018ed-73a8-4eef-8c47-d5cde5ac77c4/fd/0f4ae853e6eabf549c6d │
│ f43e24f04d/dbsnp_146.hg38.vcf.gz"No such file or directory, exiting.

seems like parabricks only looks for the files in the stage dir 😞 the files are available in the work dir but not in the stage. Any idea what I can do to fix that?

this looks like the issue I had... 😞

can we do some symlink magic?

@Gary Burnett We ran into issues regarding symlinks and copying files into the work-dir https://nfcore.slack.com/archives/C04SDJYV12L/p1731581652491889?thread_ts=1731421258.620959&cid=C04SDJYV12L

I ported all remaining modules to nf-test 🙂 now I am just waiting for the original PR (applybqsr) to pass all tests (there is some issue with singularity and the runners?) to finish the rest of accordingly (adding to the nf-test gpu section and so on). But I would be happy if you find some time to review what I changed: https://github.com/nf-core/modules/pull/6977 https://github.com/nf-core/modules/pull/6980 https://github.com/nf-core/modules/pull/6991 https://github.com/nf-core/modules/pull/6995 https://github.com/nf-core/modules/pull/6996

I created a

gpu

label to keep track of PRs that relate to GPU

The subworkflow for the alignment with fq2bam and applybqsr is done I think, the tests pass locally. Maybe someone here can have another look and add their review? https://github.com/nf-core/modules/pull/6876

Hey @Gary Burnett - @Simon Pearce and me were wondering whether it would be possible to somehow enable

bgzip

within the parabricks containers. Reason:

deepvariant

only outputs non-zipped vcf files which tend to be larger than needed.

I am trying to update the haplotypecaller module and even though I am using the

--htvc-low-memoy

flag cuda always exits with 255 out of memory ([PB Error 2025-Feb-11 145730][src/likehood_test.cu:1011] cudaSafeCall() failed: out of memory, exiting.) The question is: Is this a haplotypecaller problem or is there something wrong with my setup? Running

nvidia-smi

yields:

Tue Feb 11 16:01:41 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA H100 PCIe               Off | 00000000:21:00.0 Off |                    0 |
| N/A   66C    P0             295W / 310W |  79788MiB / 81559MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA H100 PCIe               Off | 00000000:A1:00.0 Off |                    0 |
| N/A   36C    P0              77W / 310W |    463MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    226463      C   /usr/local/envs/colabfold/bin/python       1652MiB |
|    0   N/A  N/A   3269196      C   ...rm/miniconda3/envs/scGPT/bin/python    75420MiB |
|    1   N/A  N/A    226463      C   /usr/local/envs/colabfold/bin/python        454MiB |
+---------------------------------------------------------------------------------------+

I am trying to expand the output of the fq2bam module with cram to integrate it into sarek. But for some reason the same test data that works when the output file is a bam file does not work when its a cram file. Is there an additional flag that needs to be set when I wanr cram output? I did not find anything in the docs. Here is the PR: https://github.com/nf-core/modules/pull/7502

Hi everyone! 👋 I’m running into some frustrating and inconsistent OOM (Out of Memory) issues when using Parabricks within a Nextflow pipeline, hoping someone here might help me untangle what’s going on 🙏 The problem When I run

pbrun deepvariant_germline

(or

collectmultiplemetrics

) directly via SLURM, it completes just fine. But when I run it through Nextflow, using the same input data, container, and resource parameters, I randomly get OOM errors. Some samples succeed, others fail — and I can’t spot any pattern in input size or content. My setup - Nextflow: 24.10.5 - Parabricks container: clara-parabricks_4.2.1-1.sif (via Apptainer/Singularity with --nv) - SLURM cluster node: 2× NVIDIA A100 80GB GPUs, 32 CPUs, ~80 GB RAM I don’t have access to --gres, but I can confirm the GPU is visible and used in the container (via nvidia-smi) *-*Process resource definition (in

nextflow.config

):

process {
    executor = 'slurm'
    errorStrategy = "terminate"
    cleanup = false
    cache = 'lenient'
    stageInMode = 'copy'
    stageOutMode = 'rsync'
    environment = [
        TMPDIR: "${params.tmp_dir}",
        NXF_TEMP: "${params.tmp_dir}",
        APPTAINER_TMPDIR: "${params.tmp_dir}",
        NXF_SCRATCH: "${params.tmp_dir}"
    ]

    withLabel: use_gpu {
        executor = 'slurm'
        containerOptions = params.use_gpu ? "--nv" : ""
    }

    withName: RUN_PARABRICS {
        label = 'use_gpu'
        cpus = 28
        memory = '74G'
        containerOptions = '--nv --bind /shared -W /shared/work/tmp'
        clusterOptions = { "--account ${params.account_name} --partition=${params.slurm_config.gpu_partition}" }
        container = 'clara-parabricks_4.2.1-1.sif'
        scratch = true
        stageInMode = 'copy'
        stageOutMode = 'rsync'
    }
}

And this this the command in

<http://main.nf|main.nf>

process RUN_PARABRICS {
  label 'use_gpu'
  containerOptions = '--nv'
  scratch true
  publishDir "results/$sample_name", mode: "copy"

  input:
  tuple val(sample_name), val(fastq_file_path)

  output:
  tuple val(sample_name), path("${sample_name}.cram"), emit: cram
  tuple val(sample_name), path("${sample_name}.cram.crai"), emit: crai
  tuple val(sample_name), path("${sample_name}.g.vcf.gz"), emit: gvcf
  tuple val(sample_name), path("${sample_name}.g.vcf.gz.tbi"), emit: gvcf_idx
  tuple val(sample_name), path("${sample_name}.metrics.csv"), emit: metrics

  script:
  if (params.fastq_pattern == 'single')
    """
    pbrun deepvariant_germline \
      --ref ${reference_fasta} \
      --in-fq ${fastq_file_path}/${sample_name}_R1.fastq.gz \
          ${fastq_file_path}/${sample_name}_R2.fastq.gz \
          "@RG\\tID:${sample_name}\\tLB:lib1\\tPL:ILLUMINA\\tSM:${sample_name}\\tPU:unit1" \
      --gpuwrite \
      --gpusort \
      --num-gpus 1 \
      --bwa-cpu-thread-pool 28 \
      --gvcf \
      --out-variants ${sample_name}.g.vcf.gz \
      --out-bam ${sample_name}.cram \
      --out-duplicate-metrics ${sample_name}.metrics.csv \
      --memory-limit ${params.memory_limit}
    """
  else if (params.fastq_pattern == 'lanes')
    """
    pbrun deepvariant_germline \
      --ref ${reference_fasta} \
      --in-fq ${fastq_file_path}/${sample_name}_L001_R1_001.fastq.gz \
          ${fastq_file_path}/${sample_name}_L001_R2_001.fastq.gz \
          "@RG\\tID:${sample_name}_L001\\tLB:lib1\\tPL:ILLUMINA\\tSM:${sample_name}\\tPU:unit1" \
      --in-fq ${fastq_file_path}/${sample_name}_L002_R1_001.fastq.gz \
          ${fastq_file_path}/${sample_name}_L002_R2_001.fastq.gz \
          "@RG\\tID:${sample_name}_L002\\tLB:lib2\\tPL:ILLUMINA\\tSM:${sample_name}\\tPU:unit2" \
      --in-fq ${fastq_file_path}/${sample_name}_L003_R1_001.fastq.gz \
          ${fastq_file_path}/${sample_name}_L003_R2_001.fastq.gz \
          "@RG\\tID:${sample_name}_L003\\tLB:lib3\\tPL:ILLUMINA\\tSM:${sample_name}\\tPU:unit3" \
      --in-fq ${fastq_file_path}/${sample_name}_L004_R1_001.fastq.gz \
          ${fastq_file_path}/${sample_name}_L004_R2_001.fastq.gz \
          "@RG\\tID:${sample_name}_L004\\tLB:lib4\\tPL:ILLUMINA\\tSM:${sample_name}\\tPU:unit4" \
      --gpuwrite \
      --gpusort \
      --num-gpus 1 \
      --bwa-cpu-thread-pool 28 \
      --gvcf \
      --out-variants ${sample_name}.g.vcf.gz \
      --out-bam ${sample_name}.cram \
      --out-duplicate-metrics ${sample_name}.metrics.csv \
      --memory-limit ${params.memory_limit}
    """
  else
    error "please specify fastq_pattern"

  stub:
  """
  touch ${sample_name}.cram
  touch ${sample_name}.cram.crai
  touch ${sample_name}.g.vcf.gz
  touch ${sample_name}.g.vcf.gz.tbi
  touch ${sample_name}.metrics.csv
  """
}

A SLURM job outside of Nextflow runs fine: max RSS ~68 GB (with following configuration)

#!/bin/bash
#SBATCH --ntasks=28
#SBATCH --mem=74G
#SBATCH --partition=gpu
#SBATCH --error="job-germline-%j_%x.err"
#SBATCH --output="job-germline-%j_%x.out"

apptainer exec --nv clara-parabricks_4.2.1-1.sif \
pbrun deepvariant_germline ...

Same command inside Nextflow: OOM, sometimes using >75 GB RSS, expecially during

BQRS

steps but also

collectmultiplemetrics

Sometimes it runs fine on fews samples, and then crashes on one, even with identical input sizes. I’ve been struggling with this for quite a while, and even the smallest insight would be incredibly helpful — this SLURM OOM error is turning into a nightmare for me 😩 because I just can’t understand or get a handle on what’s causing it. I’m wondering if anyone can help me to figure out how Nextflow+Apptainer handle memory or task allocation differ from plain SLURM in a way that explains this? Is there any recommended config for running Parabricks tools in Nextflow on GPU clusters like mine? This has been a bit of a rabbit hole and I’d really appreciate any help, tips, or even guesses from folks who’ve run Parabricks in similar setups. Thanks a lot in advance!

@Gary Burnett I was wondering whether the options for memory usage could be a bit more flexible rather than either having

--low-memory

for 16GB or 38GB (https://docs.nvidia.com/clara/parabricks/latest/tutorials/fq2bam_tutorial.html). Background: my cluster for example has 24GB GPU RAM and I would like to use that all but I need to add

--low-memory

to fq2bam otherwise the process always fails.

I was wondering if we should change the way we report versions on the parabricks modules. This is what we get as output:

------------------------------------------------------------------------------------------
Version Information
------------------------------------------------------------------------------------------
Tool:			fq2bam
pbrun:			4.5.1-1
------------------------------------------------------------------------------------------
Compatible With:
BWA mem:		0.7.15
Picard:			2.18.25
------------------------------------------------------------------------------------------

I would vote to add the pbrun version but also the compatible with versions to the verions.yml string. What do the other parabricks users think?

Porting the conversation here: https://nfcore.slack.com/archives/CJRH30T6V/p1761060776603539

Alternatively, can I get a snapshot / AMI / description of the nf-core test environment so I can reproduce it exactly?

Does anyone run parabricks on batch and if yes, what marketplace AMI would you reccomend for running parabricks (on GPU hardware) in AWS batch?

Any idea why parabricks' test fail on singularity test 1 and 2 but pass on the rest? I have this PR but I can't figure out how to fix those two tests (and the errors are not that informative to me). Thanks. https://github.com/nf-core/modules/pull/9309

It seems NVIDIA Parabricks website is down. It just displays blank page everywhere. Are you also experiencing the same issue? Any idea what might be going on?

Hey Y'all When running fq2bam manually (on a dedicated RTX 6000 Ada), via the docker run command, I can successfully process a one of the paired pod5 files. This uses 39GB of my 48GB graphics RAM. The sample was sequenced across 4 lanes. When I try to use Sarek to run fq2bam, I apparently run out of memory. I reduced the number of bwa streams to 2 from 4, using the custom config script here. Is there anything else I can do to make it run?

Anyone else having issues with the PG field / line in the bam header when using fq2bam ?

Just opened a PR to update all the Parabricks modules to 4.8 if anyone wants to review or has any comments!