https://nf-co.re logo
Join Slack
Powered by
<@U05LY7TP5SL> I was wondering whether the options...
# parabricks
f
@Gary Burnett I was wondering whether the options for memory usage could be a bit more flexible rather than either having 
--low-memory
for 16GB or 38GB (https://docs.nvidia.com/clara/parabricks/latest/tutorials/fq2bam_tutorial.html). Background: my cluster for example has 24GB GPU RAM and I would like to use that all but I need to add 
--low-memory
to fq2bam otherwise the process always fails.
g
Hey Famke. Thank you for letting me know about how this is affecting you. The behavior you'd prefer is that it takes up the full 24 GB if it's available?
f
I think I would prefer that I can set 
--max-memory 24G
(or something along that side) instead of 
--low-memory
which makes it more customizable for me instead of being fixed to the 16GB because I cannot provide the full 38GB. But I am not sure if there is a reason behind this fixed value.
g
I see. Because if your GPU is between 16 GB and 38 GB then you have GPU memory that isn't being used when 
--low-memory
is active. Let me talk to the team and see what they think.
👍 1
f
I just found a small description in the docs somewhere that I missed beforehand:
For 16GB-40GB devices (e.g. L4, A10): Optimize performance by adjusting 
--bwa-nstreams
to a value between 2 and 3. Depending on what the specific device can handle given its device memory capacity.
So maybe if that documentation would be a bit more specific for different architectures that would probably suffice for now.
g
Maybe this is a good case for us to create a better "optimizing performance" section. With examples and graphs to make it clear what each of the flags are doing and how to maximize the GPU hardware.
f
That sounds good, maybe also with the focus on different types of (nvidia) hardware - for example I have A30 in my cluster but its impossible to find out what to do for 24GB A30 
--bwa-nstreams
just from searching so now I'll try around but I guess having this listed somewhere makes it easier to estimate what would work.
Also it says set the value to something between 2 and 3 (from which I assume a float) but it only takes an integer so either 2 or 3 I guess.
g
I definitely understand. There are a lot of different hardware configurations with different memory sizes so something more detailed would help.
f
For me
Copy code
--low-memory
works on my nodes, but the fq2bam process was running for an hour or so on one Exome sample. But
Copy code
--bwa-nstreams 3
gives me the same out of memory error as with out the flag. And
Copy code
--bwa-nstreams 2
seems to work (still waiting for the processes to finish in order to be able to tell how long it would take).
g
Please let me know the results of 2 streams
👍 1
f
For now I think something is stuck at bqsr 😞
Copy code
[PB Info 2025-Jul-25 15:49:20] Sorting and Marking: 20.001 seconds
[PB Info 2025-Jul-25 15:49:20] ------------------------------------------------------------------------------
[PB Info 2025-Jul-25 15:49:20] ||        Program:                                  Sorting Phase-II        ||
[PB Info 2025-Jul-25 15:49:20] ||        Version:                                           4.4.0-1        ||
[PB Info 2025-Jul-25 15:49:20] ||        Start Time:                       Fri Jul 25 15:49:00 2025        ||
[PB Info 2025-Jul-25 15:49:20] ||        End Time:                         Fri Jul 25 15:49:20 2025        ||
[PB Info 2025-Jul-25 15:49:20] ||        Total Time:                                     20 seconds        ||
[PB Info 2025-Jul-25 15:49:20] ------------------------------------------------------------------------------
[PB Info 2025-Jul-25 15:49:20] ------------------------------------------------------------------------------
[PB Info 2025-Jul-25 15:49:20] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2025-Jul-25 15:49:20] ||                              Version 4.4.0-1                             ||
[PB Info 2025-Jul-25 15:49:20] ||                         Marking Duplicates, BQSR                         ||
[PB Info 2025-Jul-25 15:49:20] ------------------------------------------------------------------------------
[PB Info 2025-Jul-25 15:49:32] BQSR using CUDA device(s): { 0 1 }
[PB Info 2025-Jul-25 15:49:48] progressMeter -	Percentage
[PB Info 2025-Jul-25 15:49:58] 0.0
[PB Info 2025-Jul-25 15:50:08] 0.0
[PB Info 2025-Jul-25 15:50:33] 0.0
[PB Info 2025-Jul-25 15:51:01] 0.0
[PB Info 2025-Jul-25 15:51:31] 0.0
[PB Info 2025-Jul-25 15:52:01] 0.0
[PB Info 2025-Jul-25 15:52:34] 0.0
[PB Info 2025-Jul-25 15:53:17] 0.0
[PB Info 2025-Jul-25 15:53:58] 0.0
[PB Info 2025-Jul-25 15:54:35] 0.0
[PB Info 2025-Jul-25 15:55:14] 0.0
[PB Info 2025-Jul-25 15:55:47] 0.0
[PB Info 2025-Jul-25 15:56:17] 0.0
[PB Info 2025-Jul-25 15:56:54] 0.0
[PB Info 2025-Jul-25 15:57:23] 0.0
[PB Info 2025-Jul-25 15:57:52] 0.0
[PB Info 2025-Jul-25 15:58:25] 0.0
[PB Info 2025-Jul-25 15:59:00] 0.0
g
Is it just this one sample? Or have you seen this with others, too?
f
Honestly thats the test-dataset for sarek and I just discovered this. Sorry for spamming you (that was not my intention with this request). I will try and investigate what is going on but that also explains why the 
--low-memory
run was seemingly running forever . Might be something with my setup but at the moment I have no idea. Just wanted to let you know that you'll probably have to wait a bit until I can get you any more feedback on this.
g
That's okay! I'm glad we could talk in real time 🙂
f
I am trying again with 
--gpuwrite --gpusort
lets see if the error persists. And I will try turning off the BQSR completely to see whether my nextflow processes finish properly. I'll keep you updated on this.
g
Yes let me know how it finishes!
f
Yeah that did not help as far as I can tell. Should I post in the nvidia forum with all my log files and more info on the gpu nodes out cluster uses? Or what would be the best way?
g
You can send me your logs and I can reproduce it and pass it along.
You said you're using nf-core test data? Can you point me to which one?
f
Let me grab it for you: I am running nf-core/sarek in the current dev version with:
Copy code
nextflow run nf-core/sarek -r dev -profile m3c,test_full --aligner parabricks --outdir testing
Complete config and params attached
nextflow.config
g
Thank you. I will take a look at this today.
f
elated_perlman-params.json
elated_perlman-params.json
Here is our cluster wiki: https://wiki.m3c.uni-tuebingen.de/en/Slurm/Overview And the settings I used specifically for fq2bam:
Copy code
process {
   withName: 'PARABRICKS_FQ2BAM' {
      ext.args = '--bwa-options="-K 10000000" --bwa-nstreams 2 --gpuwrite --gpusort'
   }
}
g
Okay this will help me try to reproduce the issue.
f
It seems like this issue was resolved with Version 4.6.0-1. Thank you. The change seems to be that by standard now: instead of "Running with 2 GPU(s), using 4 stream(s) per device with 16 worker threads per GPU" grouping is used "Running with 2 GPU(s), using 2 stream(s) per device with 2 group(s) of 16 CPU worker thread(s)". So I do not directly run out of memory - lets see if the bqsr issue resolves as well 🙂
I mean something seems to happen in the bqsr process:
Copy code
[PB Debug 2025-Oct-16 10:44:24][src/PBTempFile.cpp:184] Bin 701 Meta 27.944 seconds
[PB Debug 2025-Oct-16 10:44:24][src/PBTempFile.cpp:184] Bin 702 Meta 28.132 seconds
[PB Debug 2025-Oct-16 10:44:24][src/PBTempFile.cpp:184] Bin 706 Meta 28.092 seconds
[PB Debug 2025-Oct-16 10:44:24][src/PBTempFile.cpp:184] Bin 707 Meta 28.193 seconds
[PB Debug 2025-Oct-16 10:44:24][src/PBTempFile.cpp:184] Bin 705 Meta 28.242 seconds
[PB Debug 2025-Oct-16 10:44:24][src/PBTempFile.cpp:184] Bin 708 Meta 28.314 seconds
[PB Debug 2025-Oct-16 10:44:24][src/PBTempFile.cpp:184] Bin 704 Meta 28.308 seconds
[PB Debug 2025-Oct-16 10:44:24][src/main.cu:858] Write 0 Delete 0
[PB Debug 2025-Oct-16 10:44:24][src/main.cu:861] Write 0
[PB Info 2025-Oct-16 10:44:24] 0.0
(I turned on the verbose out put). Lets wait for some more time. Could it be that the progress meter is broken?
I am removing --gpuwrite and --gpusort again to see if that fixes my problem. When I run --align-only I get my crams after 5 Minutes but I need the other steps as well.
Without those flags I get
Copy code
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 0 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 1 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 2 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 3 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 4 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 5 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 6 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 7 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 8 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 9 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 10 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 11 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 12 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 13 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 14 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 15 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 16 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 17 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 18 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile process queue 19 size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile output queue size: 0
[PB Info 2025-Oct-16 13:03:05] PBCramFile pool queue size: 0
[PB Debug 2025-Oct-16 13:03:05][src/main.cu:858] Write 0 Delete 0
[PB Debug 2025-Oct-16 13:03:05][src/main.cu:861] Write 0
[PB Info 2025-Oct-16 13:03:05] 0.0
Yeah I cannot resolve this so any help is welcome @Gary Burnett - I also searched the whole developer forum. Just as a brief summary I am running this command:
Copy code
#!/usr/bin/env bash -C -e -u -o pipefail
INDEX=`find -L ./ -name "*.amb" | sed 's/\.amb$//'`
cp Homo_sapiens_assembly38.fasta $INDEX

pbrun \
    fq2bam \
    --ref $INDEX \
    --in-fq SRR7890919_WES_cross-site_study_1.fastq.gz SRR7890919_WES_cross-site_study_2.fastq.gz \
    --out-bam normal-wes.cram \
    --knownSites dbsnp_146.hg38.vcf.gz --knownSites Mills_and_1000G_gold_standard.indels.hg38.vcf.gz --knownSites Homo_sapiens_assembly38.known_indels.vcf.gz \
    --out-recal-file normal-wes.table \
    --interval-file S07604624_Covered_human_all_v6_plus_UTR.liftover.to.hg38.sorted.bed \
     \
    --read-group-id-prefix normal-wes.lane_1 --read-group-sm seqc2-wes_normal-wes --read-group-lb normal-wes --read-group-pl ILLUMINA

cat <<-END_VERSIONS > versions.yml
"NFCORE_SAREK:SAREK:FASTQ_PREPROCESS_PARABRICKS:PARABRICKS_FQ2BAM":
        pbrun: $(echo $(pbrun version 2>&1) | sed 's/^Please.* //' )
END_VERSIONS
and everything works nicely up until Marking Duplicates, BQSR where I somehow get stuck... and th elog looks like this:
Copy code
INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred

[Parabricks Options Mesg]: Checking argument compatibility
[Parabricks Options Mesg]: Auto mode: Setting --bwa-nstreams and device memory parameters automatically based on
available GPU memory. These settings are optimized for the GRCh38 reference genome and may need manual adjustment for
other references or optimal performance. For manual configuration guidance, see the fq2bam documentation.
[Parabricks Options Mesg]: Set --bwa-options="-K #" to produce compatible pair-ended results with previous versions of
fq2bam or BWA MEM.
[Parabricks Options Mesg]: Read group created for /tmp/nahbu450-1940266/nxf.7ypNWcatal/SRR7890919_WES_cross-
site_study_1.fastq.gz and /tmp/nahbu450-1940266/nxf.7ypNWcatal/SRR7890919_WES_cross-site_study_2.fastq.gz
[Parabricks Options Mesg]: @RG\tID:normal-wes.lane_1.1\tLB:normal-wes\tPL:ILLUMINA\tSM:seqc2-wes_normal-wes\tPU:normal-
wes.lane_1.1
[PB Info 2025-Oct-16 13:29:40] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:29:40] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2025-Oct-16 13:29:40] ||                              Version 4.6.0-1                             ||
[PB Info 2025-Oct-16 13:29:40] ||                      GPU-PBBWA mem, Sorting Phase-I                      ||
[PB Info 2025-Oct-16 13:29:40] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:29:40] Mode = pair-ended-gpu
[PB Info 2025-Oct-16 13:29:40] Running with 2 GPU(s), using 2 stream(s) per device with 2 group(s) of 16 CPU worker thread(s)
[PB Info 2025-Oct-16 13:29:41] Read 3171 alt contigs
[PB Info 2025-Oct-16 13:29:50] # 16 10  2  2  0  12 pool:  0 0 bases/GPU/minute: 0.0 
[PB Info 2025-Oct-16 13:30:00] # 20  1 19  0  0   0 pool:  3 1219295033 bases/GPU/minute: 3657885099.0 
[PB Info 2025-Oct-16 13:30:10] # 19  2 18  2  0   0 pool:  2 2289260484 bases/GPU/minute: 3209896353.0 
[PB Info 2025-Oct-16 13:30:20] # 20  0 20  0  0   0 pool:  3 3367570375 bases/GPU/minute: 3234929673.0 
[PB Info 2025-Oct-16 13:30:30] # 19  0 19  1  0   0 pool:  4 4462445796 bases/GPU/minute: 3284626263.0 
[PB Info 2025-Oct-16 13:30:40] # 20  3 17  2  0   0 pool:  2 5549008584 bases/GPU/minute: 3259688364.0 
[PB Info 2025-Oct-16 13:30:50] # 19  2 18  1  0   0 pool:  5 6643881673 bases/GPU/minute: 3284619267.0 
[PB Info 2025-Oct-16 13:31:00] # 20  1 18  2  0   0 pool:  4 7697371969 bases/GPU/minute: 3160470888.0 
[PB Info 2025-Oct-16 13:31:10] # 19  6 13  8  0   0 pool:  4 8759031771 bases/GPU/minute: 3184979406.0 
[PB Info 2025-Oct-16 13:31:20] # 20  0 20  1  0   0 pool:  9 9903707951 bases/GPU/minute: 3434028540.0 
[PB Info 2025-Oct-16 13:31:30] # 20  2 18  0  0   0 pool: 10 10982026928 bases/GPU/minute: 3234956931.0 
[PB Info 2025-Oct-16 13:31:40] # 20  0 20  2  0   0 pool:  8 12076942176 bases/GPU/minute: 3284745744.0 
[PB Info 2025-Oct-16 13:31:50] # 20  1 19  0  0   0 pool: 10 13130318049 bases/GPU/minute: 3160127619.0 
[PB Info 2025-Oct-16 13:32:00] # 20  2 18  2  0   0 pool:  8 14216915981 bases/GPU/minute: 3259793796.0 
[PB Info 2025-Oct-16 13:32:10] # 20  3 17  4  0   0 pool:  6 15270390197 bases/GPU/minute: 3160422648.0 
[PB Info 2025-Oct-16 13:32:20] # 20  8 12  7  0   0 pool:  3 16356944992 bases/GPU/minute: 3259664385.0 
[PB Info 2025-Oct-16 13:32:30] # 20  0 20  0  0   0 pool: 10 17443629942 bases/GPU/minute: 3260054850.0 
[PB Info 2025-Oct-16 13:32:40] # 20  0 20  0  0   0 pool: 10 18455601213 bases/GPU/minute: 3035913813.0 
[PB Info 2025-Oct-16 13:32:50] # 20  9 11  9  0   0 pool:  2 19467546099 bases/GPU/minute: 3035834658.0 
[PB Info 2025-Oct-16 13:33:00] # 20  8 12  8  0   0 pool:  4 20521009166 bases/GPU/minute: 3160389201.0 
[PB Info 2025-Oct-16 13:33:10] # 20  3 17  3  0   0 pool:  9 21582725683 bases/GPU/minute: 3185149551.0 
[PB Info 2025-Oct-16 13:33:13] Time spent reading: 62.619445 seconds
[PB Info 2025-Oct-16 13:33:20] Rate stats (based on sampling every 10 seconds):
	min rate: 0.0 bases/GPU/minute
	max rate: 3657885099.0 bases/GPU/minute
	avg rate: 3083246526.1 bases/GPU/minute
[PB Info 2025-Oct-16 13:33:20] Time spent monitoring (multiple of 10): 220.009
[PB Info 2025-Oct-16 13:33:20] bwalib run finished in 219.672 seconds
[PB Info 2025-Oct-16 13:33:20] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:33:20] ||        Program:                    GPU-PBBWA mem, Sorting Phase-I        ||
[PB Info 2025-Oct-16 13:33:20] ||        Version:                                           4.6.0-1        ||
[PB Info 2025-Oct-16 13:33:20] ||        Start Time:                       Thu Oct 16 13:29:40 2025        ||
[PB Info 2025-Oct-16 13:33:20] ||        End Time:                         Thu Oct 16 13:33:20 2025        ||
[PB Info 2025-Oct-16 13:33:20] ||        Total Time:                           3 minutes 40 seconds        ||
[PB Info 2025-Oct-16 13:33:20] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:33:21] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:33:21] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2025-Oct-16 13:33:21] ||                              Version 4.6.0-1                             ||
[PB Info 2025-Oct-16 13:33:21] ||                             Sorting Phase-II                             ||
[PB Info 2025-Oct-16 13:33:21] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:33:21] progressMeter - Percentage
[PB Info 2025-Oct-16 13:33:21] 0.0
[PB Info 2025-Oct-16 13:33:26] 28.5
[PB Info 2025-Oct-16 13:33:31] 58.0
[PB Info 2025-Oct-16 13:33:36] 79.4
[PB Info 2025-Oct-16 13:33:41] Sorting and Marking: 20.001 seconds
[PB Info 2025-Oct-16 13:33:41] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:33:41] ||        Program:                                  Sorting Phase-II        ||
[PB Info 2025-Oct-16 13:33:41] ||        Version:                                           4.6.0-1        ||
[PB Info 2025-Oct-16 13:33:41] ||        Start Time:                       Thu Oct 16 13:33:21 2025        ||
[PB Info 2025-Oct-16 13:33:41] ||        End Time:                         Thu Oct 16 13:33:41 2025        ||
[PB Info 2025-Oct-16 13:33:41] ||        Total Time:                                     20 seconds        ||
[PB Info 2025-Oct-16 13:33:41] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:33:41] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:33:41] ||                 Parabricks accelerated Genomics Pipeline                 ||
[PB Info 2025-Oct-16 13:33:41] ||                              Version 4.6.0-1                             ||
[PB Info 2025-Oct-16 13:33:41] ||                         Marking Duplicates, BQSR                         ||
[PB Info 2025-Oct-16 13:33:41] ------------------------------------------------------------------------------
[PB Info 2025-Oct-16 13:33:53] BQSR using CUDA device(s): { 0 1 }
[PB Info 2025-Oct-16 13:34:09] progressMeter -	Percentage
[PB Info 2025-Oct-16 13:34:19] 0.0
[PB Info 2025-Oct-16 13:34:29] 0.0
[PB Info 2025-Oct-16 13:34:39] 0.0
[PB Info 2025-Oct-16 13:35:00] 0.0
[PB Info 2025-Oct-16 13:35:29] 0.0
[PB Info 2025-Oct-16 13:35:51] 0.0
[PB Info 2025-Oct-16 13:36:07] 0.0
Everything works nicely with 
parabricks/fq2bam
up until Marking Duplicates, BQSR where I somehow get stuck and the process never finishes. If anyone has an idea here feel free to jump on this thread and let me know 🙌
g
I will talk to the eng team about this behavior. We're all at a conference this week, so the earliest I could get some eyes on it is next week.
f
Yeah no worries - now you know!
3 Views
Open in Slack
PreviousNext
Powered by