Hi, everybody! Has anyone encountered any kind of problem with inconsistent steps in a pipeline succeeding and failing with no apparent pattern? I am having problems with

damageprofiler.

I'm running nf-core/eager, v2.5.1 using Nextflow v22.10.6 with DSL1. I have three samples from the same sequencing run that are pretty similar to each other in terms of size (after clipping they each have from 145 to 175million reads), quality and reads average length. I ran nf-core/eager separately for each of them. Sample 1 worked on first try in 15 minutes; Sample 2 first failed 3 times (submitted damage profiler 4 times, canceling it after 1.5 hours). I got impatient, canceled the run and re-sent it and then it succeeded after two tries (failed one was 1 hour long, the second, successful one, lasted 25 minutes). Sample 3 is still running but has failed already 3 times with a similar pattern as sample 2. All failed runs display the same error message (memory problems):

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at htsjdk.samtools.util.StringUtil.bytesToString(StringUtil.java:301)
	at htsjdk.samtools.BAMRecord.decodeReadName(BAMRecord.java:440)
	at htsjdk.samtools.BAMRecord.getReadName(BAMRecord.java:252)
	at htsjdk.samtools.SAMUtils.validateCigar(SAMUtils.java:937)
	at htsjdk.samtools.SAMRecord.validateCigar(SAMRecord.java:1807)
	at htsjdk.samtools.BAMRecord.getCigar(BAMRecord.java:284)
	at htsjdk.samtools.SAMRecord.isValid(SAMRecord.java:2103)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:848)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:834)
	at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:802)
	at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:574)
	at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:553)
	at calculations.DamageProfiler.extractSAMRecords(DamageProfiler.java:107)
	at calculations.StartCalculations.start(StartCalculations.java:298)
	at RunDamageProfiler.main(RunDamageProfiler.java:37)

I'm confused because all samples and all tries have had the same resources (16 CPUs and 20GB of memory) and sent the same command (I checked in the work/ directories in the .sh file), and they fail randomly. The only certain thing is that if damageprofiler takes more than 30 minutes it will fail, but I'd like to avoid having runs stuck on a step over and over if I can help it. One thing that I've noticed, though, is that sample 2 and 3 both got scheduled on the same HPC node (not at the same time): could that be the problem? Thank you for anybody that has an idea!

Hi Flavia! Are you using a container profile? It could be that there are inconsistencies across different nodes in your HPC, and without containerisation the behaviour of damageprofiler is slightly different on each node. If you are using a container, then it might still be that some nodes in the HPC have reduced performance. That would also explain why the jobs failed when scheduled on the same node 3x. You could configure nextflow to exclude that node, and have your HPC managers check those nodes. Quick note, DamageProfiler is a Java program, and hence uses a garbage collector. In my experience this has often caused odd behaviours when using on an HPC. When the amount of available memory is borderline for the needs of the program, it can get stuck in a loop where the memory is low, to the GC jumps in to free some, and as soon as it does, more memory is needed, starting the GC again. The scripts can then get “stuck” with really long runtime before they eventually fail/succeed. Bumping up the memory solves this issue, and often cuts down on the runtime considerably. I’ve had picard MarkDuplicates runs fail after 16h, then finish in a matter of minutes once the memory was bumped up! Given you provide DamageProfiler 20GB, I doubt this is the specific issue here, but it’s good to keep in mind. Finally, since v 2.5.0, you can use mapDamage for producing damage patterns within nf-core/eager. This is technically slower than DamageProfiler, but mapDamage allows you to set a maximum number of reads to be used for the calculation. This makes it much more consistent in terms of runtime, even for more deeply sequenced data. Generally, ~100k-1M reads should be enough to accurately estimate damage %, so there’s hardly a need to go through all the reads in most cases (which DamageProfiler does). So for processing data of differing sequence depths, I would suggest using mapDamage instead.

Hi!! Yes, I'm using singularity! And about the node performance, I'll try to gather more information on its status. I already found out how to exclude specific nodes, so I will do that for the time being and keep it only for short partition jobs/interactive sessions. Thank you for all the helpful tips and explanation on how DamageProfiler works: unfortunately I can only use this version of eager as we have to use the version of nextflow with DSL1 for the time being (I don't have all the information, but the server itself needs more updates). I'll surely check MapDamage once I have the okay to use the new version 😊👍

Happy to help! (from your other message, I saw you’re acctually using v 2.5.1, so you can already use mapDamage 😉 )

Ah! Even better!! 👀