Hi again, I’ve realised that the performance of oncoanalyser has decreased significantly compared to previous versions. In particular, marking duplicates and structural variant calling. I see that for the same sample Markdups took around 3h (with 6cpus, 36GB) while Redux is taking 15h (12cpus, 72GB). Similarly, all rules in the SVPREP subworkflow took less than 1 hour, while Esvee is taking 6hours with more resources. These numbers are not directly comparable since the previous runs were done with genome version 37 while the current ones are done on 38. Is this something you have also observed? And is there any plans to either improve performance of Redux, or support alternative tools for this step (markdups or other)? Thanks in advance!

Hi @Eva Caceres thanks for flagging this, that's super weird since that on our data (both hg19 and GRCh38) the opposite was observed (i.e. quicker than old version)

Hi Eva, is this sample-specific or across cohorts as well?

Not what we've been seeing (with fresh frozen or FFPE samples). We might not notice EsVee as it can run in parallel to other tasks but REDUX would have shown up.

Hi, thanks for replying! We are using the genome files from you, so that should not be the issue. I have only done few tests on 38, all of them tumor-only, but I’ll try to run few more to see if this is general or was sample-specific. Could you let me know the number of cpus and mem you usually set for Redux and Esvee, as well as, then number of reads you usually have? We are sequencing with Illumina Novaseq X, was redux tested with data from that machine?

Tumor only should not have an impact on the REDUX runtime (we're running T/N), at around 80-100x coverage. We're still on NovaSeq 6000 though.

@Eva Caceres we have just finished our Novaseq X validations and haven't seen anything like this

on premise cluster with slurm

check, @Rayan Hassaïne has done the most work on SLURM here but he's off today so I'll aks him to get back to you with some configs he's been using. Could you please share the run (CPU/mem/walltime) usage stats for those 3 runs ?

If the jobs finished .. do you have the duplication rate for these? Just checking they are not through the roof.

will check and come back to you

Will also try to get you some information next week.

Thank you!

Hi @Eva Caceres , as pointed out by Joep and Oliver, this is quite odd and the complete opposite of what we've seen ourselves. Could you share the .command.log files of the REDUX task so we can see what it was doing for 15 hours (same for ESVEE) ? Thanks

Thanks Rayan, I will do

Hi, thanks for sharing, I'd like to try to understand the issue first since it's the first we've seen this degree of regression

We are just wrapping up the 2.3 release then it's bac to looking at some internal benchmarks.

Hi Eva, with the 2.3.0 release out of the way I’ve had some time to look at this

Thank you, Stephen! Over to you, Eva; any information on the alignment process for both versions would be valuable, too.

Hi @Stephen Watts thanks a lot! This is very useful. Since we have tested different unrelated samples from different batches and sequencing runs, I don’t think this is sample related. I will run the HCC1395 data as soon as I get to it and report back. Hopefully we find something there. Just one more question, I see you are using a lot of memory for REDUX. We have seen that when allocating a high amount of memory, our system ends up using more cpus than allocated. Could you provide the CPU hours for redux and markdups instead of the wall time? It would be good to see that they are in agreement with wall times. Thanks a lot for looking into it! 🙏

Hi @Eva Caceres, the behaviour where more CPUs are used when SLURM jobs are allocated more memory is surprising and I wonder whether there is some other HPC-related issue there - how are you measuring CPU allocation / utilisation in this case? In terms of CPU-hours, both REDUX and MarkDups were allocated 16 CPUs giving: • oncoanalyser 1.0.0 (MarkDups): 40.0 CPU hours [tumor]; 20.8 CPU hours [normal] • oncoanalyser 2.2.0 (REDUX): 38.4 CPU hours [tumor]; 17.6 CPU hours [normal] (Also just to note, I mistakingly described the SV calling data as CPU hours but this is actually walltime)

Hi thanks! I don’t discard this is a cluster issue, specially if you don’t see the same behaviour; and we have had cluster related issues in the past. It has been busy here, but I’ve just started this run, and we’ll see what we learn from it

Thanks for the update, curious to see how it will run

And we'll learn something from this either way. If it's HPC related I doubt this will be the last time we run into this.

No worries, I’m confident that with enough understanding of your compute / samples we’ll be able to get this running for you as well!

Back in office. Eva, did you make any progress before the holidays?

Hi! Sorry, had long holidays but I’m back to the office now. I’m checking the results and Redux took around 15hours realtime for the tumor to complete with the following parameters:

withName: 'REDUX' {
    cpus = 16
    memory = 72.GB
  }

I see you use a lot more memory but we are limited by memory so we have to set it lower.

Might be hitting memory bottlenecks and start swapping. Do you have any 'public' sample -- SEQC-II, any synthetic controls that you could share? I don't know how REDUX is set up and whether it might be better to have fewer cores with more memory per core.

It might be something like that. I started a run yesterday, with the same memory but 32 cpus instead and redux took way longer (10hours for the normal sample vs 3h with 16 cpus). I’m running another one now with higher memory and 16cpus. If that one takes shorter swapping might be the issue.

Any and all log files would help.

Eva shared a BAM with us @Oliver Hofmann @Stephen Watts

@Rayan Hassaïne I believe that’s the case. Is that relevant?

potentially, depending on that - the cluster might be splitting resources differently than what you requested

increasing the memory to 120GB did reduce a bit the time that redux takes to 2h 15 minutes (normal sample). It is still double as long as it takes for you there

Just want to make sure we're not duplicating work that Team Netherlands are already tackling (ie Rayan) -- do we have a copy of the nextflow logs to explore?

I haven’t shared them for this run. It was a different one. I’m testing a couple of more things here

Okay!

Ok, neither running redux standalone (outside of nextflow), nor lowering the memory given to java with respect of the one given in the sbatch made it run faster

Yep, happy to explore -- we can coordinate with Rayan and Joep on this.

I only see this now, I was reading in diagonal and only read about REDUX here. I posted a related experience with ESVEE in OA 3.0 just below: https://nfcore.slack.com/archives/C04CQ1LK7U1/p1771334794577719

Hi we fixed the issue with redux by setting max_fastq_records=0. The issue happens when there are too much fastq file splitting generating too many bam files. It seems that the memory needed by redux steeply increases with the number of BAMs. However we see the same issue with ESVEE, but we haven’t had the time to properly troubleshoot yet

Ah, I use max_fastq_records=-1 for no fastq splitting. Redux took 1.5h and 18GB RAM with 8 CPU for the tumor sample. I did not repeat for OA 2.1, started directly from this Redux bam file. My issue is ESVEE indeed.