Hi all, Some more questions about pipeline performance improvements: 1. During the UMI Build Consensus step, I notice that core usage is low (average load of 2 cores) despite setting high-core usage for this step using config files and the parameter being forwarded to the python job (--nproc 64). At the start of this step, I see that all requested cores are used, but after a brief interval only 1/2 cores remain in use. Why is that? How can this be improved? Runs of large datasets are failing due to reaching time limits, if this step could be sped up it would make a big difference for me. 2. The "argument is of length zero" error of the Collapse Duplicates step is still causing failures in v. 5.0.0 for some of my samples. The workaround I used in version 4.3.0 does not work anymore so I am stuck. Any idea when this issue might be solved? Cheers, Thomas-Wolf

Hi Thomas, can you provide the error message or the link to the originally reported issue with the Collapse Duplicates step? What workaround did you use for version 4.3.0?

Hi Gisela, The link is https://github.com/nf-core/airrflow/issues/420 The workaround was as described in the thread. I emulated the Collapse Duplicates step outside of the pipeline to identify which collapse groups caused errors, then just removed the sequences of that group from the _meta-pass.tsv table and changed the log files to reflect the altered sequence counts. Now, the script can't identify the failing groups anymore. Not sure why that is. Regarding the build consensus step: I have attached screenshots from the node where a BuildConsensus step was being run at the start with full core usage and some time later when the usage dropped to ~2 cores

ok if the tool started using all CPUS and ended up using just 2, this is explained by having 2 very big consensus groups that can't be processed in parallel that take most of the time. Not sure if you can confirm this from some of the samples that took a long time. Do you have lots of reads with the same UMI barcode before the build consensus step? Are you using the approach to cluster by similarity before build consensus in case your UMI diversity is not sufficient, in this case?

Hi, @Thomas-Wolf Verdonckt I fixed the bug in the Alakazam collapseDuplicates function which failed at the boundary condition that only one sequence remains after removing ambiguous sequences in Oct 2025. Now the new change is available in Alakazam 1.4.2. However, nf-core/airrflow has not used the latest Alakazam version yet. We are preparing for a new nf-core/airrflow release in March or April that will use the latest Immcantation packages including Alakazam 1.4.2. If you want to try the nf-core/airrflow before the formal release, I will inform you once I have things ready in our nf-core/airrflow dev branch.

Hey @Huimin Lyu, any updates on this issue? I have a couple samples that I am waiting to run. Regards

Hi, @Thomas-Wolf Verdonckt We are still working on the new release. As we are adding a brand-new feature to the pipeline in this new release, it took longer time than we expected. We are almost there. We want to have new release as soon as possible this month.

@Huimin Lyu thanks for the update. In the meantime, there is another issue with the collapse_duplicates step that is holding me back, which should be addressed: When processing a single sample with 5 million rows after the meta_pass step, using a single core, the pipeline uses more than 512 Gb RAM, which causes it to fail on my setup. Please adapt the step to be more resource friendly.

Hi @Thomas-Wolf Verdonckt. Yes, collapse_duplicates needs large memory if the sample size is large, as all sequences in one sample will be read in to find duplicated sequences. I am not sure what do you mean by saying "resource friendly". If you want to customize the resources needed in this step, it's easy. You just need to set the resource you want in a configuration file (e.g. resource.config) and added

-c resource.config

to your command. Please check this webpage to see how to customize configuration: https://nf-co.re/airrflow/5.0.0/docs/usage/FAQ. However, if you want us to make this step to use less memory in general, it's more complicated and we need to think about a way to achieve it. Welcome any suggestion from you.

Hey @Huimin Lyu, I already set

process {
  withName: '.*COLLAPSE_DUPLICATES.*' {
    cpus   = 1
    memory = 170.GB
  }
}

But that did not prevent the pipeline from reaching >512 Gb during the collapse step. Some things off the top of my head: are only required columns loaded during the collapse step? or is the entire dataset loaded into memory? Am I right that during processing of one collapse group ID, the full parent table is typically still in memory (plus grouped/intermediate structures), not just that one group? If so, this could be changed.

First of all, setting resources for each step in the configuration file does not limit the actual resource usage. On the contrary, it limits the available resources for the specific step. By default, 'COLLAPSE_DUPLICATES' requires 16 CPUs and 72 GB memory for the first try and double the CPUs and memory for the second try if the first fails. However, if there's sample of large size which cannot be processed with only 144 GB, we need to set it manually. In your case, probably set it to be 600GB (if your computational infrastructure has enough resources.) Meanwhile, you should also modify

resourceLimits = [cpus: 16, memory: 144.GB, time: 24.h]

accordingly as this set the cap for resources for all steps.

@Huimin Lyu I had already set:

process {
  cpus = 9
  time = '31d'
  withLabel: process_low               { cpus = 9; time = '31d' }
  withLabel: process_medium            { cpus = 9; time = '31d' }
  withLabel: process_high              { cpus = 9; time = '31d' }
  withLabel: process_long_parallelized { cpus = 9; time = '31d' }
  withName: '.*' {
    cpus = 9
    time = '31d'
  }
  resourceLimits = [
    cpus: 9,
    time: '31d'
  ]
}

To try and override all existing limits, as the pipeline almost always takes more than a full day to run, in some cases more than a week (!). I only have access to 512 Gb nodes. So the 512 Gb is a hard limit. I set 1 cpu for the collapse step because any parallelization seems to increase memory usage, causing the pipeline to fail earlier. Setting soft limits is not the issue. I have been trying to overcome these limitations for over a year by now. I think the solution will have to come from software optimization.

Hi @Thomas-Wolf Verdonckt, thank you for sharing. This profile does not seem to be changing the memory limits, though. How are you handling those?

Hey @Gisela Gabernet indeed, for the other steps there are no issues, it's only collapse_duplicates that exceeds the available RAM, and for that I set a process-specific limit (170 Gb).

process {
  withName: '.*COLLAPSE_DUPLICATES.*' {
    cpus   = 1
    memory = 170.GB
  }
}

But that does not seem to have effect (possibly because I am not running through SLURM for this sample). I know that the config is loaded because the process is limited to 1 cpu as requested. Furthermore, unless I am mistaken, setting a memory limit wouldn't really help given that there is no parallel processing. The physical limit is reached by a single task. It would probably just fail 'cleanly'.

Even if you are not running through SLURM, the memory limit should be set. Do you see that 170.GB are requested for this process?

Mine is likely an edge case. I am processing clinical samples of patients with plasmablast hyperexpansion during severe disease. Out of a couple hundred samples, two are currently triggering this OOM issue. The samples were sequenced with a pooling strategy that aims to allocate a consistent read/UMI ratio per sample. The largest sample was assigned 28M reads, which resulted in 5 million sequences in the meta_pass table.

Hi Thomas, just getting back to this. Having an example dataset to work with would be really helpful to make sure we are tackling the relevant problem

Hi @Gisela Gabernet, please see DMs.

Hi @Thomas-Wolf Verdonckt Thank you for sharing the fastq files with us. I am trying to run the sample with nf-core/airrflow to reproduce your issues. I just have one last question to ask before starting the pipeline, is it human sample or mouse sample?

Hi @Huimin Lyu, it's a human sample.

Thank you.