https://nf-co.re logo
Join Slack
Powered by
Hi all, I have a question about the pipeline. Is t...
# airrflow
s
Hi all, I have a question about the pipeline. Is the qc-filtering step particularly slow? It takes me a lot of time every time I run it, but it doesn't seem to take up a lot of computing resources.
I used the test procedure and got the following log file: 136 2a/d65405 434314 NFCORE_AIRRFLOWAIRRFLOWREPERTOIRE_ANALYSIS_REPORTING:PARSE_LOGS (logs) COMPLETED 0 2026-04-09 110428.847 2.5s 1s 2244.8% 11.3 MB 33.7 MB 10.1 MB 9 KB 130 44/addd43 433895 NFCORE_AIRRFLOWAIRRFLOWVDJ_ANNOTATION:FILTER_QUALITY (Sample2) FAILED - 2026-04-09 110427.482 1h 1h - - - - - 131 43/79ddf0 434006 NFCORE_AIRRFLOWAIRRFLOWVDJ_ANNOTATION:FILTER_QUALITY (Sample7) FAILED - 2026-04-09 110427.534 1h 1h - - - - - 132 5d/a4ea63 434069 NFCORE_AIRRFLOWAIRRFLOWVDJ_ANNOTATION:FILTER_QUALITY (Sample3) FAILED - 2026-04-09 110427.642 1h 1h - - - - - 133 6e/e9720f 434128 NFCORE_AIRRFLOWAIRRFLOWVDJ_ANNOTATION:FILTER_QUALITY (Sample4) FAILED - 2026-04-09 110427.726 1h 1h - - - - - 134 77/2de385 434198 NFCORE_AIRRFLOWAIRRFLOWVDJ_ANNOTATION:FILTER_QUALITY (Sample8) FAILED - 2026-04-09 110427.839 1h 1h - - - - - 135 e3/7ed3a0 434277 NFCORE_AIRRFLOWAIRRFLOWVDJ_ANNOTATION:FILTER_QUALITY (Sample6) FAILED - 2026-04-09 110428.139 1h 1h - - - - -
Where can I find the source code for this process? I want to profile the code to understand why it's taking so long to execute.
In addition, for downstream analysis using other specialized tools, which output files would you recommend? Should I use the AIRR-compliant TSV files located in the 
qc-filtering
directory?
h
Hi, Suchen. Sorry for the late reply. Which qc-filtering step did you talk about? We have several different qc steps in our nf-core/airrflow pipeline. According to the log file you provided, it seems that you talked about "VDJ_ANNOTATION:FILTER_QUALITY". Actually, this step is supposed to be quick.
I don't know what downstream analysis you want to do. If you need the sequences for further analysis, use Airr format tsv file. The one located in qc-filtering is before clonal analysis, so it does not have clone related info. The one located in clonal_analysis has clone id and mutation frequency. BTW, if qc-filtering you mentioned before is this specific step, it will be slower than VDJ_ANNOTATION:FILTER_QUALITY but usually not noticeably slow. In addition, qc-filtering has two different versions based on whether your data is bulk or single cell. Could you please provide more info about your data and share your command and configuration file?
s
@Huimin Lyu Hi, thanks for your response. Apologies for the previous lack of clarity. I have a few follow-up questions regarding my project: First, I am using the pipeline for single-cell BCR data, but the default reports are a bit too general for my needs. Specifically, I want to quantify BCR sharing across different clusters, which likely requires downstream analysis. Based on your feedback, I assume the TSV files in 
clonal_analysis/define_clones
are the correct source for this. Is that right? Also, should I merge these individual TSV files into a single dataset for cross-sample analysis? Second, While running 
airrflow (4.3.0)
, I noticed that certain steps are extremely time-consuming (as shown in the attached file). Is this expected behavior for datasets of this scale, or is there a way to optimize the performance? Third, I also tried upgrading to 
airrflow (5.0.0)
last week but encountered a known error that has been mentioned by other users. I will @ you in that specific issue thread with the error logs for better tracking. Could you please help me look into a workaround? Thanks again for your time and guidance!
execution_report_2025-10-10_18-01-29.html
h
Hi, Suchen. If you want to check BCR clones shared across different samples, you should start from Airr format tsv file located under 
clonal_analysis/define_clones.
However, you should be aware that clonal inference by default is at subject level. If your samples are from the same subject, the same clone_id means of the same clone. However, if your samples are from different subjects, the clones are assigned separately and there's no way to compare them.
As for the time-consuming steps in your case, the main bottlenecks are define_clonal_threshold and define_clones_report. Both steps aggregate sequences across all samples and involve computationally intensive calculations. Because you have a large number of samples, these steps take significantly more time to complete.
s
@Huimin Lyu Thank you. Also, would it be possible to access the source code for these processing steps? Having a look at the code would help me better determine which intermediate files are most suitable for my follow-up analysis.
h
All our source code is on Github (https://github.com/nf-core/airrflow). However, due to the complexity of Nextflow structure, it may be not easy to locate the code straightforwardly. If you want to have a try, you can start from workflows which will direct you to corresponding subworkflows and then redirect you to modules which points to the real code for calculation with immcantation tools, enchantr and others (all in another github, https://github.com/immcantation). It needs some effort to put all of these pieces together to understand the generation of each file.
s
Thank you! I will have a try. @Huimin Lyu
Open in Slack
PreviousNext
Powered by