Hi everyone. I hope this is ok to ask here! I’m trying to run the pipeline on a fairly large dataset and I’m having a bit of trouble. I’ve subsetted the data down from approx. 80 million reads to 6 million, and the pipeline is still taking a long time (around 20 hours). Does this seem normal? I’ve updated the config file to try and increase the memory and time allowed for some tasks. When I tried with the full dataset the pigz uncompress step would always run out of time. Thanks so much for any help with this. Currently I am using the following resources (on the Dardel HPC in Stockholm) on an 11GB gzipped FASTQ file: • 256GB RAM • 80 CPUs (can’t control this, upper limit set by RAM requested) • 20 hours

process
{
    withName: '.*:.*FASTQC.*'
    {
       cpus = 80
       memory = '200.GB'
    }
}
 
process
{
    withName: '.*:BLAZE'
    {
       cpus = 80
       memory = '200.GB'
    }
}
 
process
{
    withName: '.*:TAG_BARCODES'
    {
       cpus = 80
       memory = '200.GB'
    }
}
 
process
{
    withName: '.*:SAMTOOLS_SORT'
    {
       cpus = 80
       memory = '200.GB'
    }
}

process
{
    withName: '.*:MINIMAP2_ALIGN'
    {
       cpus = 80
       memory = '200.GB'
    }
}
 
process
{
    withName: '.*:ISOQUANT'
    {
       cpus = 80
       memory = '200.GB'
    }
}

process
{
    withName: '.*:PIGZ_UNCOMPRESS'
    {
       cpus = 80
       time = '15.h'
       memory = '200.GB'
    }
}

Hey @Josie Gleeson - thanks for trying out our pipeline. Here are a few follow-ups. 1. is 80 M reads the full / not downsample dataset? 2. And no for 6M this is not normal. Could you send us here your params and also at what step is the pipeline at? (to assess what the blocker is). In terms pipeline speed / optimization. We have provided the following to aid in processing time (which are user options, not enabled by default): 1.

split_amount

param (see https://nf-co.re/scnanoseq/1.1.0#troubleshooting). Briefly this will split your fastq files to aid in parallel processing. One downside here is that it will also increase the temporary storage while the pipeline is running (we are currently evaluating other approaches to reduce storage). But if you can I would certainly recommend enabling this parameter (e.g.:

split_amount: 500000

). Once the pipeline is finished, you can safely delete the temporary files (from the

work

dir). 2. In the latest version of the pipeline we have enabled a process which compiles some QC information into MultiQC from NanoPlot. In the past this used to be fetched only from FastQC. Thus, if you want to speed this up a bit while not losing QC information, I recommend to skip FastQC (

skip_fastqc

). 3. We also recommend to

skip_bam_nanocomp

if you are experiencing delays or high memory usage here. We will likely deprecate this one in future releases since we have added several other layers of complimentary QC and nanocomp (specifically for BAMs, not so much for FASTQs), is not very efficient. 4. Lastly - the pipeline provides 2 quantifiers. IsoQuant and oarfish. oarfish is significantly faster than IsoQuant (we have communicated the speed caveats to the IsoQuant authors - they continue to work on optimizing this and we have done some levels of optimization within the pipeline as well). So in the end, if your blocker is at the IsoQuant step, you could run the pipeline with just oarfish. a. That being said this comes at the cost not having the isoquant’s gene level matrices since oarfish only provides transcript-level matrices. I have posted an issue in oarfish (https://github.com/COMBINE-lab/oarfish/issues/41) asking the authors for feedback on how we can implement gene-level matrices. I have some ideas but would like a response first ideally before making this into an official release. All that to say, that I would recommend following steps 1-3 above, and leaving 4 as a last resort. IsoQuant should be able to still process 80M reads (we have some benchmarks in our preprint with read levels beyond this range: https://www.biorxiv.org/content/10.1101/2025.04.08.647887v1.full.pdf). Let me know if this works for you. We will be happy to provide additional guidance and support.

Hi Lara, thanks so much for your help with this. I have replied with a DM and some files.

In my case, I have ~150M reads, one sample, and it took ~2 days. Fastqc step was taking lots of time (seems it only used one cpu on one sample), it retried several times due to reaching the time limit.

@Henry Qian: yes I would certainly recommend to go ahead and skip FastQC. @Austyn Trull we might want to consider disabling FastQC at this point I think by default (I know we discussed keeping it up to users, but I think we are recommending skipping enough where I believe we should set a default). Possibly the same to nanocomp_bam as well.

@Lara Ianov Yes, I increased the resources according to the execution report. IsoQuant is fine I think (since I see it's manually paralleled in scnanoseq), taking ~8 hrs at maximum. While FASTQC is taking long (if it fails and retries,) I think the information it provides is quite useful. Increasing the default time limit can help reducing the chance of rerunning and thus saving time.

@Henry Qian ok that's great to know for when I run it on the full sample! thanks

Yep that is correct. If you have the resources and time increasing is indeed the solution. We provide alternatives to give options for everyone 🙂

Yep and I'll also add that we do have alternative QC tools in the pipeline that provide similar plots/information to FastQC such as ToulligQC and Nanocomp(FASTQ)/Nanoplot. So if FastQC does end up taking too long, it should be able to be safely skipped and you'll still be able to review similar information in different reports