Hi (again) everyone. I am facing a space issue when running scnanoseq pipeline. I am currently trying to run scnanoseq on 3 samples (totalling to 380 GB of fastq.gz file) and my current nextflow work directory has consumed about 7.5TB so far (without any of the 3 samples finishing unfortunately due it running out of disk space). For context, I am running each sample on separate nextflow runs, so there are some duplications that can be avoided (e.g. genome indexes) though I believe this does not significantly contribute to the issue. When checking nextflow

work

folder, I noticed that the main culprit taking up this space are the uncompressed FASTQ files. I am also using splitting when processing my samples, which further exacerbates the issue as I have multiples of uncompressed, split, concatenated 'plain' FASTQ files, which is taking up a lot of space. I've been going through the mermaid diagram and the workflow nf file to try and understand which steps requires uncompressed FASTQ files and if I understand correctly, I believe Nanofilt is the reason for the decompression step as it require plain FASTQ input, with the remaining steps - Post trim QC and BLAZE - being able to handle both compressed and uncompressed FASTQ file (I know barcode extraction currently only support uncompressed file, but the script should be able to easily be modified to support compressed FASTQ file). Is it possible at all to keep the FASTQ files always compressed when on disk instead of having uncompressed FASTQ files? I understand that Nanofilt works on uncompressed input only, but the Nanofilt mentioned it can work with piping in the uncompressed fastq file and piping out to a gzip stream. (I also noticed that nanofilt is now deprecated in favour of chopper according to their github?) Alternatively, do you have suggestions on how I can minimise disk usage while also keeping the run-time of the pipeline manageable? Should I disable the splitting parameter entirely?

Hi Andrian, for the first point. We do have a PR under review that addresses both items (replacing Nanofilt with Chopper + keeping FASTQs compressed) here: https://github.com/nf-core/scnanoseq/pull/65 The reason we split is because Nanofilt does take a very long time to trim large FASTQs (and is notably not multithreaded, so splitting the file is the way to get around this). It could be worth disabling the splitting parameter to test speed, but I believe it will be long running An alternative could be to trim the FASTQs outside of the pipeline, (potentially using chopper), then use the newly trimmed FASTQs as input to scnanoseq and then enable the

skip_trimming

flag when running. At least until we can get the PR merged, we can discuss prioritizing this PR to at least get it into

dev

branch so it'd be easier to use chopper + reduce storage limitations

Thanks for getting back to me Austyn! I do believe splitting the file is probably a worthwhile step to speed up the pipeline (especially when analysing promethium runs), so I'm fine with accomodating extra storage space for this. However, it would be great if it is possible to reduce the extra storage space as that would allow more samples to be analysed concurrently without blowing our storage budget too much. I saw the PR you linked and I think that it would be really good if we can get the PR for that prioritised as I think it would be useful for others as well.

No worries at all! BLAZE generates a final barcode whitelist that we use to correct barcodes to. In order for that whitelist to be correctly generated, the FASTQ needs to be whole otherwise we'd have a lot of smaller whitelists filled with potential false barcodes or missing barcodes because they'd be spread across multiple FASTQs in too low concentration

I see. That explains the need for number 4 in the list I created above, but I don't think we need number 5, no? Especially if we don't end up using the FASTQ file from BLAZE?

Its for speed, we split the FASTQ and the raw barcode calls from BLAZE to perform barcode+umi extraction.

Sorry, maybe I'm not explaining myself properly, but can't we not use the split file in number 3 for the barcode extraction?

Oh I see, that's a good catch, yes I think you're correct. We could save some time by reusing the outputs, so that could be something we think about if its still needed after the PR is accepted.

I have a couple of ideas on optimising the split 'path' of the pipeline and would be keen to contribute to the pipeline development if the ideas do work out. How can I get involved?

Yeah for sure, typical workflow is to fork the repo, make the changes in the forked repo, submit a PR, and we'll review it

Is there a public 'large' test data I can use for testing the 'split' path? I'm afraid my data is too large for use in testing as it takes at least 1.5 days to process one file with the high CPU/RAM config!

Sure so we've got data you can use that you can invoke by adding the

-profile test

and

-profile test_data

to your nextflow command. They do come with some default settings that you can modify by adding in your own config or params. •

test_full

uses two GridION samples (LSK110 and Q20+) that were published with the BLAZE paper. The

test_full

samplesheet is found here: https://raw.githubusercontent.com/U-BDS/test-datasets/scnanoseq/samplesheet/samplesheet_full.csv if you want to download the FASTQs for offline use •

test

uses the same samples but downsampled. The pipeline will download the FASTQs and use params from the git repository their stored in, but you can also download them for offline use here https://github.com/nf-core/test-datasets/tree/scnanoseq.

Great. Thanks Austyn!

We just talked about that PR - I agree we likely should prioritize this one for a next release along with a few other changes. But until then, feel free to let us know of anything else that we can optimize in terms of storage.

BLAZE_test.sh

We will try the best we can 😁 . But i think there were a few things that weren't working as expected. Will let you know.