Hi all, I’m currently setting up nf-core/eager on our HPC cluster (Apollo at CBiB, Bordeaux, France) and had a question regarding disk usage. We are working on ancient DNA data, mainly human (but also including some animal, plant, and pathogen samples). Our sequencing batches can include a mix of capture data and low-coverage genomes processed together on the same run. Each batch typically corresponds to ~100–150 GB of raw FASTQ data. While testing eager, we noticed that the

work/

directory becomes very large during execution (several hundreds of GB to Tb). I understand this is due to intermediate BAM files and Nextflow caching, but I wanted to check: • Is this scale of disk usage (e.g. 1 TB for ~150 GB input) expected? • Do you have a rough estimate or rule of thumb for the size of the work directory depending on input data and typical parameters? We’re currently discussing storage solutions with our HPC admins, so any guidance would be very helpful. Thanks a lot for your help and for the pipeline! Best,

Hi Fanny, Yes, unfortunately it does run quite large, and 1 TB could be expected, particularly in cases when you have high coverage date (being duplicated across each BAM file). The pipeline is just simply quite complex with many steps. v3 of the pipeline (which is a whole re-write) should handle this better, which you could test on

-r dev

but it is still months away from formal release. Are you already using the

cleanup = true

option in a custom config? This should remove all intermediate files in

work

on the completion of a successful run (I think that was already around in Nextflow v22.10)

Hi James, Thanks a lot for your quick reply, that’s really helpful! Good to know that this scale (~1 TB for ~150 GB input) is expected. That matches what we’re observing. We’re not currently using

cleanup = true

, so that’s definitely something we’ll test. Thanks also for the suggestion about splitting the runs into two steps. We might explore that depending on how things go with our HPC setup. Thanks again for your help! Best,

One thing that helped me when running nf-core/eager on a system with limit hard drive space was to restrict which files will be copied to the

results

directory. While this won't limit the size of

work

, it helped me to reduce the amount of duplicated intermediate files that I wasn't interested in. Particularly, I got rid of FastQ files by changing the files published by

FastP

and `AdapterRemoval`:

withName: fastp {

    publishDir = [
        path: { "${params.outdir}/FastP" },
        mode: 'copy',
        pattern: '*.json'
    ]

  }

  withName: adapter_removal {

    publishDir = [
        path: { "${params.outdir}/adapterremoval" },
        mode: 'copy',
        pattern: 'output/*.settings'
    ]

  }

Hi Alex, thanks so much for sharing. That sounds really helpful!