I’m running nf-core/rnaseq 3.22.2 on a Slurm-based HPC with Singularity, where compute nodes have fast node-local scratch (/localscratch) and both home and shared scratch (/zfshomes, /sanscratch) are NFS/ZFS-backed. The core issue I’m seeing is intermittent Input/output (EIO) errors on file close, especially during large write or publish steps. Typical failures occur in early genome preparation steps such as PREPARE_GENOME:GUNZIP_FASTA and during later publish of large outputs (STAR indices, BAMs). The error is often something like cp: failed to close <file>: Input/output error, which suggests the write succeeds but the filesystem errors on close/flush. Related to this, I am observing silently corrupted BAM files produced by otherwise successful alignment steps. These BAMs have readable headers and will sometimes even load in IGV, but samtools quickcheck consistently reports a missing EOF block and samtools flagstat fails with BGZF read errors. This strongly suggests truncation during stage-out or publish rather than a problem with STAR itself. When I generate BAMs manually on node-local scratch and stage them out once, they are valid. A key observation is that concurrency strongly correlates with failure. When multiple nf-core processes finish and publish large files at the same time, EIO errors and BAM corruption become much more likely. This makes the issue look like a filesystem stress/locking problem rather than a pipeline logic error. As a control, I ran STAR genomeGenerate manually on a single node with all I/O confined to /localscratch, then staged the result to shared storage once. That completed cleanly and produced a reusable ~26 GB index with no corruption. Using this index via --star_index avoids genome generation inside the pipeline and removes one major failure point. To mitigate the issue, I’m currently running with a custom Nextflow config: executor { queueSize = 10 } process { executor = 'slurm' queue = 'mw128' resourceLimits = [ cpus: 24, memory: 100.GB, time: 48.h ] beforeScript = ''' source ~/.profile module load singularity '''.stripIndent() } singularity { enabled = true autoMounts = true cacheDir = '/zfshomes/aparikh02/singularity_cache' } My params file supplies explicit --fasta, --gtf, and --star_index, and disables iGenomes. I’m happy to paste params if useful, but wanted to first ask at a higher level. My question for the nf-core team is whether there is recommended guidance for safely publishing large files on NFS-backed HPCs, especially under concurrent workloads. In particular, is there a preferred stageOutMode or pattern for environments where filesystem close() under load can return EIO? And are there known rnaseq defaults that assume stronger filesystem guarantees than some HPCs provide? Or is there another issue that i'm not even thinking of related to my parameters? Even if i run it on local execution on a single node, the BAM's are coming back (with no errors during the run and the QC metrics and counts looking fine) with the same EOF error. Thanks in advance for any help.

are you familiar with/ are you using

scratch

? https://www.nextflow.io/docs/latest/reference/process.html#scratch

Absolutely - I use scratch = true on the processes, but somehow that doesn't fix the problem. Issue I’m running into is that it'll complete on a local scratch node and then when it tries to send it back into the network available working directory or outdir it closes prematurely. I’m wondering if the working directory has to be available from all compute nodes?

It's sounding like something related to your cluster storage config, or the way it interacts with Nextflow, you might need input from your admins. I haven't used HPC for a few years, so maybe other folks here have more info, or maybe ask on the #infrastructure-hpc channel of the Nextflow Slack, since it doesn't sound rnaseq-specific

sweet - thank you, i'll shoot that channel a message

Does the

scratch

point to

/localscratch

? (You could specifically set it to that.) You could also try without it, in case the problem is exacerbated when copying the results from the scratch dir to the work and output directories. But in the end this a file system problem that your sysadmins need to solve ... 🤞

haha i've tried that! yeah i'm in ontact wiht my sys admins so hopefully something works out. feel like i've tried all of the things