Hello, We are having seemingly random issues with ...
# pipeline-maintainers
a
Hello, We are having seemingly random issues with “cannot overwrite existing file” error on processes with a redirection from stdout (see this one for example: https://github.com/nf-core/modules/blob/754825005d1c009e0221aca276aeabf1972ca31b/modules/nf-core/tabix/bgziptabix/main.nf#L26) on nextflow 24.10.0. Has anyone experienced the same? Any idea on what could be going wrong?
We can definitely relate is to https://github.com/nf-core/tools/pull/2991 but it does not explain the randomness of the problem
m
Check none of your inputs match the output name there
a
It does not seem to be the case
and the error is transient
I.e. it goes through on resume
m
what does
input
end in?
a
.vcf
if that is what you are asking
r
It happens in raredisease too. As @Annick Renevey mentioned, the error is transient. I started six runs yesterday, and three of them failed at different tasks. One was bgzip, another was cadd, and another was GATK. However, when I resumed the runs, they all finished successfully.
m
And it's occurring because bash thinks the file already exists in the work directory even though it doesn't?
a
yes, it seems so
m
Can you put the exact error here?
r
Here are two examples
m
OK. So the first example, is actually when it's trying to link the input file, and not from the code, and the second example is writing
versions.yml
which shouldn't exist
What are you running on?
r
Local SLURM cluster
m
Do you know anything about how the storage is setup?
a
How do you conclude the first example is from linking the file? Just for my understanding
m
Ah, my mistake. It's written by the
awk
output. I missed that.
OK. But they're the first file to be written by redirection.
How many processes are running at the same time?
I'm wondering if you're getting process hash collisions
a
Many processes for sure ~ 20 if I had to guess
at the same time
m
That doesn't feel like it would be enough to generate a collision.
a
But there are 1000 processes written up in the work dir
r
A few, but I think its quite unlikely that the process hashes are identical between two processes. Especially for GATK, because at least in the example above I don't think there was another GATK process running at the same time.
m
Can you check the timestamps of all the files in that directory when you next encounter a crash like that. The files written by Nextflow I think will overwrite the older ones, but not the files that the script tries to write ( if it is a task directory collision )
a
The hashed work dirs look clean
No old timestamps and no indication of other processes writing in the same folder
m
Are you using the
scratch
directive?
r
Could it be that in example1: bgzip has gotten hold of the file before awk has finished writing all of the data?
Nope.. no scratch
m
For
ADD_VARCALLER_TO_BED
can you check the
.command.sh
to see if the
awk
command is actually on line 8.
r
Yupe, it is.
Copy code
#!/usr/bin/env bash

set -e # Exit if a tool returns a non-zero status/exit code
set -u # Treat unset variables and parameters as an error
set -o pipefail # Returns the status of the last command to exit with a non-zero status or zero if all successfully execute
set -C # No clobber - prevent output redirection from overwriting files.

awk '{print $1" 0       "$2"    mutect2"}' grch37_homo_sapiens_-d5-.fasta.fai.sizes > mutect2.bed
bgzip --threads 1 -c  mutect2.bed > mutect2.bed.gz
tabix -s 1 -b 2 -e 3 mutect2.bed.gz

cat <<-END_VERSIONS > versions.yml
"NFCORE_RAREDISEASE:RAREDISEASE:CALL_SNV:POSTPROCESS_MT_CALLS:ADD_VARCALLER_TO_BED":
    tabix: $(echo $(tabix -h 2>&1) | sed 's/^.*Version: //; s/ .*$//')
END_VERSIONS
m
Another long shot to test. In the example where the versions.yml triggers the process failure, it would mean the
*.reverted.bam
has already been overwritten. But if it's not a task hash collision, then there might be two copies of that file potentially with the same name and importantly md5sum. Reruns would have different timestamps in the bam so the md5sums would be different, but a bam file that overwrites another processes output would have the same one.
Did you get to the bottom of this? I think it's very important to understand what is going on here, and why these files already exist in a supposedly empty directory
a
We are seeing this also with pipelines that do not have set -C as default, so it is most likely due to our cluster
Current theory is based on SLURM acting out, but difficult to boil it down