Anyone have any ideas on this?

Gemini tells me:

This is a classic Java Virtual Machine (JVM) issue, not a Singularity/Apptainer problem itself. The CI environment is just exposing it.

That error comes from Java. The directory

/tmp/hsperfdata_runner

is where the HotSpot JVM writes performance data to allow external tools (like

jps

or VisualVM) to monitor it.

The problem is that on your CI runner, the

runner

user is likely being re-used for multiple concurrent jobs, or a previous job's process died uncleanly and left a stale, locked file. When your new Java process (with PID 93) starts, it tries to create its file (

/tmp/hsperfdata_runner/93

) but finds the directory is locked by another process (or a ghost of one).

🚀 The Quickest Fix

The easiest and most common solution is to simply tell the JVM not to create these files. You almost never need them in a non-production, non-debugging CI environment.

You do this by passing a flag to your

java

command:

-XX:-UsePerfData

This flag tells the JVM to disable the creation of the

hsperfdata

files entirely, which completely bypasses the problem.

I guess this is something that needs to be exported in the environment so it gets passed to the JVM.

I was about to say that I didn't think any of these components use Java, but SnpEff does apparently

Nextflow itself is running on the JVM

oh, true

Oh, I think I’ve seen that. By default, the way Nextflow runs Singularity, there’s the risk that the pids of the processes run in the task will collide with pids outside of the task (on the host machine). There’s an option somewhere to disable that. Let me search

I /think/ you can also do this directly in Singularity with a flag:

singularity {
    runOptions = "--pid=false"
}

Incidentally, anyone want to actually review the PR that I opened with 😉 ?