https://nf-co.re logo
Join Slack
Powered by
Yes, Java error is gone, but now there's something...
# helpdesk-hpc-sweden
j
Yes, Java error is gone, but now there's something else wrong that was fine before: 
Command error:
INFO:  Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
INFO:  Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
INFO:  Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
INFO:  Environment variable SINGULARITYENV_SNIC_TMP is set, but APPTAINERENV_SNIC_TMP is preferred
/bin/bash: line 9: .command.run: Permission denied
Guessing that I need to change my script then, how exactly?
Current version of script
Current version of script
p
Now you're just having bad luck, here's a ticket for free help! Seriously, though, I don't have a theory for how you get that error yet, but it would surprise me a lot if it was due to something you did.
😅 1
• Can you show the permissions in the work directory where you got that error? • Do you get it consistently (nextflow has the 
-resume
flag which can be really useful to get a run through despite random failures)?
j
I've had it several times already today, when I tried to run it via pixi, I got the same error. I thought it was because of something with pixi and wouldn't come up if I don't use pixi. Evidently I was wrong. Yes, there's something wrong with permissions, it seems.
p
At the same time, you get singularity/apptainer launched which implies it's not something with actual permissions...
But I don't have a theory for why it would be broken suddenly (you're not running from a new place, e.g. different project space)?
j
Permissions for the folders used: 
drwxrwsr-x
No, same folder as before
On a most likely completely unrelated issue, on my local machine (MacOS) I'm also suddenly getting weird permissions issues. Same software I ran successfully last week, same folder, same input script, just trying to replicate last week's run. Same on a different computer running a different (older) version of MacOS. Very very strange to me. I'm just putting this here in case you have an idea, but of course this is not HPC related, so totally fine if you don't.
Untitled
Untitled
p
There you have a segmentation fault at least which is easier to explain as a random fluke (or if you get it consistently; Apple doing something that broke that).
Can we get a 
ls -la
from a problematic work directory? It should list 
.command.run
, `.command.sh`and so on.
j
Yeah that's what I thought, but what could they have done since I got the same problem on a machine that hasn't been updated for ages...? The program has not been updated, the plugin it has to call is still the same.
🤷 1
[julgr393@rackham1 griese_xj-4128]$ cd work/f6/5b5ad90eef02f293aa451577f03c20
[julgr393@rackham1 5b5ad90eef02f293aa451577f03c20]$ ls -la
total 40
drwxrwsr-x 2 julgr393 p_griese_xj-4128 4096 May 13 16:31 .
drwxrwsr-x 3 julgr393 p_griese_xj-4128 4096 May 13 16:31 ..
-rw-rw-r-- 1 julgr393 p_griese_xj-4128   0 May 13 16:31 .command.begin
-rw-rw-r-- 1 julgr393 p_griese_xj-4128  534 May 13 16:31 .command.err
-rw-rw-r-- 1 julgr393 p_griese_xj-4128  598 May 13 16:31 .command.log
-rw-rw-r-- 1 julgr393 p_griese_xj-4128   0 May 13 16:31 .command.out
-rw-rw-r-- 1 julgr393 p_griese_xj-4128 12434 May 13 16:31 .command.run
-rw-rw-r-- 1 julgr393 p_griese_xj-4128  867 May 13 16:31 .command.sh
-rw-rw-r-- 1 julgr393 p_griese_xj-4128   3 May 13 16:31 .exitcode
That's the directory that got .command.run Permission denied
p
Ok, something weird going on.
j
😅 It has been a very frustrating day computer-wise. I'm trying to teach a student how to process crystallography data and then suddenly nothing works. And no luck with the HPC either.
p
At least your student gets rid of some illusions of life as a researcher, or something. Without a better theory of why/how things break, my best suggestion is to try rerunning :(
😅 1
j
Alright, will try again tomorrow. Thanks for your help! 🙂
I'm still getting the same error today. I removed the .pixi directory in case that messed things up somehow, but that doesn't change anything.
Untitled
-- Check script '/proj/griese_xj-4128/assets/nf-core/rnaseq/subworkflows/nf-core/utils_nfcore_pipeline/main.nf' at line: 97 or see '.nextflow.log' file for more details
// Get software versions for pipeline
//
def processVersionsFromYAML(yaml_file) {
def yaml = new org.yaml.snakeyaml.Yaml()
def versions = yaml.load(yaml_file).collectEntries { k, v -> [k.tokenize(':')[-1], v] }
return yaml.dumpAsMap(versions).trim()
That's what this script says at line 97 (the second to last line here is line 97).
p
I think the earlier error will cause others that aren't really worth investigating. @Pavlin Mitev No bright idea? Things I believe; • The file is accessible/readable, slurm manages to start the job far enough that singularity is launched. • It's not in home, so nextflow changes not binding home by default should not matter • It's happening in the container so it relies on apptainer binding correctly Things to check: • Is this all on the same node or does it happen on any node? • Is it always the same task? Does the same happen if one goes there, opens a shell with the container image and runs 
.command.sh
p
I will be back in the office tomorrow from the current meeting. Is there a way for me to recreate the setup, so I can reproduce the error?
j
I guess, what do you need from me?
p
Tutorial or documentation I can follow or complete setup - I might not have access to your files on Racham.
j
Can I give you access to my files?
p
Things to check possibly? 
sacct -u YOURUSERNAME
should show that if I remember correctly. Always the same ask - check the logs for recent failures and see if it is the same process that fails. Possibly change to the one of the work directories where it failed, 
apptainer shell
the corresponding image (e.g. 
/proj/griese_xj-4128/SINGULARITY_CACHEDIR/depot.galaxyproject.org-singularity-fq-0.12.0--h9ee0642_0.img
) and execute 
bash .command.sh
Especially the last bit is a bit complex.
Alternatively, what was this? rnaseq 3.18?
j
Yes, 3.18.0
If you meant that 
sacct -u
should show which nodes were used, it does not. I just ran 2 more attempts just to see and they were both on r35.
The latest 2 failures indicate that it is not always the same process that fails.
Untitled
p
Ok, so both for you and for Pavlin a 
nextflow run nf-core/rnaseq -r 3.18.0 -profile uppmax,test --outdir somedir
might be instructive (possibly 
test_full
instead).
j
I assume you have considered that this could be something really stupid that I'm doing because I don't really know what I'm doing?
p
Got the error, let me see how much I can see from the train
👍 1
p
🎆 And no, I don' see any way this could be user error for the moment.
😅 1
p
This is stupid... There is a line that calls python. And this is python2 or no python if one loads python from the modules
p
Should be container images all the way, so I suspect that's a red herring. (If the 
container
bit in the process is missing, and it's relying on the installed stuff because of that that's another thing.)
p
Can't paste - have slack only on the phone
Seems .command.run is a sbatchscript but it is eun by bash instead of slurm
Nope, same if i submit it
p
There are two scripts (.sh which is the actual script snippet from the process and .run which is templated by nextflow.
When running with a container engine it will relaunch the run script inside the container, and it will then run the .sh
But here we seem to get a weird failure when we launch .run inside the container.
p
Sure
p
If it was only one it might maybe have been a bad image with 
bash
broken, but I don't think that would go undetected for multiple.
Possibly also Apptainer misbehaving but I don't have a sensible theory of why that would be.
p
For some reason, 9 line which is an environment variable definition is considered as command , not wrong, but gets permission denied...
IMG_20250514_160859.jpg
j
Yes, that's what I've been seeing
p
Probably doesn't hurt testing through different nextflow versions even if I'm not sure that would matter.
p
Will continue tomorrow
👍 1
p
👍 I'm not sure I understand the error and if it's trustworthy or an effect of parsing or similar.
Looks good enough to me.
a
Hi! Don't know if it helps, but I just wanted to report I've been experiencing a similar issue on Miarka since the last maintenance: I'm also getting permission denied on the 
NXF_CHDIR
line in 
.command.run
. It works fine with 
NXF_VER=24.10.6
though I've sent a ticket to support@uppmax.uu.se with the details
👍 1
p
Just got another user with the same error with another pipeline
The user says that it runs with version 24.10.6...
Hi Adrien 😌
👋 1
Manually triggered an update, so it got to version 25.04.2, can one try it now?
Nope same ... There are 3 releases last 3 days...
Well 2 days
m
More people are reporting that issue: https://github.com/nf-core/rnaseq/issues/1554
What I find intriguing in that error message
Copy code
WARN: Directive `process.shell` cannot contain new-line characters - offending value: [bash
Thus the issue might be related to https://github.com/nextflow-io/nextflow/issues/5686
If that is really the problem, creating a custom config file to overrule the default config of the 
process.shell
shipping with the pipeline template could fix it, I presume. Basically create a custom config 
nextflow run -c process_shell_fix.conf
… Inside that custom config, this fixed definition
Copy code
// Set bash options
process.shell = [
    "bash",
    "-C",         // No clobber - prevent output redirection from overwriting files.
    "-e",         // Exit if a tool returns a non-zero status/exit code
    "-u",         // Treat unset variables and parameters as an error
    "-o",         // Returns the status of the last command to exit..
    "pipefail"    //   ..with a non-zero status or zero if all successfully execute
]
would override the faulty definition in the pipeline template. But that is only an assumption and untested so far (because I lack a good testing environment at the moment)
j
Well, I'm kind of glad it's not just me then 😅 I'll look into trying this fix tomorrow. Thanks everyone! 🙂
p
But for my understanding; Julia got hit by this because there was an upgrade that wasn't noticed and going back to earlier versions should be the fastest resolution to get runs through?
p
Indeed, it works (so far) for me as well with version 24.10.6. I will have meeting today with colleagues from other HPC centers, but they probably have not seen this issue yet. For good or bad, they are not upgrading as fast as we do on UPPMAX. I will play around but for now perhaps we need to uninstall the 25 release from the module system, before we get flooded by this problem....
Copy code
module load bioinfo-tools
module load Nextflow/latest
export NXF_VER=24.10.6
cd /proj/XXXXX
nextflow run nf-core/rnaseq -r 3.18.0 -profile uppmax,test_full --outdir output --project XXXXX
nod 2
m
But for my understanding; Julia got hit by this because there was an upgrade that wasn't noticed and going back to earlier versions should be the fastest resolution to get runs through?
Yes, downgrading the Nextflow version should do the trick as well. But it feels strange to stay: Hey, all nf-core pipelines of 2024 are incompatible with the latest Nextflow, even though it seems that is exactly what happened.
p
Yep, just before I (or somebody else) figure out what is going wrong. Rackham is running still CentOS 7 - not best OS to debug ;-( I will try to reproduce the problem on another center to narrow down the cause...
m
If it really has to do with that `Directive 
process.shell
cannot contain new-line characters warning` that I pointed out, then it would be due to an unfortunate config entry that was included in the nf-core pipeline template for a while. Some pipeline authors fixed it manually before the change was rolled out in the template, because a CI test failed, but we must assume that most nf-core pipeline versions released in 2024 and early 2025 are shipping with that faulty 
process.shell
config entry that breaks with Nextflow 25 versions, because they ignored that the Nextflow latest CI test was failing at the time. See the pretty comprehensive discussion around the fixing PR here: https://github.com/nf-core/tools/pull/3416
p
For nf-core people, it's strange, but not the first time nextflow changes has broken stuff, probably not the last. For researchers; having a recipe (
export NXF_VER=24.10.6
) probably makes them a lot happier than troubleshooting stuff. But @Julia Griese might be a beter judge of that 🙂
🤩 1
😅 1
But yes, later on, of course one question is if this could this have been managed better (detected through nextflow nightlies) - I do think there are CI for that already. For users, this feels like a typical "good to specify the version used". For sites, changing default versions can break stuff for users.
p
Same error on Dardel@PDC if I upgrade to the latest nextflow. Indeed, it seems to be the issue that @Matthias Zepper mentions above.
I will bring this to the other centers for "discussion"... in today's meeting.
p
Long thread, I confess that I haven't read all 76 replies 👀 - but from my understanding this post on #CE6P95170 should explain what's going on: https://nfcore.slack.com/archives/CE6P95170/p1747302440656339
gratitude thank you 2
🙏 1
j
Dare I say it... It looks like the pipeline is running now (with 
export NXF_VER=24.10.6
) ☺️ Thank you!
p
I got the test_full completed on Rackham with the version "fix".
p
Great! 🎉
Note that this config fix should also work if you want the goodies in the latest Nextflow version
2 Views
Open in Slack
PreviousNext
Powered by