https://nf-co.re logo
Join Slack
Powered by
something seems to be of with our self-hosted runn...
# pipeline-maintainers
m
something seems to be of with our self-hosted runners. they are not starting suddenly. investigating now
d
Got the issue in github.com/nf-core/ampliseq/pull/1034 and all other PRs to ampliseq, any solution?
m
Still working on it. A quick workaround is to switch to github runners similar how we did it in modules https://github.com/nf-core/modules/pull/12464
d
I’m not completely sure, but it seems to be the same issue as in bacass: github.com/nf-core/bacass/pull/315. It isn’t triggering nf-test because it’s waiting for a run that is stuck (30523654851)
m
correct. cancelled the blocking run now
🙌 1
d
r
Heads up: this also affects the aws megatests run on release PRs. They cannot be triggered at the moment
🙌🏻 1
p
can anyone help me with this? nf-test doesn't even appear among the checks even after switching to github runners, thanks! github.com/nf-core/rnasplice/pull/238 EDIT: it seems that my previous job on the self-hosted server is keeping it as pending because of concurrency. But it I try to cancel the hold job, it does nothing. github.com/nf-core/rnasplice/…/30693228051 github.com/nf-core/rnasplice/…/30648287094
m
p
@mashehu that worked, thanks!
d
Jobs are still stuck in the queue. For PRs that are not urgent, should I just let them wait and they will be run eventually? For PRs that are more urgent, should I move to github runners before pushing?
p
I solved by setting ubuntu-latest and cancelling the stuck jobs with 
gh run cancel JOBID --force
You can find the queued jobs in the Actions page
@Daniel Lundin is this PR github.com/nf-core/metatdenovo/pull/452?
d
Thanks, I saw your post. I had tried to do the same but failed. In the end I managed. I was more wondering about general advice going forward particularly whether to switch to ubuntu-latest before pushing to make sure nothing gets stuck that I have to remove. Plus, I have two queued now that I can wait for if things will solve themselves. And, yes, that's one of the not very urgent. I have a more urgent coming up.
p
@Daniel Lundin I see. I don't know, I guess we go with github runners until the situation gets back to normal?
d
Yes, that's what I think too.
👍 1
g
Is there any possibility for a GPU workaround? I know it's tough since I don't think Github offers GPU runners in the same way it does for CPUs, but I figured I'd ask! Otherwise, I'll just closely follow here for updates on the final fix.
p
I don't think there's any for free
👍🏾 1
g
Makes sense!
👍 1
d
So I tried to unblock github.com/nf-core/ampliseq/pull/1038 by cancelling github.com/nf-core/ampliseq/…/91940148067 , but
Copy code
gh run cancel 30642778318 --repo nf-core/ampliseq --force
unknown flag: --force
Usage:  gh run cancel [<run-id>] [flags]
So tried with
Copy code
gh run cancel 30642778318 --repo nf-core/ampliseq
✓ Request to cancel workflow 30642778318 submitted.
But it seems to not be cancelled. Maybe I lack permissions or such? Or due to old gh version 2.45.0? 😩
p
you need to update 
gh
to the latest version otherwise 
--force
won't work
d
Ok, thanks!
👍 1
e
PR on 4.0.3 atm: github.com/nf-core/proteinfamilies/pull/182 with the workers still struggling. What should I do? Wait for an auto PR for template 4.1.0? Manually sync it? Something else?
d
I've done a couple of manual syncs, remembering to keep the "ubuntu-latest", and that has worked so far. The risk is of course that I will miss the info when it's safe to revert.
n
Did you cancel the stuck run? I think if you switch 
nf-test.yml
to use 
"ubuntu-latest"
, push that and then 
gh cancel 30998406537 --force
it should work
👀 1
e
I swapped to ubuntu-latest, but 
gh cancel 30998406537 --force
gives -> 
unknown command "cancel" for "gh"
, and 
confirm-pass
still shows to wait
n
is your 
gh
up-to-date?
e
I just downlaoded it
n
Strange, bc I also just conda installed it 2 hours ago and it was fine
oh
it is 
gh run cancel
, sorry
❤️ 1
e
worked! thanks @Niklas Schandry 😄
any idea about the failing singularity tests? github.com/nf-core/proteinfamilies/…/92626197812?pr=183
Copy code
> Command error:
      >   WARNING: DEPRECATED USAGE: Forwarding SINGULARITYENV_NXF_TASK_WORKDIR as environment variable will not be supported in the future, use APPTAINERENV_NXF_TASK_WORKDIR instead
      >   WARNING: DEPRECATED USAGE: Forwarding SINGULARITYENV_NXF_DEBUG as environment variable will not be supported in the future, use APPTAINERENV_NXF_DEBUG instead
      >   ERROR  : Failed to create container process: Operation not permitted
Seen this?
Testing rolling back Nextflow version to 25.10.4..
no luck..
2 Views
Open in Slack
PreviousNext
Powered by