The CI runners will periodically get stuck for me ...
# sarek_dev
g
The CI runners will periodically get stuck for me and keep the jobs in the queue for 24+ hours without the ability to cancel it. I think the culprit this time is the
confirm-pass
task here. Is there a consistent way to get out of this? Thank you!
r
we are experiencing some issues on the AWS side and are investigating
💚 1
g
Thanks, Rike for the update! I'm following the thread in the other channel. I am hoping it gets resolved before the next Sarek release so I can get the rest of my PRs through.
r
I think otherwise we'll need to start working on some sort of honor system / we'll run the GPU tests on 2 different machines or sth. Don't want the release to be blocked by this 🫠
👍🏾 1
g
Agreed! And I have local GPUs I am testing with in the meantime.
Hey Rike, I'm hoping to get my last two PRs merged soon. Is there a path you can recommend to complete the GPU CI testing while the runners are down? Or are there any updates on the progress of fixing the runners?
r
we just got it fixed. I am merging the runners back into dev
💚 1
🎉 1
ok merged in
g
Thank you so much!