seems like my graphite queue might be stuck, no PRs have merged in almost an hour, removing them has also become stuck. is this the right channel for requesting help on something like this?

😞 i think mine is stuck too, CI has passed, but its just sitting at the top of the merge queue

image.png

having the same issue.

Probably due to status.x.ai/?incident=…

alas..

🤷

🙃

Can we at least get the status page to reflect the outage? We're currently at 2+ hours since we noticed it and yet the Graphite status shows that everything is operational.

some kind of actual acknowledgement would be great

+1 to a desire for acknowledgement

We are also seeing issues 🙏

i also used gt feedback

same here!

Anyone got unstuck? 😓

Any Graphite employee here? What the hell is this? A core feature is down for hours and there is not even any response here?

Also finding this - would be good to get that status page updated.

This has been pretty poor, to put it politely, how has there been no response for 10 hours?

Sad rant. Before this I barely had an excuse to use Graphite but I kept using and promoting because I see that they are innovating and I wanted to support that. After having all progress stop for 9 hours Ive been forced to pull graphite our of our workflows and processes. Its unlikely that any argument I make to bring it back will succeed.

+1

Hey all, apologies for the interruption here. We're experiencing issues with our systems and are investigating. Will post status updates

Is there going to be a published retrospective on the outage? I think a lot of folks (like myself) would be curious as to what happened and how this occurred for 12 hours without a response (we tried multiple ways to get a response).

For us this incident brought a big stain on Graphite's reputation. About a year ago Graphite's support team was incredible and we happily signed, knowing that there was always somebody there to help us. Everybody knew SpaceX would not bring much positive change for Graphite customers but I did not expect "12h down time without any communication" bad - we are internally already discussing exit plans. We need to know if the on call support will come back or we are stuck with US time zone only + an useless AI bot.

Hey all, I can provide a high-level RCA summarized below: Late overnight ET on Thur, 29 Oct, a change occurred in an indirectly-related system which caused an unintended side-effect of merge queue jobs hanging indefinitely. As the jobs were not outright failing, the systemic nature of this side-effect was not picked up by our automated telemetry monitoring, and so also did not trigger our automated alerting. Support tickets are manually monitored throughout US business hours, so as soon as we became aware this issue was systemic based on user reports the following morning we immediately manually alerted the engineering team and posted status page notifications, including with guidance for working around by merging from in GitHub. The root cause from the indirectly-related system was identified and fixed then within about two hours. Our apologies again for the disruption this caused to your workflows, and our slow response to the issue overnight.

Thanks for the high level overview, Gary. Question. Is there a plan for an escalation process in light of the fact that this was reported and there was no response? I can't think of another service we pay for that doesn't have some sort of escalation.

This will be part of the incident postmortem review we do on our end