Is there a "best practice" to prevent GitHub Actio...
# how-do-i
l
Is there a "best practice" to prevent GitHub Action required checks from ruining the merge queue experience? Every single PR my team merges gets kicked back at least once, we've had to resort to merging in GitHub for now because the experience has degraded so much.
c
Common pattern when every PR gets kicked at least once is required Actions fail or never report on the merge-queue draft (
gtmq_
/
[Graphite MQ] Draft PR
), not the original PR. Make sure those drafts run your required checks, ignore
graphite-base/*
, and watch Parallel CI if tests flake. Got a recent kicked PR + the Graphite removal comment that I can view to see what happened? Thanks!
l
this one failed the first time, then succeeded on the second merge queuing: app.graphite.com/github/…/…
c
Hey Luke! Can you DM your github name so I can peak at the logs under the hood? Thanks!
✅ 1
l
This is still a big issue for us, any progress on its resolution?
I just had to retry this one several times, and at time of writing it hasn't gone through: app.graphite.com/github/…/feature-flag-onEnable%2FonDisable-functions-were-not-being-awaited
c
We did have some latency issues earlier that caused some slowness, looks like it completed now. Let me know if you see it with future stacks as we are seeing the latency get to normal levels.
l
its happening frequently again, here's another example: app.graphite.com/github/…/11807
Merge activity
• Jul 31, 6:37 PM UTC:`darthvadrismydad`added this pull request to theGraphite merge queue.
• Jul 31, 6:42 PM UTC: TheGraphite merge queuecouldn't merge this PR becauseit failed for an unknown reason.
• Jul 31, 6:44 PM UTC:`darthvadrismydad`added this pull request to theGraphite merge queue.
• Jul 31, 6:44 PM UTC: TheGraphite merge queuecouldn't merge this PR becauseit failed for an unknown reason.
• Jul 31, 6:44 PM UTC:`darthvadrismydad`removed this pull request from theGraphite merge queue.
• Jul 31, 6:47 PM UTC:`darthvadrismydad`added this pull request to theGraphite merge queue.
• Jul 31, 6:52 PM UTC: TheGraphite merge queuecouldn't merge this PR becauseit failed for an unknown reason.
c
Hey There! Did a little digging on this. The earlier Graphite merge queue failures happened because GitHub was still waiting on the required
Typecheck
check. Graphite attempted to merge while that required check was still in progress, and GitHub rejected the merge attempts. Graphite surfaced that rejection as “failed for an unknown reason,” but the underlying reason was specifically that the required
Typecheck
status check had not finished yet. Each time the PR was re-queued, the branch was rebased, which restarted
Typecheck
, so Graphite retried before GitHub considered the PR mergeable. Once
Typecheck
completed, GitHub allowed the merge and the PR went through.
l
Am I wrong in assuming that Graphite should be able to handle rebases gracefully? every single merge will result in a rebase, so I don't quite follow how merge queues are supposed to function w/o handling this case.
c
You are correct, and I think the real crux here is how we are displaying the error. If we were to respond "Waiting for Typecheck status to complete" than this might be more clear as to whats happening behind the scenes. We have an open report on this and we are currently investigating. I do not have a timetable to provide but once I have a resolution I will double back and let you know the results. Thanks!
🙏 1
l
This continues to be a productivity issue for us, and at this point we are actively exploring alternatives. It has become a waste of time & money, because our engineers get stuck behind this endless loop of babysitting merges, while doing it through GitHub has been painless. Make it make sense. Is there any chance that this issue will be addressed? how is anyone else able to use this w/o this issue?
c
There is actually an issue right now with GitHub actions https://www.githubstatus.com/incidents/qcvjkzcs7j74, is it worse today than previous?
l
It has been consistently bad for over two weeks at this point, I don't think GitHub is making things better per-se but it didn't make it worse than the baseline for us
c
I have credited a report on this but no ETA to provide. Let me know how it feels today, it was hard to troubleshot with GH actions down for most of yesterday but we are seeing normal response time currently. Thanks!
l
We'll check it out, thanks for following up
this is still happening to us consistently. We do have a Github action that has cancel-in-progress set (basically so tests get cancelled if someone does a push). It appears, notionally, that rebases cause this cancel to trigger. This leads Graphite to show the required check as "cancelled", whereas in reality it is actually just running in a new process. Maybe that is causing the issue? its hard to tell, but maybe that's the edge case
c
Ok thanks for the clarification. I have added all this to the existing report, no ETA to provide but I did bump this for the team for review. Thanks!
🙏 1
l
We figured it out -- looks like the github action that was messing things up needed this change:
Copy code
steps:
      - uses: actions/checkout@v5
        with:
-------- PREVIOUS, BAD -------
          fetch-depth: 100  # Deep enough to find merge-base with master for the rebase below; avoids fetching every branch.
------ NEW, GOOD ---------
          fetch-depth: 0
          filter: blob:none
🎉 1
Now the merge queue is working again for us
c
Updating some internal docs for this!
l
excellent! hope it helps someone out there troubleshoot in the future 🙏
🙌 1
one more to check here, just verifying it is not from the same issue: app.graphite.com/github/…/12088 this happened a full day after the action update I described above, which has definitely improved things.
c
Thanks for flagging #12088. If Graphite still showed unknown reason / Typecheck in progress after a rebase, that's the same family as before: leftover if that branch still had the old 
fetch-depth: 100
 checkout. Glad the queue is healthy again; ping if it comes back at that rate and ill peek under the hood. Thanks!
👍 1
l
Unfortunately this has come back again 😕. here's an example app.graphite.com/github/…/label-roboscreener-db-connections
c
Thanks for the extra example, I'll look at #12230. When it got kicked, was Typecheck cancelled, still running, or actually failed? If it was still running, I'd try leaving it queued rather than re-adding so we don't restart the check.
l
it was still running, but it looks like it shows up as cancelled in Graphite. We do have two separate jobs that have the same name, I'm wondering if maybe there's an issue with naming collision here?
c
GitHub required checks match by name, so two Typecheck jobs can collide (one cancelled, one still running). I’d give them unique names so each check is distinct. Are both jobs literally named Typecheck?
l
Yes they are. Its a hacky workaround so we can skip the typecheck for changes that do not have type-related effects (e.g. language updates), since it appeared that GitHub doesn't support anything like that.
c
Thanks for clarifying this does sound a bit like an edge case and let me create a report on it and do some brainstorming for workarounds.
l
We removed this, and we have ceased to see the issue anymore, so perhaps that did it. I'll report back if it comes back!
Its definitely an unusual case, but good to know about I suppose!
c
Yes agreed!