:warning: Serious bug: stack merged by Graphite me...
# report-bugs
z
āš ļø Serious bug: stack merged by Graphite merge queue with required CI checks failing We recently merged this PR via the Graphite merge queue. The merge queue ran our merge-time CI checks in this draft PR. šŸ”“ The checks failed, but Graphite merged the PR anyway. This broke all deployments for us. Please could you folks look into how this happened?
āœ… 1
p
(If applicable) To help us repro, please send the following if you haven't already: - PR links, screenshots, and/or debug output if applicable - Repro steps - Does this happen consistently? Just to you or others on your team also? - Is this a regression? Is the problem new? - If you're having an issue with the CLI, please send us debug logs using
gt feedback --with-debug-context <any message>
hey Zeshan, thanks for that info. We are investigating this
z
We just had this happen again. • Original PR • Merge queue draft PR with failing CI checks • Outcome: Graphite merged the PR anyway āš ļø It looks like Graphite is closing the draft PR without waiting for the CI checks to actually finish running. • Draft PR closed at 10:39am • CI checks actually finished running at 10:40am šŸ”“ I'd like to emphasise how important it is that we solve this Graphite's customers give the Graphite Github app a whole host of permissions to write directly to repositories so that merging stacks will work. We trust you with these permissions because we assume that the Graphite app will function as intended and not merge broken changes. We love Graphite, but I think it's critical to all of your customers – above all else – that Graphite doesn't have bugs that allow broken changes to go through.
p
hey Zeshan, thanks for those additional details. I am making the team aware of them, we are treating this as a high priority issue. Confirming we also received your in-app feedback.
ā¤ļø 1
z
Thanks Gary! Appreciate your work on this ā¤ļø
p
hey Zeshan, the team is going to take a deeper look into this today but in the meantime the initial suggestion is to double-check in your CI logs if the failing check had possibly ran and passed and then re-ran and failed after the merge occurred.
f
Hey Zeshan sorry for the delay. I work on the merge queue and took a look at this. TLDR, I think your CI skipped some required checks when this pr was in the merge queue, but re-ran them after. Here's what I see: When the draft PR was in the MQ, we see the required status checks as:
Copy code
["Lint Common","Lint Platform","SQLC Check","Test Common","Test Platform"]
I confirmed this is the same list the PR page (which you can see) has the same checks. When CI finally showed as "passing", we see this (apologies for the verbosity, but I though that better than hand-waviness):
Copy code
{
  "ciRollup": "FAILING_NON_REQUIRED",
  "statusChecks": [
    {
      "conclusion": "SUCCESS",
      "completedAt": "2025-08-06T14:40:57.000Z",
      "annotationCount": 0,
      "workflow": {
        "name": "Encore Test",
        "id": 16780130353
      },
      "appName": "GitHub Actions",
      "name": "Setup",
      "ciStatus": "COMPLETED"
    },
    {
      "conclusion": "SUCCESS",
      "completedAt": "2025-08-06T14:40:10.000Z",
      "annotationCount": 0,
      "workflow": {
        "name": "Lint",
        "id": 16780130334
      },
      "appName": "GitHub Actions",
      "name": "Setup",
      "ciStatus": "COMPLETED"
    },
    {
      "conclusion": "SKIPPED",
      "completedAt": "2025-08-06T14:40:57.000Z",
      "annotationCount": 0,
      "workflow": {
        "name": "Encore Test",
        "id": 16780130353
      },
      "appName": "GitHub Actions",
      "name": "Test Platform",
      "ciStatus": "COMPLETED"
    },
    {
      "conclusion": "SKIPPED",
      "completedAt": "2025-08-06T14:40:10.000Z",
      "annotationCount": 0,
      "workflow": {
        "name": "Lint",
        "id": 16780130334
      },
      "appName": "GitHub Actions",
      "name": "Lint Common",
      "ciStatus": "COMPLETED"
    },
    {
      "conclusion": "SKIPPED",
      "completedAt": "2025-08-06T14:40:10.000Z",
      "annotationCount": 0,
      "workflow": {
        "name": "Lint",
        "id": 16780130334
      },
      "appName": "GitHub Actions",
      "name": "Test Common",
      "ciStatus": "COMPLETED"
    },
    {
      "conclusion": "SKIPPED",
      "completedAt": "2025-08-06T14:40:10.000Z",
      "annotationCount": 0,
      "workflow": {
        "name": "Lint",
        "id": 16780130334
      },
      "appName": "GitHub Actions",
      "name": "Lint Platform",
      "ciStatus": "COMPLETED"
    },
    {
      "conclusion": "SUCCESS",
      "completedAt": "2025-08-06T14:40:54.000Z",
      "annotationCount": 0,
      "workflow": {
        "name": "Lint",
        "id": 16780130334
      },
      "appName": "GitHub Actions",
      "name": "SQLC Check",
      "ciStatus": "COMPLETED"
    },
    {
      "conclusion": "FAILURE",
      "completedAt": "2025-08-06T14:40:51.000Z",
      "annotationCount": 0,
      "appName": "encoredev",
      "name": "Encore Build – wearegradient",
      "ciStatus": "COMPLETED"
    },
    {
      "conclusion": "SUCCESS",
      "completedAt": "2025-08-06T14:39:33.000Z",
      "annotationCount": 0,
      "appName": "Graphite App",
      "name": "Graphite / mergeability_check",
      "ciStatus": "COMPLETED"
    }
  ],
  "commitSha": "6112adce11c665029de00e1d3f0c1970bc684126",
  "ciStatus": {
    "failingOptionalList": [
      "Encore Build – wearegradient"
    ],
    "failingRequired": 0,
    "failingOptional": 1,
    "failingBaseRequired": 0,
    "status": "COMPLETED"
  }
}
the summary of which is: 1. encore build failed, but its not required 2. Encore Test / Test Platform skipped 3. Lint platform also skipped So 2 of your required status checks reported "skipped" to us, so our merge queue said, "ok sounds good everything else is passing so I will merge this" I'm not sure why it skipped, or why its now failing. Without knowing your CI config I can't really tell you. But in general, you can safely only run CI on branches prefixed with
gtmq_
when the pull request opens, but not on push. > We trust you with these permissions because we assume that the Graphite app will function as intended and not merge broken changes. Agreed with this completely and a core requirement of our merge queue is correctness, and bugs definitely happen, but in this case I'm not sure what the correct answer is to allow us to prevent a broken merge
šŸ‘€ 1
āœ… 1
One thing to check that may confirm if CI did in-fact re-run, is to use the timestamps from the above JSON and compare the timestamps you see in your CI logs, then we'll know
z
Thanks so much for looking at this Frank. You were right: • We skip CI checks on draft PRs (including those created by the Graphite merge queue) • Graphite sees everything being skipped and assumes it's okay to merge • All CI tests then actually run after Graphite pushes to the
gtmq_
branch, but by then it's too late We've solved this by always running CI if
startsWith(github.event.pull_request.head.ref, 'gtmq_')
is true. ā¤ļø Thanks for all of your help! šŸ“„ It might be worth adding this to the Graphite merge queue documentation as something to avoid.
šŸ‘ 1