This message was deleted.
# report-bugss
Slackbot
08/29/2023, 10:46 AMThis message was deleted.
f
Front
08/29/2023, 2:27 PMHey Ed! I can help look into this for you - do you mind sending me your org/repo name? (DM is ok)
e
Ed Smith
09/08/2023, 1:31 PMHey we’re still seeing this issue, @Pranathi Peri any update?
p
Pranathi Peri
09/08/2023, 1:45 PMHey there! Sorry about the delay here, I'll ask our MQ engineer to take a look!
Pranathi Peri
09/08/2023, 1:46 PMHe's taking a look now! Will keep ya updated 🙂
d
David Bradford
09/08/2023, 1:55 PMHi Ed, I'm investigating. It looks like you are hitting an issue we have been trying to track down.
Do you know if you repository has the setting "Require deployments to succeed before merging" turned on, or is using Github's "Rule sets"?
e
Ed Smith
09/08/2023, 2:17 PMI believe I gave the Graphite App the required permissions when I set this up (I followed the instructions and it was working briefly). I will double check this setting.
Ed Smith
09/08/2023, 2:18 PMSo we’re not using any rulesets
Ed Smith
09/08/2023, 2:19 PMAnd we do not have that “Require deployments to succeed before merging” turned on
Ed Smith
09/08/2023, 2:19 PMThanks for looking into this!
d
David Bradford
09/08/2023, 2:32 PMIf you look at the PR on GitHub, does it provide any messages about blocking it from merging?
David Bradford
09/08/2023, 2:52 PMDo you know if you gave the
graphite-appe
Ed Smith
09/08/2023, 3:15 PMYeah I did give those permissions
Ed Smith
09/08/2023, 3:15 PMI’ll get back to you on the first point
d
David Bradford
09/08/2023, 3:42 PMTo provide some more light on what is going on...As part of the merge process, we will query github to get the "Merge State Status" of the PR, which is whether github thinks the state of the PR is (can it be merged, is it blocked by something, etc). In this case, that status is BLOCKED. Most of the time, BLOCKED indicates that there is CI that failed or is still running. Unfortunately, the github API doesn't tell you why it is blocked, just that it is. So we have been trying to track down what is causing github to think the PR is blocked.
e
Ed Smith
09/08/2023, 3:47 PMOK, this is what we see on an example PR that failed to merge
Ed Smith
09/08/2023, 3:48 PMAll CI checks passed apart from one that was skipped
d
David Bradford
09/08/2023, 3:50 PMIf you look at the PR in github, are there any message that would block the PR from merging?
e
Ed Smith
09/08/2023, 3:51 PMnope
Ed Smith
09/08/2023, 3:51 PMtrying again now
Ed Smith
09/08/2023, 3:52 PMLooks like Graphite rebased it as part of the merge process so CI checks are running again
Ed Smith
09/08/2023, 3:52 PMI’ll let you know when it’s finished
d
David Bradford
09/08/2023, 3:52 PM👍
David Bradford
09/08/2023, 3:53 PMi see that merge failure in our logs. Everything looks successful to use. But github seems to think the PR should be blocked from merging.
e
Ed Smith
09/08/2023, 4:02 PMOK all CI has passed and it’s stayed in this state in the queue for a few minutes, not sure why, I don’t think there is anything else in the queue
Ed Smith
09/08/2023, 4:03 PMI need to leave for the day soon, so can we pick this back up next week? Is there anything else I can tell you to help debug?
Ed Smith
09/08/2023, 4:05 PMAh we just got the same message again
d
David Bradford
09/08/2023, 4:45 PMI'll dig in some more. I have an idea for a workaround I'll try to get into today. Then we can sync back up next week to see if that helps.
e
Ed Smith
09/12/2023, 2:19 PMHey @David Bradford did you find anything last week?
Ed Smith
09/12/2023, 2:19 PMWe currently have the merge queue turned off and we’d really like to use it!
d
David Bradford
09/12/2023, 2:24 PMI haven't been able to figure out the root cause yet, but I did deploy some code to detect this situation and try to handle it a little better in the MQ. At the very worst, it should provide me some better information to debug what is going on. So, if you get a chance to try to hit the issue again via the merge queue, so that I can try out the workaround code, I'd appreciate it.
l
Lee Staples
10/26/2023, 2:46 PMWe are still seeing this issue, is there anything which is being or has been done?
j
Joyce Lu
10/26/2023, 6:48 PMchecking with the team, sorry for the delay!
z
Ziyao Wei
10/26/2023, 8:52 PMHey - sorry for the issue, can you DM me the PR repo and number? Thanks! I am not seeing anything on our side from the past week
l
Lee Staples
10/27/2023, 9:27 AMI don't have that information available anymore sorry
e
Ed Smith
10/27/2023, 11:18 AMHi Ziyao, we had the merge queue disabled since we last discussed this until now. We figured we’d turn it back on and see if it worked again but unfortunately it didn’t 😞 The repo is project-nous/monorepo and the PR that recently failed to merge was 6386. Let me know if you need any other info.
🙏 1
z
Ziyao Wei
10/27/2023, 3:34 PMThanks, I’ll take a look later today. Sorry for the issues you ran into and I’ll keep you posted!
e
Ed Smith
10/27/2023, 3:42 PMGreat thanks 🙌🏻 Look forward to hearing from you.
Ed Smith
10/30/2023, 5:42 PMHey if it helps, I think the merge queue just worked for us with PR #6427 on the same repo.
I followed the second half of the instructions on this page to “Enforce the Graphite merge queue”. I previously thought they were optional but maybe they are required?
💛 1
Ed Smith
11/01/2023, 1:27 PM@Carissa Jansen this is the thread I referenced in my email
c
Carissa Jansen
11/01/2023, 2:56 PM@Brendan Ngo is working on these bugs right now!
e
Ed Smith
11/01/2023, 3:25 PMGreat stuff, thanks 🙌🏻
One other thing we noticed today was that pull requests marked as hot-fixes were failing. Other PRs appear to be merging fine. Is that a problem you’ve seen before?
b
Brendan Ngo
11/01/2023, 3:38 PM@Ed Smith What PR failed the hot-fix? I can take a look
e
Ed Smith
11/01/2023, 3:48 PM#6438, #6444 and #6445 are recent examples
Ed Smith
11/01/2023, 3:49 PMSome comments from the Graphite app say that they weren’t satisfying all the requirements
Ed Smith
11/01/2023, 3:49 PMI think they actually were but can’t be certain as I’m not the author
b
Brendan Ngo
11/01/2023, 6:01 PMIt looks like these prs failed because they were missing approvals. Looking at whether this is expected behavior or not given we may be able to bypass branch protection rules
Brendan Ngo
11/01/2023, 6:07 PMturns out we do not actually bypass the branch rules so these failed because they didn't have any approvers. hotfixes will just skip the queue
e
Ed Smith
11/01/2023, 6:09 PMOk I'm not sure I fully understand... Are we able to put prs in the merge queue without approvals? (I guess it might depend on our settings)
b
Brendan Ngo
11/01/2023, 6:13 PMThat is what's currently happening. But I agree with you this is unintuitive, let me investigate why we allowed this in the first place!
e
Ed Smith
11/01/2023, 6:23 PMGreat thanks! Yeah I'd expect whatever our settings are - say we require 2 approvals for merging (certain users may be able to override that) - then we'd need 2 approvals on a PR before it enters the merge queue (and certain users may be able to override that and put the pr in the merge queue without the required approvals).
Marking as hot-fix would make it enter the merge queue at the front, but not allow it to enter without the required approvals.
I'll catch up with the authors of the prs to see if I can get more info on what actually happened.