https://linen.dev logo
Join Slack
Powered by
I’m finding the CI options a bit confusing when se...
# how-do-i
g
I’m finding the CI options a bit confusing when setting up a merge queue. We have this on currently (run on topmost) - I was expecting this to only impact when merging with the merge queue, but it seems to be also impacting just normal pushes (gt submit). For example if I submit two stacks at once, only the top runs CI. This is awkward because we have Required status checks on in GH, and we can’t merge unless everything has passed the required checks. I tried running on every PR, but then when you merge into the queue, each CI runs and it seems to defeat the purpose. Is there another setting I need to change?
p
(If applicable) To help us more clearly understand what you're trying to do, please provide additional details about: - Screenshots / debug logs / list of commands you've tried to run so far - Your workflow - Current state
Apologies for the confusion! Taking a look here and will circle back.
Hey George,
it seems to be also impacting just normal pushes (gt submit). For example if I submit two stacks at once, only the top runs CI.
This doesnt seem correct, would you be able to share an example PR where youre seeing this?
Looks like oyu might have had the CI Optimization on too (and disabled today) which was causing this?
g
So I tried the CI optimisations to see if that would help at all but it didn’t so they have been turned off the whole time whilst I have been testing. I can run a test now and send you the PRs - will get back to you this morning
Just ran a test - I have a stack of branches - • alb-metrics-db/schema • alb-metrics-db/persist-from-cron • alb-metrics-db/backfill-and-tooling But CI only ran for 
alb-metrics-db/backfill-and-tooling
after running gt submit Is this something to do with the way that you push under the hood? Is it doing one push updating all of the remotes at once? Not sure if GH would trigger separate webhook events to circle for that
Okay i have a proper reproduction for you! Just ran some tests with branches: • push-test-1 • push-test-2 The first push does not cause all branches to have ci run - but if you edit something lower down and re-push all stacks again, they do run. Is there a difference in your git commands on a first time push vs changes with rebases etc?
p
hey george, within graphite, can you copy the PR link and paste it into this chat so i can take a look at our logs. thanks!
g
p
i noticed that your PRs are in a draft state. do you have any rules on your CI that you don't run on drafts?
g
No they do always run - it’s not PR driven, just branch pushes. They did run fully even in a draft state when we did a modify which updated the stack and then submitted again.
👍 1
p
okay so i see you have circleci set up. for ci optimization, we only have integrations for buildkite and github actions so the skip behavior wouldn't occur.
g
This doesn’t appear to be a CI optimisation though - i’m not asking for an optimisation change. If I push all of the branches at once, do we not expect the branches to all run in CI? Why does it work when you push up a rebased stack but not work on an initial push? Sorry I think we are talking about different things here.
p
hey george. i think we are. let's reset here! when you're using your CLI, what command are you using to push your commits?
g
gt submit
p
> If I push all of the branches at once, do we not expect the branches to all run in CI? > > Yes, we absolutely expect CI to run on all branches when you push them all at once. If it's not, that's a GitHub quirk, not intentional Graphite behavior. > Why does it work when you push up a rebased stack but not work on an initial push? > > What do you mean by 
Why does it work
? What works?
g
When you first push stacks, only the top CI runs. If you make some more changes, then modify which causes the rebases, a push then causes CI to run on all stacks. I had a chat to the AI about it, and it suggested that maybe the git commands used under the hood are different or something based on these scenarios, but that’s obviously speculative! Our whole team are experiencing this FYI - it could be something on our end with GH but it seems weird that it only does not work on the first push.
p
that is the expected behavior. you have 
main > stack_a > stack b
then you run 
gt submit
--> this pushes all the branches github. ci will run on 
stack_a
before 
stack_b
due to github's own behavior. gh procceses the push events and triggers Actions -- graphite doesn't control the CI execution order. you merge 
stack_a
but haven't yet merged 
stack_b
, you create 
stack_c
and modify 
stack_b
. then you run 
gt submit
. this pushes the changes from 
stack_b
and creates 
stack_c
. ci gets re-triggered because the head SHA changes. 
stack_c
is a new branch so gh triggers CI for its first push. do you happen to have concurrency set up within gh?
g
Unfortunately this isn’t what I’m seeing with the branches that I sent through to you and described above. In your example with 
main > stack_a > stack b
- only 
stack-b
is running after a submit. Is that the expected behaviour? We don’t have concurrency turned on. In my examples above, I’m not talking about creating a new branch. It works as expected and runs ci for both if I make a change to 
stack_a
and a restack 
stack_b
and submit. Is there any chance I can get the same person to reply or a DM or something - I’m having to re-explain the same thing quite a few times and I feel like we’re not getting anywhere.
p
> It works as expected and runs ci for both if I make a change to 
stack_a
and a restack 
stack_b
and submit. > > this is expected behavior due to github
We don’t have concurrency turned on.
We recommend using GitHub’s concurrency configuration to ensure that you do not have duplicated CI. This may be the behavior that you're seeing since this is not enabled.
g
Hello again. Sorry I don’t think you replied to the bit where I stated that it is not working as expected? Yes, if I restack after changes and push it is working as expected. If I push for the first time, only the top of the stack runs ci. This is where the problem lies, as i did mention in my last message. Could you explain if this is expected behaviour? This concurrency angle is only related to too much ci running rather than it not running? I would really appreciate some clarity on this otherwise I’m not sure if we can continue using graphite unfortunately.
p
Good Morning George, had a look through this conversation.
If I push all of the branches at once, do we not expect the branches to all run in CI?
You are absolutely correct we would expect this to run on all Prs in the stack.
If I push for the first time, only the top of the stack runs ci.
This is not expected. Going to set up Circle CI and have a test myself. Will report back. In the meantime, would you be able to push up a test stack and leave it in that state so i can look into our logs to see if there is anything that stands out?
g
Have pushed them up. Also attached a screenshot so you can see the flow of commands etc! 
gt-test-two
is the only CI run.
👀 1
p
perfect thanks! as a side note, seems like a slightly inefficient workflow youre using for the creation. I typically (and what we reccomend) is from parent branch, make new changes, stage them and then 
gt create -m "branch name"
in fact i usually just 
gt c -am "branch name"
to stage add all changes to the new branch, saves you from needing to modify which is a bit more the classic git workflow of making a branch then committing changes. Maybe yoyu prefer that workflow but just in case-
Alright George, thanks for your patience. I can reproduce the behavior you're experiencing with CircleCI. Ran a bunch more tests and have isolated the issue: When 
git push
sends multiple refs in a single invocation, even though GitHub sends multiple push webhook events (one per ref), CircleCI only creates a pipeline for one of them (appears to be the last ref in the command, aka top of the Graphite stack as youve seen). Sequential pushes of the same branches work fine, atomic multi-ref pushes don't. 
gt submit
pushes stacked branches in a single 
git push
, so the update is atomic and you hit this. Quick workaround (not ideal, but unblocks you): submit PRs individually instead of running 
gt submit --stack
. So rather than creating branch A, B, C and submitting the whole stack at once, 
gt submit
on each one as you go. I'd also recommend filing this with CircleCI, since the fix ultimately lives on their side (or they can share how to get around this). Here's a minimal repro you can attach: # Triggers only 1 pipeline (BUG)
Copy code
git checkout -b atomic-1; git commit --allow-empty -m "1"
git checkout -b atomic-2; git commit --allow-empty -m "2"
git checkout -b atomic-3; git commit --allow-empty -m "3"
git push origin atomic-1 atomic-2 atomic-3
# Triggers 3 pipelines (expected)
Copy code
git push origin atomic-1
git push origin atomic-2
git push origin atomic-3
g
Thank you for the reply! I thought it might be this - I actually suggested this on the 13th earlier in the conversation! Okay good to know - I’ll let the team know! Appreciate you helping and reproducing the issue
2 Views
Open in Slack
PreviousNext
Powered by