Our merge queues are becoming very long, for examp...
# how-do-i
t
Our merge queues are becoming very long, for example today it is 14 PR stacks long. Our CI is reasonable - between 10 - 20 minutes. It seems like the problem we have is that we work in a monorepo. What strategies can we use to deal with this other than splitting away from a monorepo (since this benefits us a lot)? Ideally we could have multiple merge queues and divide it out by folders, but this isn't currently possible
p
(If applicable) To help us more clearly understand what you're trying to do, please provide additional details about: - Screenshots / debug logs / list of commands you've tried to run so far - Your workflow - Current state
Hey Tommy, I noticed in your current merge queue settings you are still only set to 1 stack at a time. Definitely some room to increase this and improve queue times. Re: dividing out by folders... this is something we are working on.
t
Oh great, that was a good suggestion. I've enabled that and its moving a lot faster Is there any documentation on how this works? Does the 3rd PR in the stack gets stacked on the top of the previous two?
p
t
Hi Since enabling this feature, we've seen developers complain a lot more about their pull requests getting cancelled "randomly". I have been seen lots of our cloud builds getting cancelled. I understand that with processing 3 PR stacks at a time, we should expect that say if you are 3rd in the queue, and either the 1st or 2nd stack in the queue flakes, your stack CI will get cancelled, rebased, and rerun. However I am seeing devs saying they've been kicked out the queue even though the other one on top got merged with no issues. Any ideas on why this could happen?
p
Hi Tommy, would need to have an example or 2 of this to tell you why it would have been ejected.
t
We've just had one, I've attached a screenshot for our repository relaycode I can't really see a trace of what happened, our CI did not fail - it was cancelled by graphite. When I look at the PR it says CI in progress again When I look in our CI system (google cloud build), I don't see any of the gtmq branches failining - but I do see lots getting cancelled
p
thanks, if you could share the graphite PR link for one of the ones with CI failures I can take a look in our logs this afternoon
t
there are no CI failures
graphite is cancelling our PR CI runs without failures is what it looks like
p
right right... send me a link so i can see why we would have thought there was
t
Here is the PR, the engineer has queued it to merge again for the third (?) time https://app.graphite.com/github/pr/relaytech-co/relaycode/17532/Solver-for-Dynamic-Presort-Generation
👍 1
The same PR CI just got cancelled again This PR was second in the queue, and got cancelled after the PR ahead of it got merged. Graphite activity says it errored, but it did not, it was cancelled.
I'm changing the "How many stacks should CI process on simultaneously?" setting back from 3 to 1 for now
p
Hey Tommy, idk if you wanna chase this down further but I do see the status of
relayapi-pull-request (relaytech-sandbox)
holding this back from merging and representing the "failed" test even though its actually cancelled. Looks like we checked in for ~15 minutes before this cancelled status appeared. Can you show me where in your logs you see that graphite cancelled this run?
its suspiciously always right around 15 minutes. Wondering if maybe there is a timeout on that test
t
Interesting, we do have a timeout for the test step within the CI run, but its set to 21 minutes, and was cancelled after 10m20s, with a total runtime of 12:12 Unfortunately cloud build does not tell us what cancelled the run, but every time it has coincided with the merge queue activity log telling us it has evicted the PR
image.png
Hi, any update on this?
p
I have no update here unfortunately. I do not see anything indicating that Graphite specifically is cancelling things. I see us polling status and then we receive a CANCELLED status causing us to eject from the MQ for "failing"
t
Right I see, thanks for lookking