Sometimes.... seems like speculative builds just don't work. I've got 6 stacks queued, only one has CI running on it, yet I am configured to run 3 at a time... MOST of the time it works, but then just randomly.... nope, just one. Ideas?

hey Jason, could you please share the URL(s) for any impacted PRs and will check in our logs

i believe it was this app.graphite.com/github/…/10876

For that PR I see in our logs that Graphite did kick off extra speculative merge jobs. Several of those jobs (including 10876, plus others in the queue) failed on rebase conflicts, so those speculative branches never got a full CI run. When a downstack entry fails, the rest of that stack gets evicted instead of sitting there with parallel CI. 10876 did get through on the later enqueue and completed. If it happens again: rebase the queued stacks onto current trunk so speculation can create the extra CI branches.

I do see the next one queued was evicted due to merge conflicts, this seems to be a problem. Speculative builds stop here? can it not get evicited before it reaches #1 ?

Yes, conflicts can be detected and the affected stack evicted before it reaches #1. That is part of what the speculative builds are doing. The catch is that Graphite cannot know there is a conflict until it attempts the speculative rebase. When one attempt fails, its upstack entries are evicted and later speculative work is temporarily reset, then restarted on the next queue pass. That reset is why the queue can briefly look like speculation stopped. In this case, rebasing the conflicting stack onto the latest trunk should let the other speculative builds continue.

Its not clear if this behavior is expected behavior or not. It is certainly not the desired behavior on my end. Are you saying the conflict is only if the pr at #1 merges? and not conflict if #1 failed? I would much rather assume that #1 will succeed and kick the potential conflict out of the queue or move to the end

Yes, the conflict is calculated on the assumption that #1 succeeds. The next speculative build uses #1’s prospective result as its base. If #1 fails, that downstream speculative result is discarded because its assumed base will no longer become trunk. If #1 succeeds but the next stack cannot apply on top of it, Graphite should evict that conflicting stack and its dependent PRs before it reaches #1. Graphite does not currently move the conflicting stack to the end, since it would encounter the same conflict again without being rebased. It should, however, resume speculative builds for independent stacks behind it. If those remained stopped with only one CI running after the conflicting stack was evicted, that is the behavior I want to investigate further.

so, root cause is graphite does not have the ability to speculate a #1 + #3 build ? things did get moving back to normal once #1 merged, #2 got kicked, and #3 and on started speculating again

Yes, that's the right read. Speculative builds are chained strictly in queue order: each one is created on top of the branch of the build immediately ahead of it, and it carries every entry ahead of it in the queue. There's no way to assemble a "1 + 3" build that skips a queued entry, so when the entry at position 2 can't be rebased, there's no branch for 3 to stack onto and everything behind it waits. The queue only re-forms once the conflicting entry is failed and removed, which is exactly the sequence you saw: 1 merged, 2 got kicked, and 3 onward started speculating again. Your repo is set to run 4 at a time, and we can see it actively trying to start the extra builds but they were dying on the rebase before CI ever got a chance to run. Two things that help in the meantime: • Keep PRs restacked on trunk before they enter the queue. • If you know a PR is going to conflict, pulling it out of the queue rather than letting it reach the front gets the rest of the queue moving sooner. I'll note this underlying behavior as an unexpected product gap for the team.

please