Graphite Community

Merges from the Graphite dashboard seem to be failing with 500s.

```POST <https://api.graphite.dev/v1/graphite/mutation/merge-pr> 500```

Actually, this is maybe a Github bug. Merges are failing on Github too.

<https://news.ycombinator.com/item?id=35325709>

image.png

<https://downdetector.com/status/github/>

thanks for flagging! i’ll look into this for you :slightly_smiling_face:

Screen Recording 2023-03-27 at 11.07.48 AM.mov

Maybe it’s not on Github’s side, and is instead of Graphite’s side? He just ran `gt rs --debug` and it’s been hanging for a long time (30 min so far and still going) on `Persisting cache`:

```% gt rs --debug
🧹 Checking if any branches have been merged/closed and can be deleted...
2023-03-27T17:37:43.191Z: Checking if should delete [branch-name]...
2023-03-27T17:37:44.843Z: Unblocked branches:
.....
2023-03-27T17:38:16.001Z: Unblocked branches:
2023-03-27T17:38:16.001Z: Persisting cache...```

In case it matters:
```% gt --version
0.20.15```


Okay, my new theory after looking at <https://github.com/withgraphite/graphite-cli/blob/fb6422f0112651e51657b2927fe7ebffe3a65e3d/apps/cli/src/lib/engine/cache_loader.ts#L44|source code> is that this is based on the number of branches the engineer has locally, and not Github-side rate-limiting. I asked, and he confirmed that he has a lot of branches.

Is it a known problem that Graphite commands can take 30+ min when you have sufficiently many local branches?

Sounds like he has ~240 local branches, spanning many years (not all have the same parent). I vaguely recall something like this coming up in the Graphite slack a ~year-ish ago (I remember someone saying performance was roughly quadratic in the number of commits between your newest and oldest branch parents, or something like that?), and the recommendation being to add those branches to the ignore list. But the ignore list seems to be <https://docs.graphite.dev/guides/graphite-cli/a-note-on-cli-v0.19|gone>, so I’m not sure what the new workaround is.

Ah, yes  — we moved from the ignore list to a model where branches need to be explicitly tracked — are all 240 of these branches tracked?

Also, any chance he's on an older version of the CLI? (pre-0.19)

but also performance should no longer be any worse than O(branches) since v0.19

He’s on 0.20.15, as mentioned earlier in the thread.

I can ask if all of his branches are tracked, but before I do so: is 30+ minute runtime expected when there are 240 tracked branches?

Okay, in that case: do you have any idea what the problem might be, or (if not) how we could collect more debugging info to help diagnose the issue?

i am not sure, we don't currently have great debug logging for this case unfortunately.  i will add this in a future version, but for now it should be safe to ctrl-c at that part of the command (the cache will self-heal on next run)

:wave: Running into this same issue today. It gets stuck at "Persisting cache..."

Is there a command to clear the cache potentially? It might help? <@U02RREE9GKH>

pressing CTRL-C here will clear the cache, and is fully safe — the "cache" here is just used for speeding up commands (everything is already written to git before this happens)

separately i'm looking into resolving this issue but it's not the highest priority atm given the workaround

If you're seeing this transiently, it's likely an issue on GH's end, as this is just the underlying `git push` that graphite performs failing.  Is there any pattern here?

The only pattern is I'm much more likely to see issues the larger my stack grows

uncertain if this is related but I also will see commands just hang for 10+ minutes often when my stacks reach a size of 4 diffs or more

I appear to have fixed it by adding an entry to Github to my ssh config that has it send its data over 443

*shrug*

Screenshot 2023-03-27 at 8.49.16 PM.png

We’re thinking about this and expect improvements in the medium term (2-3 months)

This as in the whole filtering experience. I’ll try and fix this bug soon. 

Also I make mistakes when using this input sometimes which leads it to erasing all the work I put into creating a section (e.g. clicking out of the box instead of clicking 'save'). Probably a PEBCAK but may be potential for UX improvement.

Screenshot 2023-03-28 at 3.45.17 PM.png

Screen Shot 2023-03-29 at 12.06.21 PM.png

FYI all, Github Actions has an incident and this will affect all Github Action related syncs &amp; checks on Graphite as well (they won’t go green). This may block your ability to merge.
Feel free to check for updates on the Github status page: <https://www.githubstatus.com/>

thanks for flagging! we’ll look into it :slightly_smiling_face:

fix going out now! sorry about that we missed a call-site in a migration

Screen Shot 2023-03-30 at 12.53.14 PM.png

not sure if the fix is out, but just got this notification - might be happening for users as well?

our pipeline was temporarily blocked, it should be going out soon!

should be out now! please let me know if you're still seeing errors!

I tried clearing the cache, and it seemed to be ok for a bit but now its erroring out again

Somehow forgot about this - should be fixed later today!

Screenshot 2023-04-03 at 09.51.32.png

Sorry you ran into this <@U03PEJWF8JF>, I know thats super frustrating

It's ok, gt has saved me waaaay more than 20 minutes. I only ask to help y'all out, and also because I started writing the description again but my heart just isn't in it, so I'm hoping there's some way to recover it?

And I did try going to the dir above: `/var/folders/7b/y1yhl2mx7554ndf3pq_zllfh0000gn/T/tmp-68511-GnWJcKvwrxEy` but it doesn't exist :cry:

When we save the description, it's not saved in the temp file — we cache it in the branch reference so that it's loaded into memory the next time you try to submit

Try running `git cat-file -p refs/branch-metadata/&lt;branch-name&gt;`

if it doesn't show up there, then it might be lost unfortunately

Oof, its lost I think. Ah well. Thanks! FWIW I find that `vi` very often returns a non-0 exit code when I happen to accidentally put in a bad command (even if I end up saving correctly in the end)

mm, interesting, that must be what it is — we should probably try to cat the file even if the command fails. thanks for the feedback! do you happen to have consistent repro instructions of that case by chance?

I just tried it and can consistently reproduce:
1. `gt bc -a -m "test"` with some changeset
2. `gt ss`
3. Press enter for default PR title
4. Press enter to edit body in vi
5. Press 'a' to enter append mode and enter any body
6. Press Esc to exit append mode
7. Press the backtick/tilde key, then enter. I don't know why but vi spits out an error: `E78: Unknown mark`
8. Enter `:wq` to write and quit vi
9. Witness gt error.
Note that when you try to `gt ss` again after this, your changes are not recovered.

Additionally, I tried this same process with vi just using a regular file, just to confirm that despite the non-zero exit code, vi DOES write the file, and indeed it does.

More info: <https://stackoverflow.com/questions/46665403/why-does-vim-run-as-the-alias-vi-return-a-non-zero-exit-code-when-any-error>

Looks like vim non-0 exit codes are not to be trusted! :angry:

Thanks for the thorough instructions! Makes my job a lot easier :slightly_smiling_face:

nvm i figured out that i didn't authenticate graphite cli

a better error message would be helpful here though

Hey there, <@U02RREE9GKH> I’m getting this error as well on a downstack get. Just ran gt auth, and repo sync runs with no issues. Any idea of what else could be causing this?
```$ gt downstack get active_status
Pulling main from remote...
main is up to date.

ERROR: Failed to get dependencies: [object Object]```

<@U04UL878KCP> -- sorry to hear. What do you mean by broken?

Screenshot 2023-04-03 at 1.46.37 PM.png

Can't see any data and it takes about 30s/1m to query

I'll take a look -- but the high query time is expected today. Overall, insights is not great right now but we expect to overhaul this experience this month.

Thanks for reporting, we're investigating

Hi <@U045W8WMUN5> did you see this consistently, or just for a brief period?

I’ve been seeing intermittent but significant latency for what feels like all day, but couldn’t give you a specific time window.

Several times the interface loaded but file content didn’t populate, so I had to fall back to GH

And several times, posting comments hung, and gave the red toast… and I think they went through, but not positive if some didn’t

The one that says “An unknown error occurred: Some data may have not been saved”

The review UI has been failing to load about 50-75% of the time for me today

heads up — github is haivng some issues with git ops that might affect CLI operations: <https://www.githubstatus.com/incidents/5nlnzt80fk7g>

cleaning the indexdb usually mitigates the issue.

cc <@U02DGG322G5> <@U02RREE9GKH> who have looked into this in the past

It’s definitely not just you, I went to demonstrate a few of the features I was using and have been getting sharpening pencils / houston we have a problem most of the day :disappointed:

<@U051LEHHR2P>: when it happens: open the developer console, go to "Application" and hit "clear site data"
It will destroy the browser state of graphite and rebuild it on refresh (so yes you'll have some loading but at least it will load)

I fondly regret services doing sync HTTP queries to their backend instead of trying to preload everything :sweat_smile:
(sometimes it was slow but there wasn't this "quantum state" where it is either "super fast and snappy" OR "showing a loader forever")

I just triggered this again. This time it triggered with the following series of commands:

1. `gt bco branchname`
2. `gt repo sync`
3. `git add filename` 
4. `git commit -m "commit message"`
5. `gt sr`
6. `gt ss`
So it appears that this isn't isolated to pushes without changes attached. But maybe its connected to the fact that I do `gt sr` after I make commits?

githubedited2.png

Saw an example of this occur and thought maybe the following github screenshot (lightly edited) might help show whats happening?

When I try, I get an errors in the javascript console:
```POST <https://api.graphite.dev/v1/graphite/auth-blocker> 401
    at main.bdb9d1ea.js:2```
and

```Uncaught (in promise) APIRequestError: Fetch Error: Non-2xx status code
    at main.bdb9d1ea.js:2:4485986```

1. Can't sign in with Github
2. I don't use Github for work
3. Submit

No... the identity and data on the site is based on your GH repositories and PRs — we don't support other code hosting services

Ah, but that survey response failure is something that we should fix

Tried on Windows 10 and Linux Mint 20.2 on multiple web browsers, Google Chrome, Firefox, Brave.

Screen Shot 2023-04-05 at 1.09.52 PM.png

Screen Shot 2023-04-05 at 1.11.00 PM.png

Yeah - google is supposed to recrawl our docs soon now that we've migrated them to the new site, but it looks like it hasn't happened yet, unfortunately.  We can add some redirects for these, but there are a bunch of them cc <@U04GQFJNDC7>