There was an issue with VSCode's markdown features that slowed down `Go to`:
https://github.com/microsoft/vscode/issues/152494
The issue is resolved in the insider build (I checked
1.71.0-insiders
), but I don't think it's released yet for the regular build.