Hey <@U093XHCH2D8>, I was chatting with Robert and...
# feedback
l
Hey @gentle-apple-32922, I was chatting with Robert and we thought it’d be good to have a consolidated place where we can see all open issues on the OpenHands repo that have not yet gotten a response. Here’s an example of a query that allows us to do this for now:
Copy code
org:OpenHands is:open comments:0
There are a few possible improvements to this. 1. We might want some way to tell that the issue has been triaged/prioritized (e.g. add a
priority:{low,medium,high}
tag). 2. We might want to try to find ones where the last comment is from a non-maintainer in particular (as maintainers might just be creating issues to keep track of things to do). WDYT? Maybe we could add it to the community dashboard?
👀 1
e
I've been thinking about this as well - e.g., a MVP process for better acknowledging new issues. I think a labeling scheme + using GitHub Saved replies to help with communicating what happens next may need to be figured out. Happy to also come up with some proposals here.
👀 1
l
OK, that’s great, could you come up with a proposal and we’ll implement it? I think the hardest part is how we can indicate if an issue needs attention or not. Once we can do that we just need someone to go through the list of issues, and that takes time but is not horrible. We could probably even come up with a github action that gets OpenHands to suggest replies that we could vet and post 🙂
👀 1
e
yep will come up with something lightweight by tomorrow
🙏 1
PROPOSAL: OpenHands Issue Triage Process Exec Summary: Classification workflow: All new issues automatically receive needs-triage label, then human reviewers classify info: • needs-info (insufficient details), • enhancement (feature requests), or • bug (broken functionality), each with defined next steps. Bug severity prioritization: Bugs default to medium priority; additional data about bug will increase or decrease that bug’s severity. Enhancement evaluation framework: Feature requests default to under-consideration status while the team evaluates reach, impact, and effort • accepted items get a roadmap label • exception flows exist for out-of-scope requests, and duplicate requests
👀 1
g
• re: consolidated place to see all PRs -- lemme think over this a bit • re: github saved replies -- I love this! I wonder if we can trigger then based on the label. I know we have an auto comment for good first issues. • re: community dashboard -- I think it was a good start, but may need deeper revision with our new repos etc. So the primary goal is to make sure issues that need attention get attention from the OH team or maintainers 👍 • I think a dashboard to surface is good, but will people who need to review an issue know to check there? • What is the most effective way to get attention for a maintainer to review an issue/PR right now? ◦ If the current method is effective, then maybe we can create some sort of process that works best to get their attention ◦ If we have a dashboard people will need to check Laso maybe @better-park-34863 has some ideas/feedback for this too. Thank you @elegant-branch-54423 for creating this triage process. 🙌 Reviewing the doc now
c
What is the most effective way to get attention for a maintainer to review an issue/PR right now?
Effective in getting our attention? Probably Slack DMs. I don’t recommend it, though, other than occasionally, I doubt it scales, and I don’t think this should be it. I think it will not be, and it is not, used by the majority of people who come around OH. I suggest github mentions (and review requests for PRs), as standard way. Github allows to filter notifications on github by mentions and review requested. They can be made automatically by a workflow/agent where possible. (Fallback to other ways to make things visible.) At least I would love to propose to consider this. People do need to talk to us. As maintainers of an open source project, I do think we can’t live in some ivory tower, away from the community, and just committing stuff coming from non-public channels. lolsob
👀 1
g
thank you for the feedback @calm-artist-7206 I feel github SCALES but github is noisy. We could explore some sort of slack channel that is a feed of PRs that need review thinky spin mmm would people mute that? 😆
l
I think one thing is if we can get all of the maintainers to regularly complete all of their pending reviews and slap people on the rest if they aren't doing it that might solve the problem
Specifically from the point of view of getting PR's reviewed, then we would just need someone or even an agent to assign the reviews to appropriate people
c
> I feel github SCALES but github is noisy. Sorry, Jamie, just to clarify whether we're talking about the same thing, what do you mean, noisy with filters applied? Could you perhaps explain why you say "noisy" under the conditions above? Are you talking about mentions or review requested or filters per repo?
For example, it can look like this on github itself
For the record, what Graham is saying is actually happening (-ish) on agent-SDK. We're basically running an experiment getting the agent to assign reviews and see how it goes 😅
l
BTW: do you think the auto-assignment is working well on the agent sdk? We could possibly scale it to the other repos too if so
Hey @elegant-branch-54423, sorry, just got back to this, but the issue triage part looks good. Let’s try it on the agent-sdk repo for now. One small nitpick — what do we do when we have “needs-info” and the poster responds with more info? Move it back to “needs-triage”?
@nutritious-balloon-65098 open a PR on the software-agent-sdk repo that does two things: 1. It makes a github workflow so that all newly created issues get the “needs-triage” label. 2. It adds to the existing “review assignment” workflow (both in the “.github” and “examples/” directories that (a) if there are any issues that have “needs-triage”, that it should follow the workflow, (b) for all issues with “needs-info”, if the original poster has responded with the necessary info, then we move it back to “needs-triage”.
n
I'm on it! Graham Neubig can track my progress here.
Summary I've successfully created PR #1300 on the
OpenHands/software-agent-sdk
repository: https://github.com/OpenHands/software-agent-sdk/pull/1300 Changes Implemented ✅ All requirements completed: 1. Created new auto-label workflow (
.github/workflows/auto-label-issues.yml
) ◦ Automatically adds
needs-triage
label to all newly created issues ◦ Includes smart filtering: only adds the label if the issue doesn't already have
enhancement
or
priority
labels (per your additional requirement) 2. Updated review assignment workflows (in both
.github/workflows/
and
examples/03_github_workflows/01_basic_action/
) ◦ Added
needs-info
handling
: Monitors issues with this label and automatically moves them back to
needs-triage
when the original poster responds ◦ Added
needs-triage
handling
: Processes issues waiting for triage (>2 days), but skips issues that have already been triaged (those with
enhancement
or
priority
labels) Files Changed • Created:
.github/workflows/auto-label-issues.yml
• Modified:
.github/workflows/assign-reviews.yml
• Modified:
examples/03_github_workflows/01_basic_action/assign-reviews.yml
The PR is currently marked as a draft and is ready for your review. <sub>View full conversation</sub>
l
c
BTW: do you think the auto-assignment is working well on the agent sdk? We could possibly scale it to the other repos too if so
I think this is a very good question, I kept thinking about it, and I end up in the same place: it did start to work, but there are a few wrinkles that interfere with the workflow and might be confounding, on agent-sdk: 1. Xingyao is working too much 2. Most maintainers respond to mentions or review requests, but a couple don't seem to respond, ever or I'm not sure how or when 3. Sometimes the agent does things we can maybe tweak... Example 3.1: PR blocked by the reviewer, but the agent asks me (the author) if I'm working on it: https://github.com/OpenHands/software-agent-sdk/pull/1163#issuecomment-3557929378 Example 3.2: I think I occasionally see also: - PR that is Approved - and the agent assigns a different reviewer. WDYT, do 3.x seem like we could tweak the prompts?
l
Thanks for the comments! I’ll think about how to improve this