https://docs.all-hands.dev/ logo
Join Slack
Powered by
The attempts we made to keep a list of `good-first...
# feedback
c
The attempts we made to keep a list of 
good-first-issues
made sense IMHO, in "The Before Times", that is, before the agent got too good and agents became widespread. Their point was to familiarize the human with the codebase a little bit, with the repo conventions, and a slight brush with how its structure looks like. This doesn't fill its former purposes if agents may do end to end everything, even from the repo clone -> to the PR. I guess we could: • give them up, might not be worth the labeling effort if it's human effort • maybe only set up the agent to label this, periodically, and let it try to have an impact... even if it probably doesn't, anymore • make a policy requiring human written PRs and/or human written description/testing, in the hope that it does still achieve its goals. WDYT?
🤔 1
👀 1
cc @gentle-apple-32922
g
1. Are our community members using models with agents that a good enough to discern what issues are appropriate for a first timers to tackle? 2. I think it is fine if we have OpenHands label good-first-issues since it doesn't take human time per se. a. I can set up an automation to do so on times intervals
• make a policy requiring human written PRs and/or human written description/testing, in the hope that it does still achieve its goals.
I don't think this one is realistic 😢 Is anyone hand writing anything anymore?
c
Re: 1 Pretty sure… SOTA or close. Though I’ve seen Cursor Agent, Claude Code, codex… so different agents, and Sonnet 4 or 4.5 … close
Ha. Re: human writing policy Some projects are doing that. Others are doing 1 or 2. Yeah I think those with 3 are maybe a bit behind, or reacting strongly - you know, denial stage 😅
lolsob 1
But the strange question in my mind is, if we don’t require human effort, even minimal, then why are we doing it and how will people catch up with more complex stuff?
g
if we don’t require human effor
for PRs or for labeling GFI
c
For PRs
g
i think it is reasonable to ask a human tested it
I test my PRs 🤷
❤️ 1
I don't think it's reasonable to ask them to write the PR description
There should be some sort of video, screenshot verification in the PR
c
The reason why I’d like maybe a paragraph in PR description, is to see what the human thinks about the issue…
g
Now Xingyao was talkinga bout OpenHands may get to the point where it can test and provide a video 🤯
c
It can post screenshots, I haven’t tried video but Rohit has made recordings of the SDK/CLI
g
[logging off for the day in a moment, but will def reply back tomorrow 🤝 feel free to send whatever in the thread dog cool ]
c
Some projects do something like: • before you submit your first PR, talk to us • either make an issue • or talk in Slack/discord • maybe, say what you think about the issue you want to solve, and why you care, so we see a sign that you’re in charge, not only your agent
@nutritious-balloon-65098 Find the good-first-issue label, and then find a workflow that attempts to set it, do we have such workflow? Clone yourself both openhands and extensions repos, from openhands org. If we don’t have one, take a step back, understand the oh repo, and its other workflows including for triage and pr-review, figure out the criteria and include it in triage prompt. Open PR and link it for us. Note: when I ask you to summarize, don’t summarize, just repeat what you just told me. Really, in this session don’t summarize, only repeat.
n
I'm on it! can track my progress here.
Repeating what I just told you: I found the label and checked the workflows. • The label exists in 
OpenHands/OpenHands
as the standard GitHub label 
good first issue
with description 
Good for newcomers
. • We do not have a workflow that applies that label automatically. • The only matching workflow is 
.github/workflows/welcome-good-first-issue.yml
, and it only runs after the label is added. It posts a welcome comment; it does not set the label. • I also checked the surrounding repo automation: ◦ 
.github/workflows/issue-duplicate-checker.yml
is the closest issue-triage automation, but it only checks for duplicates / overlap. ◦ 
.github/workflows/pr-review-by-openhands.yml
and 
.github/workflows/pr-readiness-confirm.yml
helped anchor the criteria around bounded scope, clear testing, and reviewability. • I cloned both repos from the OpenHands org: ◦ 
/workspace/project/org-clones/OpenHands
◦ 
/workspace/project/org-clones/extensions
Since there was no auto-labeling workflow, I updated 
ISSUE_TRIAGE.md
to add explicit 
good first issue
criteria, including: • it should be narrow and single-issue in scope • the expected outcome should already be clear • it should stay within a bounded area of the repo • it should avoid deep architecture / cross-repo / enterprise / migration / security / infra work • validation should be straightforward with clear “How to Test” steps • reviewers should not need special credentials or unusual environments I ran: • 
make install-pre-commit-hooks
• 
poetry run pre-commit run --files ISSUE_TRIAGE.md --config ./dev_config/python/.pre-commit-config.yaml
I committed, pushed, and opened the PR here: • https://github.com/OpenHands/OpenHands/pull/14498 Branch: • 
clarify-good-first-issue-triage-criteria
Commit: • 
53562c94b
r
@calm-artist-7206 I would support a “human written PR and comments” policy. Though I'll be the first to admit I don't always do that. I should. I suggested the same policy at Mathlib and they adopted it. It may be harder to enforce here in the OpenHands ecosystem, though.
@modern-fireman-99755 has been very kind reviewing my PRs, but I bet he would be appreciative if they were more concise 😅
c
Thank you, Nelson. I’m looking at this, maybe it’s not what you meant but sounds like this kind of policy? Lean - mathlib 4 - Use of AI
Note that using an LLM when writing comments on github or Zulip is not allowed: use your own words.
When reviewing PRs, we are particularly concerned that the pedagogical value of the reviewers work is wasted if there is not a human contributor actively learning.
Members of the review team will summarily close without comment any low quality PR produced using LLMs, especially if the author has made little effort to directly engage in the community in a discussion about its merits before opening the PR.
If you just want to help and not put in the learning effort, making a PR to mathlib is counterproductive: the effort required from the mathlib maintainers is larger than the benefit, because the time used to improve the quality of the code will not result in a better quality in future PRs.
Re:
@calm-artist-7206 I would support a “human written PR and comments” policy.
Would you include PR code, or could we say: “human written PR description and comments”?
r
I would not include code. Mathlib is experiencing an influx of LLM generated proofs, and the majority of them suffer from poor proof construction and APi design, so they are not generally composable by future mathematicians. For a library that's sole purpose is to be generally useable for future proofing, that's a huge problem. For OpenHands I would suggest pr description and comments only. Perhaps provide a concise example. An AI will output a huge block of text that, for the most part, could be cut by 2/3s without losing the ability to describe the PR. A human can do that, and it helps both learn and demonstrate understanding of what they are proposing, even if the agent did the coding.
nod 1
c
I used to do something like, use a PR description structure where the agent wrote the body, but I inserted a text on top before submitting it for review, e.g.:
HUMAN:
This PR proposes... stuff
---
OpenHands-GPT-5:
stuff-stuff
I'm not doing that anymore, specially since we also added a note on the bottom of the agent's output to say it's agent output; it now says:
This text is written by an AI agent on behalf of enyst.
But maybe something like that could help?
👍 1
r
Yeah. I don't think any major policy change is needed, just a nudge in that direction.
I don't want to us to feel hostile to ai assisted work like mathlib is becoming. But a clear role/responsibility distinction may be useful and also help protect maintainers/reviewers time.
l
Personally I don’t care if code or a description is written by a human or AI as long as it’s good 🙂
❤️ 1
➕ 1
w
I used to do something like, use a PR description structure where the agent wrote the body, but I inserted a text on top before submitting it for review, e.g.:
I really like this pattern
I like being able to see quickly if part was written by a human, since that is generally MUCH higher signal to noise
👍 1
l
So IMO the better thing is that we have a clear definition of “good” and make all PRs adhere to it, I think we’re moving in that direction with stricter PR templates.
nod 2
➕ 1
g
I think the 
[ ] A human has tested these changes.
At the top helps insure a human had some oversite and say on this. I worry the writing a description by human exclusively could not cover it all. I do like the concept of expanding the top and having a human summarize in their own words. But just want to make sure our maintainers can follow this rule like our contributors.
🙌 1
w
Additionally I think it'd be good for the template to have a space for a human description
➕ 1
Often I just write a sentence like "hey this was broken for me, this is the simplest fix I can think of"
even if we don't require it
h
I can't remember where I saw it but someone mentioned that all PRs will be auto closed unless the user has previously created a human written issue that has gotten a specific label or comment from a committer. It stopped bot PRs and was easy to automatically enforce. I'll see if I can find the source for reference.
👀 1
w
I don't love making humans jump through "I am not a robot" hoops 🙂 Are bot PRs hurting us badly today?
💯 1
Maybe we need a "slop detector" AI reviewer 😂
lolsob 1
g
I worked with @chilly-france-44439 to help implement measure with OpenHands to try to insure the PRs were high quality and a real person wrote them
n
• The follow-up request was addressed: I found the PR-opened comment workflow and told you what it says. • No new repository changes were made since my last message. • There are no extra changes to revert. Yes — the main one is 
.github/workflows/pr-readiness-confirm.yml
. It posts this comment when a PR is opened or marked ready for review, but only for non-draft PRs from 
FIRST_TIME_CONTRIBUTOR
or 
NONE
authors:
👋 Thanks for opening this PR!
Your PR is marked for review. Can you confirm with a 👍 reaction on this comment or a reply that this PR is ready to review?
If no confirmation is received within 7 days, this PR will be converted back to a draft.
It includes this hidden marker at the top: There are also two other PR-commenting workflows, but they are more conditional: • 
.github/workflows/pr-artifacts.yml
posts only if the PR contains a 
.pr/
directory: 📁 PR Artifacts Notice This PR contains a .pr/ directory with PR-specific documents. This directory will be automatically removed when the PR is approved. For fork PRs: Manual removal is required before merging. • 
.github/workflows/enterprise-check-migrations.yml
posts only if the PR touches `enterprise/migrations/**`: ⚠️ This PR contains migrations. Please synchronize before merging to prevent conflicts.
Done. I updated from 
main
, created a fresh branch from the updated 
main
, made the requested changes, pushed them, and opened a new PR: https://github.com/OpenHands/OpenHands/pull/14527 What I changed: • verified the existing PR template already had the human-tested checkbox • added 
HUMAN:
above that checkbox and 
AGENT:
below it at the top of 
.github/pull_request_template.md
• updated 
.github/workflows/pr-readiness-confirm.yml
so it now checks all non-draft PRs on 
opened
, 
edited
, and 
ready_for_review
• added a new 
check-human-context
job that looks for non-empty text between 
HUMAN:
and 
- [ ] A human has tested these changes.
• if that HUMAN block is blank, it posts a reminder comment • if that HUMAN block is filled in, it adds a 👍 reaction and removes any earlier reminder comment • added 
issues: write
permission so the workflow can add that reaction with the default workflow token • recorded the new behavior in 
AGENTS.md
Validation I ran: • 
poetry run pre-commit run --files .github/pull_request_template.md .github/workflows/pr-readiness-confirm.yml AGENTS.md --config ./dev_config/python/.pre-commit-config.yaml
• a small local regex check for the HUMAN-section parsing logic Commit: • 
a5f36aab3
— 
Require human context in PR descriptions
🙌 1
PR ready: https://github.com/OpenHands/OpenHands/pull/14527 • Final PR SHA: 
1fe4b98a20ba5b80abd7f2f68478fa1a0e3810da
• CI status: 
18/18
passed, 
0
failed, 
0
pending • Review status: 
APPROVED
by 
all-hands-bot
• Mergeability: 
CLEAN
• Draft status: converted back to ready for review • Fixes pushed during iteration: ◦ 
6e54ab396
— 
chore: address PR review feedback (#14527)
◦ extracted the checkbox text into a constant for the HUMAN-section regex ◦ added 
warnOnly
handling around the GitHub API calls used by the reminder/reaction logic ◦ kept the 
.trim()
behavior as-is and replied explaining why it already handles blank/whitespace-only HUMAN sections • Flaky retry cycles used: 
0
• Review threads resolved: 
3
• Remaining unresolved failures/comments: 
0
• QA layer: none present in this repo, so nothing to wait on I also: • rebased once because the remote branch moved underneath the PR branch • re-requested bot review after pushing the fix commit • marked the PR ready only after CI went green and unresolved threads reached zero
c
Maybe like this
🐕 1
I don't love making humans jump through "I am not a robot" hoops 🙂
Are bot PRs hurting us badly today?
So it’s a little complicated. Like a month ago, • agent-generated PRs were some over 90% • claw or maybe human, but who likely doesn’t use OpenHands: 50% or more This is 
PHclaw
. Made in April 2026. 100 repos. “First-time contributor” to OpenHands. lolsob
👀 1
r
From a security perspective, I have mixed feelings about autonomous claw contributions. But also, cool. confused dog
😅 1
c
However funny that is, it isn’t actually the most - the most I think are still human accounts, but it’s not obvious the account holder uses OH or speaks to us.
r
@calm-artist-7206 I think, perhaps, there's an aura of "awe" around OpenHands that makes bot-first communication seem lower-threshold for contributors. After getting to know you all, I see you're just a bunch of 🧸's
ty 1
m
I don't know how I feel about "We want PR authors to tell maintainers, in their own words, what the PR does." Because to be honest, agents write better commits and PRs than humans now a days 😂
c
🫠 they're additive, though, the agent description is supposed to be below the human text. TBH I do think the human author should understand the PR they're proposing, even if it's in a few sentences.
m
Yea, 100% they should definitely understand 🙂 Just don't make fun of my sentences, the agent will sound smarter than me
❤️ 2
c
nanookclaw
is trying to contribute to the CLI. So far, this attempt to showcase a proof is hilarious: https://github.com/OpenHands/OpenHands-CLI/pull/750#issuecomment-4534906013 • it made a .gif that says the 
grep
visualization passed • it made a video, where the funky 
grep
runs in the shell, not in the CLI 🫠
confused dog 3
😂 1
r
bless their heart
m
The user posted another video of the CLI but seems a fake. A good one 🫣
l
this was too good, had to share: https://x.com/gneubig/status/2059049443706462484
lolsob 2
2 Views
Open in Slack
PreviousNext
Powered by