Gotta love when people decide to spend their tokens for us 😂

Ouroboros

TBH I'm considering to close unknown PRs to the full agentic auto-testing runs I made. If a human wants to help, they should be welcome to come here and talk to us.

On one hand I understand some of these may be intended to be helpful, and may be helpful. On the other hand, maintainer attention is a limited resource. Automated PRs can act like a denial of service attack, flooding the queue making a PR that contains a vulnerability hard to detect. The same psychology of bad actors flooding a persons email or text with notifications during a breach attempt applies. It makes it much easier to miss the real notification.

I think one way to do this is to go the other way around: • close unknown PRs • post a message telling people (if there are people) what they need to know • weekly or something, we go through the list of newly closed, reopen if good • handle the reopened ones In this time, • a whitelist keeps author names who we know are helpful • move to the whitelist those reopened WDYT? Yes, some projects do one way or the other.

Currently my strategy has been: 1. make sure that issues are appropriately prioritized into low, medium, high 2. make sure that issues are given clear acceptance criteria 3. auto-review the resulting PRs for passing the acceptance criteria with a bot 4. only if the bot approves, and only if the issue priority is medium or high, request maintainer review This seems to have mostly solved the problem for me. I get about 3-7 PRs to review per day, which is manageable. Some of the issues that Engel pointed out above seem to be resulting from a gap in our reviewing criteria — the auto-reviewer should check if the PR re-uses the screenshot from the original issue, and automatically reject if so.

I think I read github introduced the ability to add limits to the number of draft pull requests people could make

@calm-artist-7206 this is an interesting discussion. One of the things I like about openhands is their willingness to engage, review, and merge contributions from new and even non technical/ai-assisted users. That tracks from what @limited-student-10747 said about generally not liking whitelists and viewing the problem as a systems one. Blacklisting bad actors could be an easy win, but new accounts are so easy to create. (Do we add a tag for prs created by really fresh bot-like accounts? That's easier to detect) I'm happy to assist on the auto reviewer side of things to improve it if you'd like :) because at the moment I'd like to help reduce maintainer burden and improve the projects security posture.

Interesting points, I appreciate the discussion. Yes, I agree automatically close in many more cases should be fine. 🤔 I'd like to muse a bit over this. On a higher philosophical level, I think of this differently. All PRs that go to main are the responsibility of maintainers of the OpenHands project. We essentially take up the work going forward. Every choice, every addition, every code design and redundant file and deep module and bad contract alike. It's always human responsibility. There is no “machine responsibility”. The agent can’t decide for us. The Before Times In the Before Times, before agents got good, it was worth it for maintainers to go back and forth with all contributors, hint something here, brainstorm something there, and get the PRs in shape. PRs and issues were essential tools in helping people learn the project, understand the intentions, figure out the best way together, take part to it. It may have been time-consuming at times, but it was worth it. First, hey, people were fellow humans, and they wanted to help. Also, because fixes we didn’t have bandwidth for make the project better; because PRs used to show, along other things, what people care about; and, it was worth it because some people will stay around and make the project better. On the other hand, though, in the After Times, after agents are autonomously doing stuff no one human even saw, this changed. The change is painfully real. The After Times Personally, I see a lot of problems with going back and forth with AI PRs (except maybe the one-shots). If there is no human I can talk to, then commenting on AI PRs is essentially like me prompting an unknown LLM, unknown context, unknown harness. There is nothing to expect from doing so, that I couldn’t otherwise do anyway, and I would have done faster. Nobody will learn, if there is nobody at the other end, to do the learning. Nobody will come help, if there is nobody who learned how to help or cared about doing it. I vouch for the PRs I merged. Please let me put this differently, with an example. Last week, I merged 73 PRs I made. I merged 5 PRs from other authors. Because I know I take responsibility for them. And for the wild runs I had my agents make. Because of many many details that give me confidence. For example, I use SOTA LLMs, I know the prompts, I know the skills, I know them, the amazing models of today, I have an intuition where they risk being sloppy and where it’s extremely unlikely to. I grilled them myself and I know the risks. I made multiple rounds of review for some of those PRs; I didn’t for others. As necessary. Including adversarial. Including reading a lot of code. I made live verification runs for all. More importantly, if I may put it plainly, I know that if something breaks, I’ll be there to fix it. So I'm fine with it. I only asked for human review for a couple of things, the rest didn't need another human to switch context to random stuff so I didn't ask them to. I also merged a few PRs from other human authors. Humans who used their agents, and I know they're human, and I know they take their own work seriously, did a nice thing that is helpful, and are confident in their own proposal. We're talking different things if we mix them with AI PRs. None or almost none of those are true about AI PRs. > (a) “unknown LLM, unknown prompt, unknown if a human ever cares, dumped on us to fix it if things break, stuff it in”. Noise vs signal I do hear you, Nelson, and I hope we can keep the project open, yes... for people. I don't care about bots. I mean, what I would like to tell a human - a human - coming over, is like: > (b) "Heeey, welcome! Yes yes and your agent too. Oh, please make sure to specify when your agent is speaking, the other humans will appreciate it, it puts quite some toll on them otherwise. Feel free to use these extraordinary tools we have today to find or read or build whatever you like, we can all do much much more than it was possible before! What did you think of this? It's up to us to make the best of it, aaaand let's play." I strongly believe (b) is the best way. Not (a). However, my argument here is that spending time on (a) makes (b) difficult or impossible to notice. Noise and signal thing. 🤷 --- That said, maybe it's just me because of a particular moment these days. TBH it was interesting to see, so I had a bunch of issues filed and my agents started fixing them, when Claude said "some already have PRs (?!), do you want to pause to review those?". My agents had just posted the issues! 😅 And I said yes... Idk if that was smart 😅 because, a few of those are okay, but the vast majority are slop. Idk, what's the point in pausing for days for something no human actually cared about?

It sounds like the effort, back and forth, the teaching and relationship building, is only really worth it if it's with a human. That makes total sense. Open source is a community effort.

I think these are good points! Just to check, @calm-artist-7206 how many of these PRs that you had to iterate with were the ones that were assigned to you by the review bot after it already approved? I’m trying to understand if the process that I’m hoping will work and cut down noise is working, or not working.