Replies: 6 comments 5 replies
|
Thanks for raising this -- I agree the influx of code is likely LLM generated, but I don't think that is the core problem itself. I think the core problem is one of fairness to other contributors. I see generating many unreviewed PRs expecting we will review them is similar to spamming an email list. Given reviewer time is a limited, finite resource that is always used, such spamming behavior you haven't even bothered to review them yourself is somewhat like stealing attention from others. We also do have our AI guidelines Though it was hard for me to find even when I knew they existed -- maybe we can make them easier to discover (e.g. put it in agents.md to direct agents to it) and add a sentence about not generating a bunch of unreviewed PRs. |
We could indeed test this out now and see if it discourages the high volume PR behaviours.
Recently read an interesting article about this: https://blog.rust-lang.org/inside-rust/2026/07/06/unite-for-clippy/
|
|
My motivation as a maintainer with a PR review is two fold, adding (or improving) functionality and potentially adding new community member, a person which has interest in project and hopefully replaces me or work alongside me in the future. In the age of LLM adding bunch of code easier than reviewing it, having a LLM generated PR only saves me few tokens, and probably not much time and not much cognitive effort. I guess if we focus on building the community instead of building codebase this might help us prioritising PRa; if a person wants to get the code reviewed, a commitment to the same goal has to be demonstrated. I believe this is somewhat aligned with previously shared links. |
This would be a neat idea. |
|
Another issue I’ve observed is that, even if we exclude the obvious AI slop and only consider carefully written PRs, the bandwidth for producing them still seems to far exceed our review capacity. It would be great to figure out ways to organize the work more efficiently. |
Turso database has rule for maximum 3 prs. Maybe we can adapt similar. Even 2 would be enough. |
I have noticed this too. The obvious fix I see for this is add a sentence is issue template that if you are already planning to work on this issue, assign it to yourself if have access or comment |
|
Although this doesn't directly solve the problem for people who want to attempt to leverage usage of LLMs to spam PRs everywhere and only get a few reviewed, but both ghostyy (less good imo) or bevy (more intresting). Have I think found intresting solutions to try out or adapt pieces of. Ghostyy (https://github.com/ghostty-org/ghostty/blob/main/CONTRIBUTING.md) I think there is some other strain here for maintainers to read vouch requests, although this scales down the amount of work vs PR reviews significantly. For Bevy (https://bevy.org/learn/contribute/helping-out/opening-pull-requests/) most notably I'm refering to step 2 here "Other community members review and comment in an ad-hoc fashion.". I process this as make the community itself do the first steps of reviewing before it is handed over to someone who can get it merged in aka most of you guys. This obviously has a training problem of getting people comfortable to do reviews, but if it's one of the few ways for non committers to get their PRs merged, I think they will go and try to review other people's code in the hope theirs gets reviewed too. Also having the side effect that trying to review a PR is a surprisingly good way to learn a codebase. |
|
I really like the idea of bevy https://bevy.org/learn/contribute/helping-out/opening-pull-requests/
@Jefffrey and I have also discussed version of this and how to use labels in past discussions too:
I think it is worth trying out (the S-Ready-For-Final-Review is particularly interesting to me -- maybe we can try to start with something like that |
|
Concerning this conversation of LLM usage by new contributors, the most effective path forward is to take their leverage away by putting the correct road blocks along the way while still allowing responsible LLM usage. Bevy's approach to me seems highly effective at while also have a potential side benefit of getting some rudimentary reviews done before maintainers are involved. While also giving people both the motivation and an incentive to review other new contributor's PRs and hopefully organically creating a community that isn't afraid of PR reviews which is very common both in Open Source and Enterprise |
|
Firing an initial PR to update some of our guidelines per discussion here: Feel free to suggest any changes to the current wording there 👍 |
Uh oh!
There was an error while loading. Please reload this page.
I've been seeing cases where a first time contributor starts submitting multiple PRs at once, especially across a variety of components in the codebase. It leads me to believe people are getting LLMs to auto-create PRs from issues and are mass sending them, in some cases not even checking if there is already a PR linked to the issue.
This gives low confidence that the PR author has actually reviewed and understood the changes, especially if they are creating PRs to other repos at the same time. I wonder if we should have a policy on how many open PRs a first time contributor should be allowed to have, similar to what polars has: https://docs.pola.rs/development/contributing/#first-time-contributions
There was a recent discussion in the arrow mailing list, though its more about a setting to apply to all non-committers: https://lists.apache.org/thread/fsfply47z9rb6bz0x50b2whlhg6hnflr
Though this reply from that mailing list is quite relevant:
All reactions