The simple idea of limiting submissions to one per author per calendar year -- no exceptions -- is brilliant. It is totally incentive-compatible at the journal level and would have clear aggregate benefits for the discipline.
I know some people who would pitch a fit about their "right" to submit as many papers as they want, but that's silly. Who cares. Create incentives for people to submit their best work only and watch the knock-on benefits for everyone.
Yeah one upshot of the exercise was to note that there are multiple goals of peer review -- assigning attention and vetting knowledge, for example. And so some "top" journals which prioritize the attention component, it makes perfect sense to constrain the submissions....and then other journals will still be there if we're really worried about slowing down academic progress
And people can always make more journals if space is really a constraint. If the supply of actually good ideas is that big, the discipline can make AEJs and AER-Insightses and Sociological Sciences
Thanks for sharing Kevin! I was actually just writing an opinion piece about volunteering for publication reform efforts, so this is super useful. And also great you all are thinking ahead--not exactly a trait academic publishing is known for.
This is really great. In the arena of "AI tools journals can offer reviewers," I'm really enthusiastic about the model of AI as verifier or editor rather than generator. That feels, to me, like a happy and defensible middle ground: a helpful time-saving productivity device that still preserves the upfront core human function of peer review and explicitly intends to lift quality.
And importantly verification of BOTH author-side claims (necessary, important) and reviewer-side claims. Shirking is one thing, but I think the problem of low-quality reviews is perhaps equally serious. Cue horror stories where at least one disgruntled reviewer didn't read or misunderstand some basic fact already spelled out somewhere in the paper. Get enough of those and it's enough to harm a junior scholar's career progression. In that case, I think many people might even explicitly *prefer* that deadwood Reviewer #2 use AI.
And while it's verifying, perhaps a gentle autocorrection mechanism to tone police Reviewer #2 ("Here's a more helpful way of saying your identification strategy is a joke / utterly lacks novelty...")
Also I think you could make a strong tragedy of the commons argument to restrict (or tax) individual productivity as you suggest--even apart from LLMs. Enforcement of course is the problem. I think you could say that the issue is that over-production degrades collective value; the more we publish collectively, the less the whole is worth. I.e. it's in some sense a version of Goodhart's law except with value not just measure.
Exactly -- if we all seriously internalize the cost of knowledge evaluation, it becomes clear that maximizing production is not actually efficiency-maximizing
Thanks, Kevin, as someone who's been considering editing a journal myself, a lot of your great points have been on my mind. Banning or discouraging AI use certainly seems like a losing, if not outright destructive, gamble, while introducing some submission fees (with potential waivers for precarious scholars) and doing mandatory computational reproducibility upon submission seem like slam dunks.
What's your sense of requiring all submissions to have an AI R3/4 using something like the refine.ink service?
I think that journals can try to outsource some of the evaluation component to an LLM that is designed to check certain, concrete elements of the manuscript. (computational reproducibility AND does the analysis match what the manuscript actually says eg). But we need to have control over the system. The idea of saying "here's what humans are doing, what if AI does literally exactly the same thing as the humans" is exactly the impulse I want to avoid...AI is good and bad at different things than humans are, we should use the respective strength of each.
And if we're doing this, I think it's essential that we don't let YET MORE random for-profit corporations enter into the system of academic knowledge production. It costs us increasingly scarce resources and also means giving up autonomy and control (look what happened when we adopted Twitter as the primary mechanism of communication..).
If we take your argument seriously, then the implication is much more radical than simple adaptation. The problem is not merely that AI will increase submissions, but that it will expose a basic asymmetry that the system has been able to ignore until now: research production can scale, but expert judgment cannot. A journal may receive 20,000 submissions, but the number of reviewers with the depth, maturity, disciplinary memory, and evaluative courage to distinguish the trivial flaw from the fatal flaw, or the fashionable paper from the field-changing one, does not expand at the same rate. In that sense, AI does not just challenge peer review; it reveals that the system was already running on a fragile illusion of scalable judgment.
From that perspective, the likely solution is not broader peer review, but narrower and more rationed peer review. That means triage, filtration, and concentration of scarce human attention on a much smaller subset of papers, while the rest are handled through lighter screening, automated checks, or editorial rejection. But once we admit that, we are no longer talking about peer review as a universal evaluative process. We are talking about a hierarchy of access to judgment. And that changes the politics of publishing. The question ceases to be “How do we preserve peer review?” and becomes “Who gets the right to be seriously reviewed by experts, and on what basis?” That is not just a procedural question. It is a disciplinary and epistemic one.
This is where the issue becomes provocative. If AI makes production abundant while human judgment remains scarce, then journals will inevitably become more dependent on gatekeeping before peer review even begins. Editors, triage systems, submission fees, compliance thresholds, and institutional signals will carry more weight than ever. In theory, this protects reviewer capacity. In practice, it may also harden orthodoxy. Work that is unconventional, early-stage, interdisciplinary, theoretically disruptive, or simply awkward in form may be filtered out before it ever reaches the kind of reader capable of recognizing its value. So the AI era may not only increase volume; it may intensify conservatism. The more the system is forced to conserve judgment, the more it may reserve that judgment for what already looks legible, credible, and familiar.
That is why I think the deepest issue in your argument is not efficiency but epistemic distribution. If expert evaluation becomes a scarce resource, then the future of publishing depends less on whether papers can be produced and more on how attention is allocated. And attention allocation is never neutral. It reflects disciplinary norms, prestige structures, institutional comfort, and hidden assumptions about what kinds of knowledge are worth slowing down for. In that sense, AI may push journals toward a model in which publication is plentiful but serious validation is rare. That would create an academic economy where many can produce, few are deeply examined, and even fewer are genuinely curated.
So perhaps the hardest question is this: are we redesigning peer review for the age of AI, or are we drifting toward a system in which expert judgment is too scarce to function as a common standard at all? Because if the latter is true, then the real transition is not from human review to AI-assisted review. It is from a culture of shared scholarly evaluation to a system of selectively allocated legitimacy. And at that point, the issue is no longer just whether peer review survives. It is whether the very idea of disciplined, field-shaping judgment can remain sustainable under conditions of infinite production.
The simple idea of limiting submissions to one per author per calendar year -- no exceptions -- is brilliant. It is totally incentive-compatible at the journal level and would have clear aggregate benefits for the discipline.
I know some people who would pitch a fit about their "right" to submit as many papers as they want, but that's silly. Who cares. Create incentives for people to submit their best work only and watch the knock-on benefits for everyone.
Yeah one upshot of the exercise was to note that there are multiple goals of peer review -- assigning attention and vetting knowledge, for example. And so some "top" journals which prioritize the attention component, it makes perfect sense to constrain the submissions....and then other journals will still be there if we're really worried about slowing down academic progress
And people can always make more journals if space is really a constraint. If the supply of actually good ideas is that big, the discipline can make AEJs and AER-Insightses and Sociological Sciences
Thanks for sharing Kevin! I was actually just writing an opinion piece about volunteering for publication reform efforts, so this is super useful. And also great you all are thinking ahead--not exactly a trait academic publishing is known for.
This is really great. In the arena of "AI tools journals can offer reviewers," I'm really enthusiastic about the model of AI as verifier or editor rather than generator. That feels, to me, like a happy and defensible middle ground: a helpful time-saving productivity device that still preserves the upfront core human function of peer review and explicitly intends to lift quality.
And importantly verification of BOTH author-side claims (necessary, important) and reviewer-side claims. Shirking is one thing, but I think the problem of low-quality reviews is perhaps equally serious. Cue horror stories where at least one disgruntled reviewer didn't read or misunderstand some basic fact already spelled out somewhere in the paper. Get enough of those and it's enough to harm a junior scholar's career progression. In that case, I think many people might even explicitly *prefer* that deadwood Reviewer #2 use AI.
And while it's verifying, perhaps a gentle autocorrection mechanism to tone police Reviewer #2 ("Here's a more helpful way of saying your identification strategy is a joke / utterly lacks novelty...")
Also I think you could make a strong tragedy of the commons argument to restrict (or tax) individual productivity as you suggest--even apart from LLMs. Enforcement of course is the problem. I think you could say that the issue is that over-production degrades collective value; the more we publish collectively, the less the whole is worth. I.e. it's in some sense a version of Goodhart's law except with value not just measure.
Exactly -- if we all seriously internalize the cost of knowledge evaluation, it becomes clear that maximizing production is not actually efficiency-maximizing
The enforcement costs seem non-trivial though. Maybe in a different peer review/publishing equilibrium.
Thanks, Kevin, as someone who's been considering editing a journal myself, a lot of your great points have been on my mind. Banning or discouraging AI use certainly seems like a losing, if not outright destructive, gamble, while introducing some submission fees (with potential waivers for precarious scholars) and doing mandatory computational reproducibility upon submission seem like slam dunks.
What's your sense of requiring all submissions to have an AI R3/4 using something like the refine.ink service?
I think that journals can try to outsource some of the evaluation component to an LLM that is designed to check certain, concrete elements of the manuscript. (computational reproducibility AND does the analysis match what the manuscript actually says eg). But we need to have control over the system. The idea of saying "here's what humans are doing, what if AI does literally exactly the same thing as the humans" is exactly the impulse I want to avoid...AI is good and bad at different things than humans are, we should use the respective strength of each.
And if we're doing this, I think it's essential that we don't let YET MORE random for-profit corporations enter into the system of academic knowledge production. It costs us increasingly scarce resources and also means giving up autonomy and control (look what happened when we adopted Twitter as the primary mechanism of communication..).
If we take your argument seriously, then the implication is much more radical than simple adaptation. The problem is not merely that AI will increase submissions, but that it will expose a basic asymmetry that the system has been able to ignore until now: research production can scale, but expert judgment cannot. A journal may receive 20,000 submissions, but the number of reviewers with the depth, maturity, disciplinary memory, and evaluative courage to distinguish the trivial flaw from the fatal flaw, or the fashionable paper from the field-changing one, does not expand at the same rate. In that sense, AI does not just challenge peer review; it reveals that the system was already running on a fragile illusion of scalable judgment.
From that perspective, the likely solution is not broader peer review, but narrower and more rationed peer review. That means triage, filtration, and concentration of scarce human attention on a much smaller subset of papers, while the rest are handled through lighter screening, automated checks, or editorial rejection. But once we admit that, we are no longer talking about peer review as a universal evaluative process. We are talking about a hierarchy of access to judgment. And that changes the politics of publishing. The question ceases to be “How do we preserve peer review?” and becomes “Who gets the right to be seriously reviewed by experts, and on what basis?” That is not just a procedural question. It is a disciplinary and epistemic one.
This is where the issue becomes provocative. If AI makes production abundant while human judgment remains scarce, then journals will inevitably become more dependent on gatekeeping before peer review even begins. Editors, triage systems, submission fees, compliance thresholds, and institutional signals will carry more weight than ever. In theory, this protects reviewer capacity. In practice, it may also harden orthodoxy. Work that is unconventional, early-stage, interdisciplinary, theoretically disruptive, or simply awkward in form may be filtered out before it ever reaches the kind of reader capable of recognizing its value. So the AI era may not only increase volume; it may intensify conservatism. The more the system is forced to conserve judgment, the more it may reserve that judgment for what already looks legible, credible, and familiar.
That is why I think the deepest issue in your argument is not efficiency but epistemic distribution. If expert evaluation becomes a scarce resource, then the future of publishing depends less on whether papers can be produced and more on how attention is allocated. And attention allocation is never neutral. It reflects disciplinary norms, prestige structures, institutional comfort, and hidden assumptions about what kinds of knowledge are worth slowing down for. In that sense, AI may push journals toward a model in which publication is plentiful but serious validation is rare. That would create an academic economy where many can produce, few are deeply examined, and even fewer are genuinely curated.
So perhaps the hardest question is this: are we redesigning peer review for the age of AI, or are we drifting toward a system in which expert judgment is too scarce to function as a common standard at all? Because if the latter is true, then the real transition is not from human review to AI-assisted review. It is from a culture of shared scholarly evaluation to a system of selectively allocated legitimacy. And at that point, the issue is no longer just whether peer review survives. It is whether the very idea of disciplined, field-shaping judgment can remain sustainable under conditions of infinite production.