AI Allows More Diversity in the Forms of Social Science

Glad to be back! Nearly three months of paternity leave was a huge privilege, and I’m glad to have both had and taken the opportunity to do so. Everyone is happy and healthy, thanks to everyone for sticking with me, I hope you enjoyed reading the old posts.

Reading through my archive has given me some perspective on what exactly this blog is about: all of the stuff I’m interested in, and only the stuff I’m interested. I’ve explicitly tried to avoid hot takes or anything related to The Discourse. When I started blogging almost six years ago, metascience was niche, navel-gazey; now it’s the ascendent paradigm for thinking about the future of science. Then I got into media theory, especially Flusser — now everybody from Sam Kriss to the Pope is talking about the decline of literary culture and the new orality.

So that was fun. But the intellectual agenda is no longer mine to set, not even on my own blog; it’s currently impossible to think about media theory or metascience without grappling with AI. AI is shaping up to be a central political issue, in terms of electoral, geo-, and cultural politics. But it’s also immediately important for my day job as a professor of political science, so I’ve been focusing my serious writing on AI and metascience.

Today, I present my contribution to an internet-native “edited volume” organized by Rohan Alexander about the future of quantitative social science in the age of AI. This is brilliant meta-scientific institutional work — he emailed prospective authors in late March, organized a workshop that’s taking place today, and otherwise simply set up a lightweight online publishing platform (a “website”) that I’m sure is using an LLM workflow to harmonize and store all of the contributions.¹ This is a methododological contribution, in the sense in which some social scientists (including me) specialize as methodologists. We have to begin to expand the scope of what counts as an epistemic contribution beyond the pdf.

And that’s the theme of my contribution: AI Allows More Diversity in the Forms of Social Science. The whole essay follows — don’t worry, this is written more like a classic Never Met A Science blog post than an academic article. But in even more classic Never Met A Science form, I first want to take a step back and emphasize the AI usage statement, an intensely contested new methodological step in the process of social science, discussed here.

AI usage statement: I dictated a first draft of this manuscript to Claude Opus 4.7 while walking my infant son. I told Claude to clean up the transcript of the conversation and to structure some of the disparate threads while using my exact words as much as possible and demarcating any significant interpolations or transitions. I then went back and edited the manuscript, added citations (I also use Claude to manage my citations in bibtex), and then rewrote or deleted the Claude-labeled Claude-generated text. Insofar as it followed my instructions, all of this text is my own – but there’s no way for me to verify this, nor do I think it especially important.

Introduction

The successful transition to the age of AI must entail greater diversity in the forms that quantitative social science takes. Of course, the same was true of the age of the internet; quantitative social science did not have a successful transition to the internet age. We should have had a revolution in the modality in which we store and communicate scientific knowledge when the internet became standard across universities. It is an embarrassment to our field, and a significant impediment to the pace of social scientific progress over the past thirty years, that we essentially just took old paper journals and put them online with no innovation in form. But AI will soon make it impossible for us to ignore the absurdity of the status quo–especially if enterprising social scientists collectively think hard about how to change social science. This collection of articles is encouraging in this regard, and makes me more optimistic that social science will muster the energy to reject the entrenched interests preserving the status quo.

The present metascientific framework is an extension of the one used in Munger (2026), coauthored with eight other editors of social science journals. We consider how peer review is going to have to change as AI becomes more widespread. The framework requires us to consider different possible equilibria; it’s not sufficient to think about how one subcomponent of the overall academic ecosystem might change. If you only change one thing, there will be adaptation by other components of the system that might dilute the desired effects of the reform, or have other kinds of negative consequences. To do this kind of metascience rigorously, it’s essential to be aware of all of the different functions that peer review serves in order to think about how we might incorporate AI in a way that satisfies all of the relevant desiderata while also producing a stable equilibrium which provides the right incentives for academics.

But in addition to theory, we need trial-and-error. It’s just as absurd to believe that we can perfectly predict how AI will be incorporated into social science as it is to predict that the status quo will persist. The essential first step is to try things out; I discuss three ways in which AI allows for innovation in knowledge production (meta-ontology, time, and rapid empirical iteration). But to give practicing scientists the appropriate incentives to try out these new forms, we must overcome an age-old tyranny.

The Tyranny of the PDF

The biggest bottleneck to the exploration of AI-enabled formal diversity is the uniformity of the outputs for which social scientists can get credit. I refer, of course, to the tyranny of the peer-reviewed PDF. There are many different types of epistemically relevant, valorous, valuable actions that social scientists take, but the only one for which we are officially recognized and rewarded is the peer-reviewed PDF.¹ The fact that all social science has to be routed into this modality of communication causes us to flatten the types of epistemic contributions that we make, and many of them are really very poorly fit for being stored and communicated in this modality.

Before getting to the ways the PDF is insufficient, it’s worth delineating the bundle of functions the pdf serves. The static PDF functions as both an archive of what actions the scientist has taken and as a reference to previous research products: the literature review, the theoretical arguments, the methods section summarizing different statistical tools taken, the statistical results, the conclusions, the references. All this made sense to bundle together, because the implicit arrangement at the point when the academic paper became the standard was that there were many humans who were going to read these PDFs.

But with the internet and especially with AI, there’s no need for every epistemic contribution to always include every one of these elements. By unbundling them and allowing for epistemic contributions to take different forms, social science can become radically more efficient.

I propose three dimensions on which those new forms can transcend the PDF. The first is the introduction of a better-defined, more explicit meta-ontology of social science. The second is time: the fact that knowledge can be kept up to date rather than stored as a single static PDF. The third is aggressive empirical exploration within a well-defined space of experimentation or quantitative study.

Meta-Ontology Over Narrative

The first problem with the PDF is that it presents each paper as a single contained narrative. Sure, the authors draw from previous studies for the purpose of making a theoretical argument connected to their specific empirical case. It’s not meant to be completely unique, but ideally, both the theoretical arguments and the empirical cases are understood to be in some way novel through the production of a narrative in the academic paper. Social scientists understand that they need to have a story. This is especially true for the most important paper they will ever write: their job market paper.

The bundle-narrative form treats the task of writing these papers as something which involves craft, for which taste is necessary. I think this is a valuable art form, one which I very much enjoy – but we’re seeing it devalued and commoditized because it is the only permissible form of epistemic contribution. We might collectively write far fewer of these narrative PDFs, and then focus our attention on these more careful, artisanally constructed arguments. The diversification of the forms of quantitative social science will in fact enable us to appreciate the narrative PDF for what it actually sets out to accomplish.

The narrative PDF requires authors to play a kind of trick: to convince readers that all of the components of the bundle hang together correctly. This particular RCT allows us to learn about the “effects of education on support for the far right”; this survey experiment on an online convenience sample does in fact generalize to a population from which it’s not randomly drawn; this other game theoretic model in which one of the actors is labeled a “voter” does in fact relate some way to what happens to people in the voting booth.

Each pdf weaves these components together into as coherent a narrative as it can manage. But contemporary methodology pays very little attention to the individual linkages. We know how to evaluate each component in the abstract; the question of whether they hang together is more of an art form.

We need a more well-defined meta-ontology of social science. We need a database that tracks each of these components separately, as well as how they have been combined. We can take from previous studies the different concrete manifestations of theories, different dimensions of variation that have been theorized to be relevant for moderating or mediating a given phenomenon, a list, say, of theoretical interventions or exogenous shocks which have been studied.

Consider the fantastic work by Mellon (2025) about the diversity of PDFs in which rainfall has been claimed to be an exogenous shock – each of which explicitly assumes that rainfall only has effects through the pathway that that specific pdf claims it operates through. This meta-scientific work had previously required painstaking human labor, but once we’ve begun to define the meta-ontology, this is an ideal task for AI, which can allow us to go back and do it at scale, extracting the relevant components of all published quantitative social science work.

With this database, we can begin to figure out how each of these components are working in the aggregate. For example, analyzing just the distribution of empirical tests with a given method has produced important evidence about the overall functioning of that test, if the statistical significance rate matches our theoretical expectations (Brodeur et al. 2016; Brodeur, Cook, and Heyes 2020). More prospectively, this database will allow us to identify methods or evidence which have become out of date (see Section 4), what combinations of components have been underexplored, or what spaces have seen sufficient exploration to begin to approach some non-qualitative synthesis of the knowledge (see Section 5). The pedagogical value would also be immense, giving graduate students more direct contact with the raw material they will need to work with.

Metascience to date has been constrained by the tyranny of the pdf. By trying to do metascience while accepting the implicit meta-ontology of academic publishing, where the fundamental building block of quantitative social science is the study and that “the study” is stored in the pdf, we have reified into givens what are merely symptoms of what social scientists actually do (Latour and Woolgar 1986).

Formalizing our meta-ontology will make possible new forms of knowledge synthesis, which I believe to be the most pressing problem in contemporary methodology. Synthesis at present either can take the form of meta-analysis: if we have a phenomenon or theoretical construct which has been studied many times in many places, but which we believe, or trick ourselves into believing, is sufficiently similar across times and places, we can simply plug these studies into some kind of statistical machine and perform quantitative meta-analysis. Or else it’s done qualitatively, in fact, where people read different PDFs and come up with their own understanding somehow of the knowledge contained there. Despite my glibness, these are both valuable parts of the process and cannot immediately be improved for many areas of social science. I’m not arguing to get rid of any of these things. But I am arguing against this monopoly of form, in favor of embracing the diversity of forms that are possible with AI.

Time

The second problem with the PDF is that it is static. This matters in two contexts. One is related to constraints in the production of knowledge, and the other is the premise that the world itself is not changing very much. Munger (2023) argues that we should speed up social science and that we need to be aware of how changes in the world change our evaluation of existing knowledge, but AI could allow us to store knowledge in a way that treats time as a first-order concern for social science.

When a peer-reviewed PDF is published, it summarizes the data that exists up to that point, references the literature that exists up to that point, uses methods which have been validated up to that point, and then is treated as a static, final, permanent contribution to the academic literature.

Perhaps the largest metascientific reform of the past decade comes out of the replication crisis (Collaboration 2015; Nosek and Errington 2020). This reform is fundamentally backwards-looking, when I think that AI-powered social science can and should be forward-looking. Replication is the idea that we should be able to directly reproduce what has been done before. But given the fact that the world is changing, the literature is changing, the data are changing, and our methods are changing, I don’t actually see why we should care very much about being able to do what was done before. Instead, we should take the logic of replication and apply it to a changing world. We should care about the ability to maintain, for example, the core theoretical logic of a paper while still updating at least the data; even more provocatively, we can potentially also update the methods and the theory.

Consider public opinion panel surveys like the ANES, where we have the same questions being asked every presidential election, and we have some measure of affective polarization (Iyengar, Sood, and Lelkes 2012). Rather than needing to write a new paper, peer-review it, and make some theoretical modification, we should simply say: we have this measure of affective polarization, this is a dynamic dashboard where we can see how this measure has changed over time. It should just be a kind of static fact that everyone in the world can reference. It doesn’t meet the standards of theoretical novelty to merit a whole new bundled PDF, but these descriptive facts are essential inputs to the rest of quantitative social science (Munger, Guess, and Hargittai 2021). Ironically, we tend to outsource the production of epistemic facts to other actors like journalists or the Pew Research Center, even as their outputs rack up hundreds of citations and go on to set the academic agenda (Smith and Anderson 2018).

We should be able to update the results in a given empirical paper with contemporary methods.² This would mean that the “paper” could be stored in a dynamic way online, where what’s actually the contribution is a data pipeline and a codebase which can be updated automatically with AI. We should be concerned about archiving the knowledge stored in this way; the PDF does have that function, and it is interoperable with libraries that are institutions tasked with archiving knowledge.

This dynamic updating/discounting is impractical or even ill-defined for many types of work undertaken by quantitative social scientists. Again, I’m not arguing that old forms must be totally abolished or that new forms be made universal or mandatory; this is rather a way in which AI can be used to make new kinds of social science possible.

High-throughput empirical exploration

The third problem with the PDF is that it flattens the magnitude of the empirical evidence “under the hood.” Quantitative social science is forced through this fundamentally qualitative layer, even though quantitative meta-analysis basically ignores the pdfs and instead works directly with the data to understand distributions of effect sizes.

Building on the formal meta-ontology, we can identify empirical components of a given phenomenon of interest where we have a relatively static, sophisticated theory and a well-defined ontology of the different elements and how they interact. The relevant margin for scientific progress is to increase, perhaps by orders of magnitude, the amount of empirical evidence being brought to bear on this problem. I’m inspired by the high-throughput virtual lab being implemented by Duncan Watts and his co-authors (Almaatouq, Becker, et al. 2021; Almaatouq, Alsobay, et al. 2021).

In the case of Houghton and Watts (2025), they study the informational properties of deliberation by conducting studies with cross-partisan dyads which disagree about some phenomenon. Each study involves an online deliberative discussion, and randomly varies some number of theoretically relevant parameters. The outcome is well-defined by both properties of the discussion and self-reports by participants. At this margin, we don’t need theoretical innovation; we simply need to rapidly iterate through the parameter space of the different empirical manifestations of those theoretical dimensions.

It’s not necessary to produce a new peer-reviewed PDF which gestures at some theoretical novelty with each of these empirical components. We are seeing a move towards this greater conjunction of empirical studies into individual mega-studies, primarily driven by the competition for publications at elite journals. Where once you might be able to publish a single survey experiment, now you have to do several of them, or perhaps a very large-scale iterative field experiment studying persuasion effects (Coppock, Hill, and Vavreck 2020; Voelkel et al. 2024).

These are steps in the right direction. It’s obviously much better to avoid breaking these into individual PDFs, but there’s not a principled reason why this is happening; there’s just more competition for publications in elite journals, so authors are packing in more data to distinguish themselves. Worse, the form remains the same; each pdf remains a single narrative. Authors engaged in this form have to make the case that all of the data are alike, that they all make sense as part of the narrative bundle.

What we should be able to do is not try to immediately summarize all of this data into a single outcome, to smooth out the differences or to embrace the narratively useful fiction that effect heterogeneity doesn’t exist. If the theoretical and ontological space were well-defined and the epistemic contributions more continuous, scientists could instead collaboratively work on these problems, have multiple labs be able to make independent contributions that are commensurable without being interchangeable. By being more explicit about our ontology, we can radically scale up empirical progress towards a deeper theoretical understanding.

Interoperability and careers

These are independent, individual reforms, and are thus according to our premise insufficient for thinking about the future of social science in the age of AI. One of the key questions is how these changes will be made interoperable with the other components of social science. For established academics, I am optimistic that the epistemic value derived from this formal innovation will make it stick; it’s hard to ignore data that speak directly to questions of scientific interest. And diversity means that people can keep producing bundled, narrative pdfs if they want; my expectation is that high-quality contributions in this form will continue to be valued, while replacement-level pdfs will, hopefully, be replaced by more efficient forms of epistemic contributions.

But it is not excessively cynical to say that it is a first-order concern for people who do not yet have permanent jobs to figure out how any reforms to the institutional setup will affect their chances of getting a permanent job. How does the current pipeline of training new graduate students into the profession work, and how do we figure out the selection mechanism in an increasingly competitive environment with fewer and fewer permanent positions?

Pedagogically, I believe that the proposed innovations will be immediately helpful and demystify the components that make up quantitative social science work. By discounting older papers using invalid methods, the relevant literature to master will be less tilted towards the distant past. And by making visible well-defined problem areas, the collaborative empirical exploration can give early-career researchers a space with a lower barrier to entry for an epistemically valuable contribution.

But “making things easier” in a hypercompetitive marketplace can actually end up making things harder. Ethical scientific reform is seriously handicapped by a concern which predates AI, but which AI will make unignorable: we simply have too many graduate students for the number of permanent academic positions we have, and unless we figure out how to address this problem, any mechanism for distribution of career rewards and incentives will be plagued by overcompetition and the dissipation of effort. The default move is to just let’em fight it out, but I find this deeply unsatisfying, even immoral. Unfortunately, I cannot see a solution emerging from within the scientific system; the problem has to do with material conditions external to whether or how we produce pdfs.

Conclusion: Talking

In the spirit of reflexivity, it seems important to point to the AI usage statement at the beginning of this article. The process which led to the text you’re reading in this PDF would have been impossible without AI.³ On the other hand, this probably should never have been a PDF at all. For open-ended questions, particularly related to the metascience of how social science should be done, verbal deliberations allow for much faster feedback and iteration through ideas.

But this doesn’t mean that the half-baked intuitions or preferences of senior faculty tossed off over lunch need to be taken seriously. We currently apply massive amounts of rigor to the pdfs we produce but almost none to the way we talk – again, with the vertiginously important exception of the “job talk”. Other epistemic traditions take verbal contributions very seriously; the Supreme Court hears oral arguments, and while we don’t have to go quite that far, we should elevate and archive (with video) verbal metascientific debates about the state of the field. Conference presentations are useful but generally still too informal, and generally not archived. Again, I’m not saying we need to livestream literally every 12-minute talk at APSA, but that this form should be taken seriously as an epistemic contribution.

The baby’s getting big — melon to scale.

---

1. The fact that some of these pdfs are printed is more of a historical curiosity than epistemic necessity, though see the discussion of knowledge archiving below.↩︎

2. More provocatively, if the methods a given paper used in the past are no longer valid, we should discount the findings from that paper: it wouldn’t be published today, and therefore the results of the original paper are no longer valid. This one could get very nasty, interpersonally, and it’s definitely not something that senior academics would endorse, but we already do this implicitly, and I’m not sure that it’s a bad idea.↩︎

3. Or, rather, it would have been entirely normal several technological ages ago, when social scientists dictated their thoughts to typists or secretaries who inevitably dealt with the task of translating spoken thoughts into written text. It is now banal to point out that seemingly novel technologies in fact represent a return to older arrangements, but the point does take some of the edge off of the petty reactionary view that the way we happened to be doing science in the recent past is sacrosanct or inevitable.↩︎

1. Aside: There’s simply no reason at all for corporations to be involved in this. I’m becoming increasingly radicalized on this point (and I wasn’t exactly a moderate before): if we cannot manage to excise the for-profit corporate leeches from the academic ecosystem, whose incentives point always for a flood of slop, we simply aren’t trying to adapt to the age of AI and deserve the societal irrelevance that eventually comes for stagnant institutions.