Again, the costs of doing quality causal research are high. Consider two of the most innovative and ambitious research programs in the social sciences: EGAP’s Metaketa Initiative; and the Many Labs program in Psychology, broadly associated with the Center for Open Science.
These are truly valuable contributions to human knowledge. They also represent dozens of years of combined effort from some of the best-trained social scientists in the world and millions of dollars. That's just what this kind of knowledge costs.
This implies that the aggregate amount of causal social science knowledge that academic researchers can produce is simply incommensurate with the scope of our inquiry. We’re asking too many different questions. Either these projects are overkill (I strongly disagree that they are overkill!) or too many individual scientists are producing local, ephemeral causal knowledge that is being thrown over a wall to no benefit to anyone.
The only long-term solution to the expense of causal knowledge is institutional (and thus political). EGAP and COS are examples of the kind of institutions that can structure scientific production towards producing sufficiently co-ordinated causal knowledge that it can be put to use. Other examples include NYU’s Center for Social Media and Politics and PSU’s Center for Social Data Analytics, to name two from which I have benefited greatly. The binding constraints (other than cold hard $$$) are the asynchronous time cycles of: institutional creation; undergrad time; grad student time; postdoc time; assistant professor time; senior professor time; project design; project implementation; and academic publication.
All of these actors could coordinate outside of a formal institution, but there are crucial frictions preventing them from doing so. I am of course riffing on Ronald Coase’s Theory of the Firm:
Coase, building on his famous theorem that never obtains in practice, explains the existence of firms from the lens of transactions costs. Independent actors, coordinating through the price mechanism, should be able to allocate their labor efficiently. The presence of firms is a puzzle, in that individual actors are more aware of local conditions and are less susceptible to principal-agent problems than hierarchical, sclerotic firms. But the costs of bargaining and creating contracts for every step of the production process is far too high, and firms exist to eliminate these frictions.
These transactions costs are serious for academics. In particular, we are a self-selected bunch, willing to forego income for autonomy. Tenured academics (in theory) have high reservation “wages,” making formal coordination unnecessary.
The benefits, however, are real. There are huge gains from specialization and division of labor. The presence of working groups, all working on a common topic, pushing forward in multiple directions and getting/giving mutual feedback, accelerates processes which can otherwise be glacially slow. Personally, I love working in a context like this, warts and all.
But the reason I’m writing about these formal academic institutions is that they are absolutely necessary to do research at the scale that I now believe it must be done. This is a qualitative difference. It is literally impossible for a lone scholar to do what Many Labs and the Metaketa initiatives have done.
One reason that the lab model has been so successful in Psychology is that that discipline’s minimal product is in a sense “smaller” and faster to produce: there are still different time horizons for the actors involved depending on their career stages, but the temporal cycle of the conception and execution of a given study/publication is sufficiently short that it does not intersect these personal time cycles. (This at least seems to be true in social psych, the only subfield I know at all. The grass isn’t necessarily greener, though, as this shortened time cycle means that many people publish dozens of peer-reviewed papers before getting their first tenure-track job.)
Other disciplines—particularly ones using field RCTs—have a much longer time cycle from conception to completion of a project. Many existing institutions recognize this, and plan to give grad students and post-docs a multi-year time horizon to execute studies of the scope we now think to be necessary. Even still, this precludes anyone from conducting a study which varies time.
There is thus a “sociology of science” reason for why temporal validity is distinct from other forms of external validity. Many Labs and the Metaketa have overcome serious transactions costs issues in replicating experimental interventions in different geographic locations simultaneously. We have none of the infrastructure for the converse: the various time cycles of humans in academia do not encourage conducting the same experiment in the same location across time.
This topic is the subject of Graeme Blair and Gwyneth McClendon’s new chapter in the new Handbook of Experimental Political Science (also, this entire volume looks like it’s going to be 🔥 🔥 🔥 ). The chapter proposes a framework for thinking through three “Multi-Context Approaches” for conducting experiments. The entire chapter is fantastic, and I especially enjoyed Table 1.
The concerns listed here make clear that meta-science is political methodology. Some of the most important constraints on the practice of social science are political.
The chapter does not, however, spend much time on the construction of formal institutions. The “coordinated, sequential” approach describes the practice of a research team taking an earlier experimental intervention and applying it in a new context; other models include the possibility of bundling treatment/measurement modules and adding levels of randomization to squeeze as much knowledge production from each high-cost RCT as possible.
None of these is the kind of “Metaketa, but across time rather than across space” that I discuss above. That’s not surprising. Time is not space. Economists are more worried about the problem of “time inconsistency” than “space inconsistency” in modelling of incentives/preferences.
Still, it’s promising to see developments at the level of reducing the costs associated with a semi-coordinated informal-institutional research program. Given the challenges associated with the “Academic Firm,” this is a pragmatic step in the right direction.
My ideal academic firm thus remains firmly in the realm of fantasy, but it’s encouraging to see these types of questions analyzed with the rigor that we had previously reserved for clustering standard errors.