Never Trust an Academic Journal Over 30
Institutional sclerosis, "Seeing like a Dean", Metascience 2021
Apologies for the delay; I was working on this!
If you’re reading this, you may already be familiar with the arguments in our manifesto, but I want to highlight one in particular.
Journal of Quantitative Description: Digital Media @journalqdWhy do we need a Journal of Quantitative Description: Digital Media? Read the "manifesto" from founding co-editors Kevin Munger @kmmunger, Andy Guess @andyguess and Eszter Hargittai @eszter https://t.co/Ncbcyp3v0C
Political methodology includes meta-science. We have to acknowledge that meta-scientific work does not always take the form of an academic paper.
The meta-scientist wants to improve the aggregate output of social science knowledge, not to perfect the individual academic paper. The most important contributions along these lines take the form of institutions.
This is not always legible to the people evaluating our academic output. I recently received comments from my university as part of a standard internal review process. I thought this was generally well-conducted (notwithstanding the generally painful process of compressing myself into a template for the purpose of “Seeing Like a Dean,” with apologies to James C. Scott), but one clause stuck out to me: I was encouraged to take pains that my service commitments — especially my journal editorship — did not interfere with my scholarly research agenda.
Institution building—co-founding and editing the JQD:DM—is part of, not distinct from, my scholarly research agenda.
The mainstream social sciences are currently coasting, the powerful institutions are playing defense against a shifting socio-demographic-technological landscape. Both the professional organizations and the “flagship journals” they control (APSA = APSR, MPSA = AJPS, SPSA = JOP) have a massive incumbent advantage in the prestige-driven economy of academia.
However, consider the famous “network externalities” of social media platforms. The more users they have, the more value they can offer to potential users. This positive feedback loop can make it feel like the first mover is permanently entrenched; you could build a better platform, but you can’t compete with the existing network.
In reality, the same network effects that create a positive feedback loop on the way up operate on the way down: when a platform starts to lose users, it loses part of its value for remaining users. The network ties unravel.
Reputation is tougher because the “network ties” reside in the brains of academics, and you don’t have to a hardcore Bayesian to see that it’d take some pretty compelling evidence to weaken those ties. One of my professors in grad school told us that he was tied for the most lifetime publications in the APSR. He’s generally a great guy, no particular shade here, but he’s unlikely to be first in line to disestablish the existing prestige hierarchy.
But the weaknesses of the legacy journals are starting to show. Declining acceptance rates and the concomitantly increasingly arbitrary nature of peer review—combined with the capacity for micro-critiques to circulate and accumulate on academic Twitter—produces growing resentment.
Remember: in 2013, APSA discussed creating a new e-journal. Arguments in favor included “timeliness (reduced R&R) creates opportunities for assistant professors’ publication needs.” Arguments against included “empirical evidence of need has not been demonstrated”; “an imbalance in APSA journals has been a source of conflict in the association earlier; desire to avoid repeat of those earlier conflicts”; and, most importantly : “proposed format is biased toward quantitative and formal methodologies.”
The topic was tabled in 2013 and then abandoned. The extant oligopoly atop the prestige hierarchy has been so long insulted from any real competition that they are insulated from any real feedback on their policies or practices.
The JQD:DM is already a beneficiary of these mistakes. We are publishing manuscripts that were clearly rejected from Political Science’s “top journals”; one telltale sign is the otherwise-pointless Appendix Table of Contents required by some of them.
Why do we need a Table of Contents for the Appendix at the point of submission? Perhaps the reviewers’ (and editors’) Ctrl + F keys are broken.
The point is, the logic doesn’t matter. Prestigious journals can say “jump” and the legions of the desperate graduate students, post-docs and untenured professors will say “how high.”
Like any entrenched bureaucracy, our time is literally worthless to them. Their purview is so broad that even a tiny amount of increased efficiency on their end is worth any amount of annoyance on our end.
The top journals in Political Science and Economics are knowledge shredders: their knowledge output is dramatically lower than an equivalent amount of energy and intellectual effort put into producing knowledge in a less pathological system. The up-front formatting costs that change with the whims of each new editorial team are merely symptoms of the underlying rot.
Skeptics can reasonably argue that the peer review process does provide value in terms of both feedback on the paper and vetting. Reputation requires costly signaling which requires setting some human capital on fire. Sure, fine, but this cannot be used to justify the change in acceptance rates (sharply downward) or the change in the review process (two rounds of R&R now not uncommon in Poli Sci; Econ review practices, from what I hear, are best understood through the metaphor of BDSM or some other kink. Perhaps the editors have a footnote fetish).
So let a thousand flowers bloom. There’s lots of science to be done, and lots of scientists wanting to do it. We just need to set up the institutions that let them.
I have recently come to understand my interest in meta-science and institutional reform as part of a larger movement, described brilliantly in a recent working paper called “Metascience as a scientific social movement” by David Peterson and Aaron Panofsky. See their Figure 1 below for a taste.
I encourage anyone interested in this topic to read their account, which is well-informed but critical in important ways. In particular, I think metascientists would do well to engage with earlier literatures in the traditions of philosophy of science, sociology of science, and STS. As a trained positivist, I understand the hesitancy to do so — and to be fair, these literatures can be maddeningly obscure at a times — but we’re going to run into dead ends they’ve already mapped out for us if we don’t.
Also, I sincerely regret not having attended Metascience 2019 (the conference Peterson and Panofsky describe), but I’m looking forward to hearing more about Metascience 2021 — and if you’re reading this, you might be interested, too.