Note: this is the second of three posts discussing the 2020 Meta Election research partnership and the resulting papers, in advance of a keynote panel at IC2S2 on the topic this Thursday July 18; here is the first.
The “numbers” and “ranges of numbers” that define the outcome of the experiment I discussed last time ultimately refer to some natural language statement, mediated by some statistical hypothesis. Ben Recht’s recent Meehl-blogging provides the following formalization of the problem:
The map from substantive theory T to testable statistical hypothesis H goes through a derivation chain involving auxiliary theories, instruments, ceteris paribus assertions, and experimental conditions. The map from hypothesis to observation is through the statistical model manufactured by the derivation chain.
Statistical theory provides a variety of means to infer the veracity of H from O. Usually this goes through Bayes’ Rule….This seems all well and good. But let’s say we now infer that H has high probability given all of our evidence. Then what? We cared about T! How do we compute the probability of T from O?
Meehl argued in these lectures from the early 1990s that there did not exist a formal procedure that could translate rigorously between O and T. Recht (who, if you’re unaware, is the author of multiple recent foundational textbooks on machine learning) does not believe that we have solved this problem in the intervening decades — nor, fundamentally, that it is possible to do so outside of controlled environments.
If you disagree, if you think we should resurrect logical positivism now that we’ve got bigger computers, best of luck to you. In actually-existing social science, we’re not asking logicians before going from H to T. In fact, after doing all that hard statistical work to get from O to H, we’re pretty hand-wavy about this final step back to natural language.
This step, I have argued, should be thought of as poetic validity. There’s not much discussion of poetry these days, but this is a mistake; we’ve conflated the outdated forms of poetry with its linguistic function. No one speaks in sonnets, and contemporary poetry has come to mean “words printed in a book with nonstandard punctuation.” Words that I know that I’m supposed to like but don’t really understand.
Actual poetry now takes place almost entirely online.1 Twitter was a hotbed for poetry, most famously the linguistically fecund “Weird Twitter” of 2012-2016. My concept of poetry comes from Flusser, who says that poetry is the first step in transforming the chaos of the unknown into something the intellect can grasp. We think with language, we communicate with language — but when we use language, we also change the meaning of the words we’re using. Old words are used in new ways, to say things that couldn’t be said (and thus couldn’t be thought) before. Sometimes, new words arise. Where did they come from? There are tremendously many different strings of characters or vocal utterances don’t mean anything; how are some of them drawn into the realm of language? The answer is poetry: the first step towards meaning.
My go-to example of a case where poetic validity is a problem is the idea of “echo chambers” and social media. I won’t rehash the argument in this post but I will add that I think that the sentence “social media is an echo chamber” is true. The resonance of this concept reflects the way it fits within our language system. The question of whether the sentence is poetically valid has to do with how we choose to map this true sentence from the world of T to the lower-dimensional world of H and then to O.
So — how do the Meta 2020 papers use the phrase “echo chamber”? Surely within this project, in the papers written by the exact same team of 32 authors, the concept is the same?
One of the experiments decreased the amount of content to like-minded sources. The abstract motivates this intervention as follows:
To evaluate a potential response to concerns about the effects of echo chambers, we conducted a multi-wave field experiment on Facebook among 23,377 users for whom we reduced exposure to content from like-minded sources during the 2020 US presidential election by about one-third.
Another paper, a descriptive analysis of the degree of ideological segregation on the platform, motivates their analysis by saying that:
Early research (4, 5) argued that personalization algorithms (filter bubbles) and social curation processes (echo chambers) increase the probability that people will surround themselves with ideologically compatible information
Taken together, the metatheoretical question seems to be whether echo chambers cause echo chambers. (Whether social curation processes cause biased media diets).
There isn’t even internal poetic validity within this project and co-author team! This is surely a bad sign for external poetic validity, the ability of other researchers to make this translation from O to H and then to T, to the world of Theory in which natural human language is our tool for thinking.
I’m going to argue that neither of these operationalizations of “echo chamber” is poetically valid. One question is whether we should abandon the concept of “echo chambers” altogether; there’s a risk of proliferating definitions deepening confusion rather than providing clarity. But I think the problem is that both papers (as well as hundreds of others) jump too quickly to asking whether social media is an echo chamber without stopping to ask what an echo chamber is. I repeat: we lack an ontology.
But social media is an echo chamber. This sentence is poetically true. Truth here is distinct from validity; I’m using the coherence theory of truth to refer to how sentences hang together in a system of thought. This is not a common conception of truth, among scientists, for whom the correspondence theory is more intuitive. But even if you want to hold the latter position, it’s incredibly important to know what an echo chamber is!
The point of this philosophical aside is to argue that the sentence “social media is an echo chamber” is actually more informative than phrase “echo chamber.” People didn’t really talk about “echo chambers” in anything like this sense until the spread of the internet in the late 90s and early 2000s. It was a technical musical tool developed in the 1930s, for tuning instruments and creating reverb. “Echo chamber” wasn’t a common concept lying around, we weren’t debating whether other media environments were an “echo chamber.”
“Social media is an echo chamber” is true because people hear it and think, yes, without being able to provide a concrete definition of what an “echo chamber” is. Whatever an echo chamber is, social media is it.
So what is it? Let’s ask the Greeks2, who knew a thing or two about poetry.
Echo and Narcissus, by John William Waterhouse (1903)
Buried deep within our linguistic reservoirs of meaning, Echo is related to Narcissus. Social media is fundamentally narcissistic. Recall the myth: jealous Hera curses the “talkative nymph” Echo to only be able to repeat back what is said, not to say anything original. (That’s what an echo is). Her voicebox may be limited but her eyes still work fine, and she encounters and becomes infatuated with the beautiful youth Narcissus.
After some classic slapstick miscommunication hijinks, Echo believes that Narcissus is reciprocating her desire.
In her delight, Echo rushed to Narcissus ready to throw her arms around her beloved. Narcissus, however, was appalled and, spurning her, exclaimed, ‘Hands off! May I die before you enjoy my body.’
So this is one sense in which the phrase “social media is an echo chamber” is, I think, poetically true. It’s a metaphor. It’s not a literal map from T to H to O; according to the correspondence theory of truth, the sentence isn’t really true or false — there is no such thing as an “echo chamber” in the physical world to check whether social media is it.
Science can’t function if we take the outputs of poetry too literally. The truth of “social media is an echo chamber” is ultimately subjective — the sentence rings true, it resonates with the experience of people who use social media. This is important data. But as social scientists, we have access to other kinds of data. The two Meta 2020 studies discussed above are not analyzing the data of Facebook users’ subjective experience—they are analyzing objective data in the language of the platform.
If we’re committed to objectivity — and I think we should be! there are other valuable points of view but the expertise developed through our methods is unique — then we need to be committed to objectivity at more stages of the scientific process than just the individual study. We need a more rigorous ontology; we need a more rigorous procedure for setting the academic agenda. We’ve seen this happen at the micro-level with the science reform movement — the Meta 2020 papers meticulously adhered to current methodological standards like pre-registration and multiple hypothesis correction. The next meta-scientific move has to be to the macro level.
Alternatively, rather than extending the purview of objective scientific rigor, we could try to extend the poetics. This is done beautifully by legendary political scientist V.O. Key in his final book (Key, 1966):
“The victorious candidate may regard his success as a vindication of his beliefs about why voters vote the way they do …… This narcissism assumes its most repulsive form among election winners who have championed intolerance, who have stirred the passions and hatreds of people ……
For a glaringly obvious reason …… the voice of the people is but an echo. The output of an echo chamber bears an inevitable and invariable relationship to the input …… the people’s verdict can be no more than a selective reflection from among the alternatives and outlooks presented to them” (p2, emphasis mine).
This obviously includes music, particularly rap music.
Apparently the myth was invented pretty much whole cloth by Ovid, and so should properly be termed a Roman myth.