I felt that "back and forth for 6 months over anonymous twitch accounts" paragraph in my chest. According to my IRB its basically ethically impossible to pay homeless people for their time.
I have made some progress with my IRB by making a habit of organising video calls with them about projects before submitting applications. They remind themselves regularly that they "want to approve research" and think that "you can take risks as long as they are justified". I find that a videocall with them to discuss these aspects of a (complex) piece of research tends to reduce the back and forth a bit. It also makes their questions less cryptic and feel less like a trap. When they ask me things like "how do will you collect consent" I usually want to reply "how would you prefer I collect consent?" but then they always deflect by invoking academic freedom. If I had academic freedom I wouldn't need your approval!
Yes I think that's exactly right. I've even been able to schedule in-person meetings sometimes and that helps immensely. It's not about ethics oversight -- it's about the enormity of the bureaucratic apparatus that makes it feel so inhuman
I read the Scott Alexander piece you linked. What's so striking in it for me is the inflation of the bureaucracy and the point about the (opportunity) costs of all this. It seems like a clear sign of rent seeking. There are people whose professional life is enhanced by amassing power to the ERB, and one way to do this is to proliferate the number of rules and their stringency. The costs of ERB don't matter to them because these are equivalent to their personal profit. The sticky thing about most bureaucracies is that they're not spending their own money. But in the case of ERBs, at least in England, they are spending the university's money in the form of staff time. The ERB ethos needs to be one of *facilitating* research not blocking it. ERBs are ironically understaffed and under-organised for this facilitation role, which would involve *them* writing your ethics approval after a discussion of it rather than you. If they were given skin in the game in this way the length and tediousness of these applications would shrink. Instead, even a good-faith ERB has just a few overwhelmed staff, and so they inevitably resort to brain dead checklists ("Will your social science study involve taking weapons grade materials across borders?") and generic training. Legal approaches proliferate when relational ones are required.
yeah that sounds like a great idea. Need to get the incentives aligned with the approval people -- if anything, they should get *credit* for approving particulalry innovative research designs.
Abundance had a large section about this, particularly relating to the NIH.
We should perhaps be advocating for a new intranational, low-hurdle, low-bureaucracy, research apparatus that's closer in form to an art-science publication house. An ESRB ratings option.
To close the loop: it's deeply ironic that the vast majority of RCTs are run without IRB approval in the AB Testing of tech companies.
Also, let me pour water on the claim that RCTs in social science have any reasonable notion of internal validity. ITEs are metaphysical. No one really knows what we're inferring from a point estimate of the ATE. Confidence intervals are almost always plucked from an unverifiable asymptotic normality assumption. And as soon as someone breaks out a correction based on a linear statistical model, they're just telling fantasy tales.
An arrow aimed right at the heart of Galtonian science. I love it. And that was true even before you closed with Dewey.
I'm trying to sketch connections to other efforts to re-democratize science and more radical ideas about citizen science. Reinventing Science. Slime Mold Time Mold. Adam Mastroianni's Experimental History. What else would you point to?
Adam Mastroianni got funding for a communal science house a couple summers ago. I don't know what ended up happening with that exactly, but in his original announcement he lists many examples of what he considers radical citizen science: https://www.adammastroianni.com/science-house
At least one of the science house residents from that summer has a Substack, and published a couple of things recently: https://counterfeitbees.substack.com/
I just wish qualitative research had more representation in these movements. Or at least. more discussion of any role it might potentially have. There's a "qualitative open science" movement gaining steam but I'm not sure that's the same thing.
Thanks for pointing us to the actual law - it's very interesting. I think you might be missing a much more compelling exemption. Instead of claiming the experiment isn't generalizable, many types of studies could convincingly claim to be public health surveillance, which is exempt:
45 CFR § 46.102(i)
For purposes of this part, the following activities are deemed not to be research:
...
(2) Public health surveillance activities, including the collection and testing of information or biospecimens, conducted, supported, requested, ordered, required, or authorized by a public health authority. Such activities are limited to those necessary to allow a public health authority to identify, monitor, assess, or investigate potential public health signals, onsets of disease outbreaks, or conditions of public health importance (including trends, signals, risk factors, patterns in diseases, or increases in injuries from using consumer products). Such activities include those associated with providing timely situational awareness and priority setting during the course of an event or crisis that threatens public health (including natural or man-made disasters).
One quibble: The IRB definition says "designed to develop or contribute to generalizable knowledge". Yet, in the next sentence the following is written: "the results of RCTs are not generalizable". The definition just says it has to contribute to generalizable knowledge, not that the studies have to be generalizable (Popper would agree with this point, I think). Given that the premise of the post is to nitpick wordings, I think the quibble is fair?
Furthermore. It is my impression, that in most European countries (well, at least the one I am part of in Denmark), IRBs purpose is to consider the risks & benefits to those that participate in studies (all kinds of studies, not just experimental studies). I don't know the US IRB regulations, but they are certainly very different to the legal & ethical basis of the one I am involved in.
I think that the quibble would allow for a definition that either /nothing/ counts as research, or that everything (other than, basically, fraud) does. So I'm happy to say either that 1) both reading a book and conducting an RCT "contribute" to generalizable knowledge or 2) "generalizable knowledge" doesn't exist.
At present, RCTs are considered unique on both epistemological and ethical grounds, and that is the position I find indefensible.
Well, the quibble is Popperian. Reading a book does not involve a conjecture and potential refutation. However, lots of research (designs) can contribute to generalizable knowledge, so arguing that RCTs shouldn't be unique in that respect, is fair enough. But, as you indicate, the (IRB) argument is linked to the fact that there is a control group, and non-RCT studies usually don't have that. So, again, in that perspective, RCTs are actually unique.
I am persuaded by the blog post in some sense: I find it truly odd that the IRB justifies its existence in this (RCT) way. Not all ethically "difficult" studies are experimental!
You likely are already familiar with causal data fusion, but perhaps this could benefit others: https://www.pnas.org/doi/10.1073/pnas.1510507113 TL;DR: external validity can, to a point, be boiled down to algorithms
If in order to qualify as a research method a method needs to have external validity, then nothing qualifies as a research method. I find the first half of this post puzzlingly overstated.
My impression was that this is largely rhetorical: an ironic commentary on the incoherence of IRB's own assumptions about RCTs as the "gold standard" predicated on their external validity. Not a claim that research lacking proof of external validity can't be valuable and still count as "research." To the contrary, given how impossible it is to guarantee external validity with current scientific tools - why tie our own hands with ossified IRB rituals and outdated assumptions of gold-standard validity, while the bad guys get to do whatever research they want and control the narrative? Hence, the cheerleading for grassroots citizen science whatever form that may take. Perhaps I'm misunderstanding though.
I presented research findings from my work at UCLA (irb approved) at a lectures on tap, where I replicated an experiment live on the audience. Say I wanted to publish that data. Is a pomo magazine my only option?
I agree that the term "generalizable" is ridiculously flimsy lawmaking - and it realistically does create a potential loophole that could help get the unhinged authoritarian IRB CYA yoke off our necks - at least for some types of desperately needed health research. But I think you're missing the idea that the easier loophole might be the word "systematic."
"*Research* means a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge."
Lawmakers seem to have started with the assumption that without the firm hand of the wise IRB masters on our necks research scientists are at special risk of becoming moustache-twirling Nazis. At the same time, lawmakers must have been aware of the fact that workaday physicians and their patients often have to use the scientific method to figure out which treatment options might work best. If the law's definition of "Research" were simply "any scientific experiment involving human subjects" then it would have the unintended effect of requiring physicians to get IRB pre-approval before experimentally testing any medical hypotheses with their patients. The "any scientific experiment" definition would inadvertently cover things like finding out whether off-label penicillin can cure late-latent syphilis. In other words, the extra paperwork required by an airtight definition of human subjects research would *encourage* workaday physicians to repeat the central medical withholding abuses of the Tuskegee Study:
Through this lens, the more closely a given experiment resembles routine medical care the more it falls outside the lawful purview of IRBs. Scott Alexander's study suddenly strikes me as entirely outside the IRB's lawful jurisdiction. Self-experiments are even more obviously outside IRB jurisdiction. It's puzzling that nobody seems to have noticed the loophole:
The term "systematic" is almost as loose and sloppy as "generalizable," but I think I see an easy bright-line test for it. Systematic investigations typically include a control arm - whereas routine medical care never includes a control arm. This puts me exactly upside down from your argument. I'd argue that any exploratory pilot experiments that lack a control arm should be viewed as outside the statutory definition of human subjects research - while investigations that use control groups may be required to seek pre-approval under the existing law. I'm not arguing that the existing law is an optimal solution - just making bets about where scientists seem like they'd be on solid ground resisting IRB over-reach under the law as it's currently written.
Also, "designed" is flimsy as all hell. The design of an experiment is whatever the designer says it is! You can say the experiment was designed to determine whether the moon is un-generalizably made of cream cheese and oops it accidentally revealed that the questionnaire you've been using is a terrible way of determining whether somebody is manic-depressive. The un-generalizability of the stated study design might not pass the smell test with a disciplinary board. But it might be worth a try. Everybody knows IRBs are needlessly obstructing life-saving science. The disciplinary board might be inclined to let you off with a wink and a nod.
Yes I think that's pretty much correct -- university IRBs go way beyond the law, because they want to be immune to any kind of liability (even just in the form of negative publicity). And, academic journals / editors / peer reviewers ultimately decide what academic "science" is, and they've agreed that IRBs are necessary and (pretty much) sufficient arbiters of ethics.
Normal people can just do what they want. Academics can *also* do whatever they want...as long as they don't use university resources or try to publish their results in an academic journal
I felt that "back and forth for 6 months over anonymous twitch accounts" paragraph in my chest. According to my IRB its basically ethically impossible to pay homeless people for their time.
I have made some progress with my IRB by making a habit of organising video calls with them about projects before submitting applications. They remind themselves regularly that they "want to approve research" and think that "you can take risks as long as they are justified". I find that a videocall with them to discuss these aspects of a (complex) piece of research tends to reduce the back and forth a bit. It also makes their questions less cryptic and feel less like a trap. When they ask me things like "how do will you collect consent" I usually want to reply "how would you prefer I collect consent?" but then they always deflect by invoking academic freedom. If I had academic freedom I wouldn't need your approval!
Yes I think that's exactly right. I've even been able to schedule in-person meetings sometimes and that helps immensely. It's not about ethics oversight -- it's about the enormity of the bureaucratic apparatus that makes it feel so inhuman
I read the Scott Alexander piece you linked. What's so striking in it for me is the inflation of the bureaucracy and the point about the (opportunity) costs of all this. It seems like a clear sign of rent seeking. There are people whose professional life is enhanced by amassing power to the ERB, and one way to do this is to proliferate the number of rules and their stringency. The costs of ERB don't matter to them because these are equivalent to their personal profit. The sticky thing about most bureaucracies is that they're not spending their own money. But in the case of ERBs, at least in England, they are spending the university's money in the form of staff time. The ERB ethos needs to be one of *facilitating* research not blocking it. ERBs are ironically understaffed and under-organised for this facilitation role, which would involve *them* writing your ethics approval after a discussion of it rather than you. If they were given skin in the game in this way the length and tediousness of these applications would shrink. Instead, even a good-faith ERB has just a few overwhelmed staff, and so they inevitably resort to brain dead checklists ("Will your social science study involve taking weapons grade materials across borders?") and generic training. Legal approaches proliferate when relational ones are required.
yeah that sounds like a great idea. Need to get the incentives aligned with the approval people -- if anything, they should get *credit* for approving particulalry innovative research designs.
Abundance had a large section about this, particularly relating to the NIH.
We should perhaps be advocating for a new intranational, low-hurdle, low-bureaucracy, research apparatus that's closer in form to an art-science publication house. An ESRB ratings option.
To close the loop: it's deeply ironic that the vast majority of RCTs are run without IRB approval in the AB Testing of tech companies.
Also, let me pour water on the claim that RCTs in social science have any reasonable notion of internal validity. ITEs are metaphysical. No one really knows what we're inferring from a point estimate of the ATE. Confidence intervals are almost always plucked from an unverifiable asymptotic normality assumption. And as soon as someone breaks out a correction based on a linear statistical model, they're just telling fantasy tales.
An arrow aimed right at the heart of Galtonian science. I love it. And that was true even before you closed with Dewey.
I'm trying to sketch connections to other efforts to re-democratize science and more radical ideas about citizen science. Reinventing Science. Slime Mold Time Mold. Adam Mastroianni's Experimental History. What else would you point to?
Ben Anderson built his own wet lab and is doing some interesting things in St. Louis, I think: https://www.consciousrepository.com/about
Adam Mastroianni got funding for a communal science house a couple summers ago. I don't know what ended up happening with that exactly, but in his original announcement he lists many examples of what he considers radical citizen science: https://www.adammastroianni.com/science-house
At least one of the science house residents from that summer has a Substack, and published a couple of things recently: https://counterfeitbees.substack.com/
I just wish qualitative research had more representation in these movements. Or at least. more discussion of any role it might potentially have. There's a "qualitative open science" movement gaining steam but I'm not sure that's the same thing.
Great stuff! Thank you.
Thanks for pointing us to the actual law - it's very interesting. I think you might be missing a much more compelling exemption. Instead of claiming the experiment isn't generalizable, many types of studies could convincingly claim to be public health surveillance, which is exempt:
45 CFR § 46.102(i)
For purposes of this part, the following activities are deemed not to be research:
...
(2) Public health surveillance activities, including the collection and testing of information or biospecimens, conducted, supported, requested, ordered, required, or authorized by a public health authority. Such activities are limited to those necessary to allow a public health authority to identify, monitor, assess, or investigate potential public health signals, onsets of disease outbreaks, or conditions of public health importance (including trends, signals, risk factors, patterns in diseases, or increases in injuries from using consumer products). Such activities include those associated with providing timely situational awareness and priority setting during the course of an event or crisis that threatens public health (including natural or man-made disasters).
I appreciate this post, as performance art!
One quibble: The IRB definition says "designed to develop or contribute to generalizable knowledge". Yet, in the next sentence the following is written: "the results of RCTs are not generalizable". The definition just says it has to contribute to generalizable knowledge, not that the studies have to be generalizable (Popper would agree with this point, I think). Given that the premise of the post is to nitpick wordings, I think the quibble is fair?
Furthermore. It is my impression, that in most European countries (well, at least the one I am part of in Denmark), IRBs purpose is to consider the risks & benefits to those that participate in studies (all kinds of studies, not just experimental studies). I don't know the US IRB regulations, but they are certainly very different to the legal & ethical basis of the one I am involved in.
I think that the quibble would allow for a definition that either /nothing/ counts as research, or that everything (other than, basically, fraud) does. So I'm happy to say either that 1) both reading a book and conducting an RCT "contribute" to generalizable knowledge or 2) "generalizable knowledge" doesn't exist.
At present, RCTs are considered unique on both epistemological and ethical grounds, and that is the position I find indefensible.
Well, the quibble is Popperian. Reading a book does not involve a conjecture and potential refutation. However, lots of research (designs) can contribute to generalizable knowledge, so arguing that RCTs shouldn't be unique in that respect, is fair enough. But, as you indicate, the (IRB) argument is linked to the fact that there is a control group, and non-RCT studies usually don't have that. So, again, in that perspective, RCTs are actually unique.
I am persuaded by the blog post in some sense: I find it truly odd that the IRB justifies its existence in this (RCT) way. Not all ethically "difficult" studies are experimental!
You likely are already familiar with causal data fusion, but perhaps this could benefit others: https://www.pnas.org/doi/10.1073/pnas.1510507113 TL;DR: external validity can, to a point, be boiled down to algorithms
indeed — if we had the right data!
as I summarize their approach in a presentation — “We solved this, the rest is a trivial empirical problem”
but for essentially all social science problems, the empirical problem is extremely far from trivial
Yeah it feels like trying to model the rostrum of a tick/a stellar collision/etc with F=ma: just draw the free body problem. *Just*
If in order to qualify as a research method a method needs to have external validity, then nothing qualifies as a research method. I find the first half of this post puzzlingly overstated.
My impression was that this is largely rhetorical: an ironic commentary on the incoherence of IRB's own assumptions about RCTs as the "gold standard" predicated on their external validity. Not a claim that research lacking proof of external validity can't be valuable and still count as "research." To the contrary, given how impossible it is to guarantee external validity with current scientific tools - why tie our own hands with ossified IRB rituals and outdated assumptions of gold-standard validity, while the bad guys get to do whatever research they want and control the narrative? Hence, the cheerleading for grassroots citizen science whatever form that may take. Perhaps I'm misunderstanding though.
I presented research findings from my work at UCLA (irb approved) at a lectures on tap, where I replicated an experiment live on the audience. Say I wanted to publish that data. Is a pomo magazine my only option?
I agree that the term "generalizable" is ridiculously flimsy lawmaking - and it realistically does create a potential loophole that could help get the unhinged authoritarian IRB CYA yoke off our necks - at least for some types of desperately needed health research. But I think you're missing the idea that the easier loophole might be the word "systematic."
"*Research* means a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge."
Lawmakers seem to have started with the assumption that without the firm hand of the wise IRB masters on our necks research scientists are at special risk of becoming moustache-twirling Nazis. At the same time, lawmakers must have been aware of the fact that workaday physicians and their patients often have to use the scientific method to figure out which treatment options might work best. If the law's definition of "Research" were simply "any scientific experiment involving human subjects" then it would have the unintended effect of requiring physicians to get IRB pre-approval before experimentally testing any medical hypotheses with their patients. The "any scientific experiment" definition would inadvertently cover things like finding out whether off-label penicillin can cure late-latent syphilis. In other words, the extra paperwork required by an airtight definition of human subjects research would *encourage* workaday physicians to repeat the central medical withholding abuses of the Tuskegee Study:
https://pubmed.ncbi.nlm.nih.gov/10724044/
Talk about blowback.
Through this lens, the more closely a given experiment resembles routine medical care the more it falls outside the lawful purview of IRBs. Scott Alexander's study suddenly strikes me as entirely outside the IRB's lawful jurisdiction. Self-experiments are even more obviously outside IRB jurisdiction. It's puzzling that nobody seems to have noticed the loophole:
https://pubmed.ncbi.nlm.nih.gov/29926769/
https://www.mdpi.com/2076-393X/12/9/958
The term "systematic" is almost as loose and sloppy as "generalizable," but I think I see an easy bright-line test for it. Systematic investigations typically include a control arm - whereas routine medical care never includes a control arm. This puts me exactly upside down from your argument. I'd argue that any exploratory pilot experiments that lack a control arm should be viewed as outside the statutory definition of human subjects research - while investigations that use control groups may be required to seek pre-approval under the existing law. I'm not arguing that the existing law is an optimal solution - just making bets about where scientists seem like they'd be on solid ground resisting IRB over-reach under the law as it's currently written.
Also, "designed" is flimsy as all hell. The design of an experiment is whatever the designer says it is! You can say the experiment was designed to determine whether the moon is un-generalizably made of cream cheese and oops it accidentally revealed that the questionnaire you've been using is a terrible way of determining whether somebody is manic-depressive. The un-generalizability of the stated study design might not pass the smell test with a disciplinary board. But it might be worth a try. Everybody knows IRBs are needlessly obstructing life-saving science. The disciplinary board might be inclined to let you off with a wink and a nod.
Yes I think that's pretty much correct -- university IRBs go way beyond the law, because they want to be immune to any kind of liability (even just in the form of negative publicity). And, academic journals / editors / peer reviewers ultimately decide what academic "science" is, and they've agreed that IRBs are necessary and (pretty much) sufficient arbiters of ethics.
Normal people can just do what they want. Academics can *also* do whatever they want...as long as they don't use university resources or try to publish their results in an academic journal