An org or team of people dedicated to Red Teaming EA research. Can include checks for both factual errors and conceptual ones. Like JEPSEN but for research from/within EA orgs. Maybe start with one trusted person and then expand outwards.
After demonstrating impact/accuracy for say 6 months, can become a “security” consultancy for either a) EA orgs interested in testing the validity of their own research or b) an external impact consultancy for the EA community/EA donors interested in testing or even doing the impact assessments of specific EA orgs. For a), I imagine Rethink Priorities may want to become a customer (speaking for myself, not the org).
Potentially good starting places:
- Carefully comb every chapter of The Precipice
- Go through ML/AI Safety papers and (after filtering on something like prestige or citation count) pick some papers at random to Red Team
- All of Tetlock’s research on forecasting, particularly the ones with factoids most frequently cited in EA circles.
- Maybe the Sequences? Especially stuff that quotes social psychology studies and we can check if they replicate
- Any report by Open Phil, especially more recent ones.
- Any report by RP, especially ones that seem fairly consequential (eg work on insect sentience)
- Much of GFI research, especially the empirical studies with open data or the conceptual stuff.
- 80k podcasts, particularly the popular ones (Rob et al don’t challenge the speakers nearly as much as we might like )
- All the public biosecurity papers that EAs like to quote
- Any EAF post that looks like a research post and has >100 karma
- early Carl Shulman or Paul Christiano posts (might want to quickly check beforehand that it’s still endorsed).
- reading carefully and challenging implicit assumptions in the impact assessments of core/semicore orgs like 80,000 Hours, CEA, and Rethink Priorities
Note that unlike my other Red Teaming suggestion, this one is more focused about improving research quality than it is about training junior researchers.
Why do you think this suggestion could be a good use of funds money?
I think it’s good for community and research health to not be built on a bedrock of lies (or less sensationally, inaccuracies and half-truths).
The more checks we have on existing research (ideally in a way that doesn’t impede short-term research progress, and post-publication Red Teaming ought to mostly do this), the more a) we can trust past work that goes through these passes and b) we can reduce research debt and allow us to build on past work.
Dan Luu has a recent post about people’s resistance to systematic measurement, documenting the successes of the original JEPSEN project (and the hidden failures of database designers pre-JEPSEN) as a core argument. Notably many people working on those databases did not believe JEPSEN was necessary (“our system is perfect, no need to measure!” Our system is perfect, don’t trust your lying eyes!” “Our system is perfect, the bug you uncovered is an intended feature!” “Our system is perfect except for one bug that we fixed, back to business as usual!“)!
I think there’s no strong reason to believe that EA research and impact assessments are systematically better or more well-understood by distributed systems used by thousands of programmers and millions (billions?) of end-users, which seems like some points in favor of more rigorous checking by external-ish parties for our core conceptual and empirical beliefs.
New Project/Org Idea: JEPSEN for EA research or EA org Impact Assessments
Note: This is an updated version of something I wrote for “Submit grant suggestions to EA Funds”
What is your grant suggestion?
An org or team of people dedicated to Red Teaming EA research. Can include checks for both factual errors and conceptual ones. Like JEPSEN but for research from/within EA orgs. Maybe start with one trusted person and then expand outwards.
After demonstrating impact/accuracy for say 6 months, can become a “security” consultancy for either a) EA orgs interested in testing the validity of their own research or b) an external impact consultancy for the EA community/EA donors interested in testing or even doing the impact assessments of specific EA orgs. For a), I imagine Rethink Priorities may want to become a customer (speaking for myself, not the org).
Potentially good starting places:
- Carefully comb every chapter of The Precipice
- Go through ML/AI Safety papers and (after filtering on something like prestige or citation count) pick some papers at random to Red Team
- All of Tetlock’s research on forecasting, particularly the ones with factoids most frequently cited in EA circles.
- Maybe the Sequences? Especially stuff that quotes social psychology studies and we can check if they replicate
- Any report by Open Phil, especially more recent ones.
- Any report by RP, especially ones that seem fairly consequential (eg work on insect sentience)
- Much of GFI research, especially the empirical studies with open data or the conceptual stuff.
- 80k podcasts, particularly the popular ones (Rob et al don’t challenge the speakers nearly as much as we might like )
- All the public biosecurity papers that EAs like to quote
- Any EAF post that looks like a research post and has >100 karma
- early Carl Shulman or Paul Christiano posts (might want to quickly check beforehand that it’s still endorsed).
- reading carefully and challenging implicit assumptions in the impact assessments of core/semicore orgs like 80,000 Hours, CEA, and Rethink Priorities
Note that unlike my other Red Teaming suggestion, this one is more focused about improving research quality than it is about training junior researchers.
Why do you think this suggestion could be a good use of funds money?
I think it’s good for community and research health to not be built on a bedrock of lies (or less sensationally, inaccuracies and half-truths).
The more checks we have on existing research (ideally in a way that doesn’t impede short-term research progress, and post-publication Red Teaming ought to mostly do this), the more a) we can trust past work that goes through these passes and b) we can reduce research debt and allow us to build on past work.
Dan Luu has a recent post about people’s resistance to systematic measurement, documenting the successes of the original JEPSEN project (and the hidden failures of database designers pre-JEPSEN) as a core argument. Notably many people working on those databases did not believe JEPSEN was necessary (“our system is perfect, no need to measure!” Our system is perfect, don’t trust your lying eyes!” “Our system is perfect, the bug you uncovered is an intended feature!” “Our system is perfect except for one bug that we fixed, back to business as usual!“)!
I think there’s no strong reason to believe that EA research and impact assessments are systematically better or more well-understood by distributed systems used by thousands of programmers and millions (billions?) of end-users, which seems like some points in favor of more rigorous checking by external-ish parties for our core conceptual and empirical beliefs.