To the best of my knowledge, internal CEAs rarely if ever turn up negative.
Here’s one example of an EA org analyzing the effectiveness of their work, and concluding the impact sucked:
CFAR in 2012 focused on teaching EAs to be fluent in Bayesian reasoning, and more generally to follow the advice from the Sequences. CFAR observed that this had little impact, and after much trial and error abandoned large parts of that curriculum.
This wasn’t a quantitative cost-effectiveness analysis. It was more a subjective impression of “we’re not getting good enough results to save the world, we can do better”. CFAR did do an RCT which showed disappointing results, but I doubt this was CFAR’s main reason for change.
These lessons percolated out to LessWrong blogging, which now focuses less on Bayes theorem and the Sequences, but without calling a lot of attention to the less.
I expect that most EAs who learned about CFAR after about 2014 underestimate the extent to which CFAR’s initial strategies were wrong, and therefore underestimate the evidence that initial approaches to EA work are mistaken.
It might be orthogonal to the point you’re making, but do we have much reason to think that the problem with old-CFAR was the content? Or that new-CFAR is effective?
Thanks a lot! Is there a writeup of this somewhere? I tend to be a pretty large fan of explicit rationality (at least compared to EAs or rationalists I know), so evidence that reasoning in this general direction is empirically kind of useless would be really useful to me!
The original approach was rather erratic about finding high value choices, and was weak at identifying the root causes of the biggest mistakes.
So participants would become more rational about flossing regularly, but rarely noticed that they weren’t accomplishing much when they argued at length with people who were wrong on the internet. The latter often required asking embarrassing questions their motives, and sometimes realizing that they were less virtuous than assumed. People will, by default, tend to keep their attention away from questions like that.
The original approach reflected trends in academia to prioritize attention on behaviors that were most provably irrational, rather than on what caused the most harm. Part of the reason that CFAR hasn’t documented their successes well is they’ve prioritized hard-to-measure changes.
Here’s one example of an EA org analyzing the effectiveness of their work, and concluding the impact sucked:
CFAR in 2012 focused on teaching EAs to be fluent in Bayesian reasoning, and more generally to follow the advice from the Sequences. CFAR observed that this had little impact, and after much trial and error abandoned large parts of that curriculum.
This wasn’t a quantitative cost-effectiveness analysis. It was more a subjective impression of “we’re not getting good enough results to save the world, we can do better”. CFAR did do an RCT which showed disappointing results, but I doubt this was CFAR’s main reason for change.
These lessons percolated out to LessWrong blogging, which now focuses less on Bayes theorem and the Sequences, but without calling a lot of attention to the less.
I expect that most EAs who learned about CFAR after about 2014 underestimate the extent to which CFAR’s initial strategies were wrong, and therefore underestimate the evidence that initial approaches to EA work are mistaken.
Another two examples off the top of my head:
Students for High-Impact Charity suspending outreach based on disappointing longer-term engagement.
Evidence Action shutting down No Lean Season due to disappointing results from a study.
It might be orthogonal to the point you’re making, but do we have much reason to think that the problem with old-CFAR was the content? Or that new-CFAR is effective?
Thanks a lot! Is there a writeup of this somewhere? I tend to be a pretty large fan of explicit rationality (at least compared to EAs or rationalists I know), so evidence that reasoning in this general direction is empirically kind of useless would be really useful to me!
The original approach was rather erratic about finding high value choices, and was weak at identifying the root causes of the biggest mistakes.
So participants would become more rational about flossing regularly, but rarely noticed that they weren’t accomplishing much when they argued at length with people who were wrong on the internet. The latter often required asking embarrassing questions their motives, and sometimes realizing that they were less virtuous than assumed. People will, by default, tend to keep their attention away from questions like that.
The original approach reflected trends in academia to prioritize attention on behaviors that were most provably irrational, rather than on what caused the most harm. Part of the reason that CFAR hasn’t documented their successes well is they’ve prioritized hard-to-measure changes.