I still think both LEAN and SHIC have a substantial risk of not being cost-effective, but I’m far more confident that there is sufficient analytical work going on now that failure would be detected and learned from. Given the amount of information they’re generating, I’m confident we’ll all learn something important even if either (or both) projects fail
Could you say more about this? When I look at their metrics, it’s a little unclear to me what failure (or success) would look like. In extremis, every group rating LEAN as ineffective (or very effective) would be an update, but it’s unclear to me how we would notice smaller changes in feedback and translate that to counterfactual impact on “hit” group members.
Similarly, for SHIC, if they somehow found a high school student who becomes a top-rated AI safety researcher or something similar that would be a huge update on the benefit of that kind of outreach. But the chances of that seems small, so it’s kind of unclear to me what we should expect to learn if they find that students have some moderate changes in their donations but nothing super-high-impact.
Could you say more about this? When I look at their metrics, it’s a little unclear to me what failure (or success) would look like. In extremis, every group rating LEAN as ineffective (or very effective) would be an update, but it’s unclear to me how we would notice smaller changes in feedback and translate that to counterfactual impact on “hit” group members.
Similarly, for SHIC, if they somehow found a high school student who becomes a top-rated AI safety researcher or something similar that would be a huge update on the benefit of that kind of outreach. But the chances of that seems small, so it’s kind of unclear to me what we should expect to learn if they find that students have some moderate changes in their donations but nothing super-high-impact.