Executive summary: In high-uncertainty fields like existential risk reduction and longtermism, it is difficult to distinguish truly high-impact interventions from those that merely appear promising due to biases and measurement noise, raising concerns about how to reliably assess effectiveness in these areas.
Key points:
Many of the most promising-seeming interventions exist in domains with inherently high uncertainty, making it hard to determine if their estimated impact is real or an artifact of bias and randomness.
Evaluators cannot directly observe an intervention’s true effectiveness but instead see a combination of its actual impact and various sources of measurement error.
A toy model suggests that interventions in high-variance domains (e.g., existential risk) have such large measurement errors that the highest-scoring interventions might not be the most effective.
Bayesian reasoning suggests adjusting impact estimates toward prior beliefs, but this does not fully resolve the problem, as priors themselves may be shaped by biases.
A major challenge is that filtering for high-apparent-effectiveness interventions selects for those most influenced by errors and biases, making it unclear how to reliably identify the best opportunities.
The author seeks practical heuristics for navigating these uncertainties, beyond just mathematical Bayesian updating, to avoid overvaluing interventions that align with preexisting assumptions.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
Executive summary: In high-uncertainty fields like existential risk reduction and longtermism, it is difficult to distinguish truly high-impact interventions from those that merely appear promising due to biases and measurement noise, raising concerns about how to reliably assess effectiveness in these areas.
Key points:
Many of the most promising-seeming interventions exist in domains with inherently high uncertainty, making it hard to determine if their estimated impact is real or an artifact of bias and randomness.
Evaluators cannot directly observe an intervention’s true effectiveness but instead see a combination of its actual impact and various sources of measurement error.
A toy model suggests that interventions in high-variance domains (e.g., existential risk) have such large measurement errors that the highest-scoring interventions might not be the most effective.
Bayesian reasoning suggests adjusting impact estimates toward prior beliefs, but this does not fully resolve the problem, as priors themselves may be shaped by biases.
A major challenge is that filtering for high-apparent-effectiveness interventions selects for those most influenced by errors and biases, making it unclear how to reliably identify the best opportunities.
The author seeks practical heuristics for navigating these uncertainties, beyond just mathematical Bayesian updating, to avoid overvaluing interventions that align with preexisting assumptions.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.