re: the latter, maybe you can get inspiration from RP’s CCM > existential risk > “small-scale AI misalignment project” and check out the graphics below. Their default params are 96.4% chance no effect, 70% chance +ve outcome conditional on effect, +30% rise in p(extinction) conditional on -ve outcome, and you can change them and see how the EV updates; these defaults don’t matter as much as the takeaway that AIS work needs to be robustly +ve and that folks whose risk aversion is greater than zero (probably wise) will do well to prioritise resolving this sign uncertainty, which boils down to Michael’s advice above (cf. the advice to build deep models, or Dave Banerjee’s advice more specifically).
re: the latter, maybe you can get inspiration from RP’s CCM > existential risk > “small-scale AI misalignment project” and check out the graphics below. Their default params are 96.4% chance no effect, 70% chance +ve outcome conditional on effect, +30% rise in p(extinction) conditional on -ve outcome, and you can change them and see how the EV updates; these defaults don’t matter as much as the takeaway that AIS work needs to be robustly +ve and that folks whose risk aversion is greater than zero (probably wise) will do well to prioritise resolving this sign uncertainty, which boils down to Michael’s advice above (cf. the advice to build deep models, or Dave Banerjee’s advice more specifically).