I think the right way is to weight each argument by its robustness and the significance of the conclusions i.e.:
If you have a strong argument that an action, A, would be very bad, then you shouldn’t do A.
If you have a speculative argument that A would be very bad, then you probably shouldn’t do A.
If you have a speculative argument that A would be a little bit bad, then that doesn’t mean much.
I’d add a consideration for relative evidence here. If you have an action, A, that you know little about aside from one speculative argument that A would be very bad, but no strong arguments, how does that work?
Here’s a relevant example: say we have two forms of animal rights advocacy. One (A) that has a speculative argument that is very very good in inspiring Democrats, but a strong argument that it’s bad in inspiring Republicans. (These are the two main US political parties.) The other (B) has only a strong argument that’s it’s decent in inspiring Republicans. There is no other evidence. Normally, we’d take down the speculative argument in (A) with a strong, low prior, like we have for developing world health charities (since the base rates for success are pretty low and after in-depth investigation many ones with high speculative arguments turn out to be duds). But in this case, there is no strong prior. To maximize expected value, we should then lean towards (A) in this example, even though this is, in some sense, putting speculative evidence above strong evidence.
Overall, to maximize expected value, I would argue that strength/robustness only matters when comparing different sources of evidence on the same effect of the same action. But when speculative arguments are the only sources of evidence available for a particular effect of a particular action, they should be accounted for no differently than strong arguments of the same effect size.
I think the right way is to weight each argument by its robustness and the significance of the conclusions i.e.: If you have a strong argument that an action, A, would be very bad, then you shouldn’t do A. If you have a speculative argument that A would be very bad, then you probably shouldn’t do A. If you have a speculative argument that A would be a little bit bad, then that doesn’t mean much.
I’d add a consideration for relative evidence here. If you have an action, A, that you know little about aside from one speculative argument that A would be very bad, but no strong arguments, how does that work?
Here’s a relevant example: say we have two forms of animal rights advocacy. One (A) that has a speculative argument that is very very good in inspiring Democrats, but a strong argument that it’s bad in inspiring Republicans. (These are the two main US political parties.) The other (B) has only a strong argument that’s it’s decent in inspiring Republicans. There is no other evidence. Normally, we’d take down the speculative argument in (A) with a strong, low prior, like we have for developing world health charities (since the base rates for success are pretty low and after in-depth investigation many ones with high speculative arguments turn out to be duds). But in this case, there is no strong prior. To maximize expected value, we should then lean towards (A) in this example, even though this is, in some sense, putting speculative evidence above strong evidence.
Overall, to maximize expected value, I would argue that strength/robustness only matters when comparing different sources of evidence on the same effect of the same action. But when speculative arguments are the only sources of evidence available for a particular effect of a particular action, they should be accounted for no differently than strong arguments of the same effect size.