Would you be able to say a little more about why part of your criteria seems to be degree of probability shift (“We will award larger prizes for larger changes to these probabilities, as follows...”). It seems to me that you might get a case where you could get analyses that offer larger changes but are less robust than some analyses that suggest smaller changes. I didn’t understand how much of your formal evaluation will look at plausibility, argumentation, soundness?
Plausibility, argumentation, and soundness will be inputs into how much our subjective probabilities change. We framed this in terms of subjective probabilities because it seemed like the easiest way to crisply point at ideas which could change our prioritization in significant ways.
Would you be able to say a little more about why part of your criteria seems to be degree of probability shift (“We will award larger prizes for larger changes to these probabilities, as follows...”). It seems to me that you might get a case where you could get analyses that offer larger changes but are less robust than some analyses that suggest smaller changes. I didn’t understand how much of your formal evaluation will look at plausibility, argumentation, soundness?
(asking as a curiosity not as a critique)
Plausibility, argumentation, and soundness will be inputs into how much our subjective probabilities change. We framed this in terms of subjective probabilities because it seemed like the easiest way to crisply point at ideas which could change our prioritization in significant ways.