I can clarify the last point which is the most important one:
A reliable recommendation about a highest-leverage tactic would require a methodology that weighs different factors against each other taking into account that different factors have different spread and different weights, which is something a filtering check list is fundamentally unable to do.
Without the ability to quantify considerations, even if this means quantifying qualitative judgments, there is no way to make reliable recommendations because you try to integrate a disparate set of considerations which are—fundamentally—related to each other in a broadly multiplicative manner (see Effectiveness is a Conjunction of Multipliers for the clearest articulation of why that is).
A checklist approach destroys a lot of information and thereby misleads about relative importance, which is what you are trying to evaluate. Because some outcomes you are trying to alleviate are much worse than others, some are much more likely than others, some are much more tractable to attract than others, etc., a lot of this operates on variables that implicitly varying by orders of magnitudes across interventions and a checklist approach will massively under-represent the differences and is thus unlikely to point at the highest impact interventions.
Thanks for the clarification. Now I understand your point. Speaking for myself, because it’s easier, and I’m not consulting with my team...
I disagree that quantification always produces more reliable judgments and think it’s context dependent, but that’s a very fundamental disagreement which perhaps we can discuss in person one day. I’ll just say that in this particular case, we are very transparent about our systematic approach to collecting and aggregating the data we have to draw conclusions (incorporating probability and impact estimates) and will be providing all of the data for others to scrutinize. As I said in our post, we are also happy to present it right away in webinar form, so that others can draw their own conclusions or produce quantitative aggregations of them if they’d like.
And in fact our data—on the harm and containment of threats and the effectiveness and practicality of tactical response—has and will continue to inform the BOTECs that others in the EA space are already doing (in many cases many are currently doing this without the underlying evidence, because it takes time to conceptualize, sift through, and digest). And even without quantifying, it can directly help grantmakers or practitioners make decisions about whether to invest in a specific tactic to counter a specific threat, which is a decision those folks have to make every day. For example, say a statewide organization is anticipating a specific threat, like security forces at the polls and is trying to organize a program in response. Our table and analysis can help them understand what tactics exist to counter those threats and whether any of them are likely more effective than others.
Perhaps a part of the misunderstanding can be boiled down to a title issue. I shouldn’t have used a superlative. I should have just said: “what are high-leverage tactics,” which is actually what I meant.
I can clarify the last point which is the most important one:
A reliable recommendation about a highest-leverage tactic would require a methodology that weighs different factors against each other taking into account that different factors have different spread and different weights, which is something a filtering check list is fundamentally unable to do.
Without the ability to quantify considerations, even if this means quantifying qualitative judgments, there is no way to make reliable recommendations because you try to integrate a disparate set of considerations which are—fundamentally—related to each other in a broadly multiplicative manner (see Effectiveness is a Conjunction of Multipliers for the clearest articulation of why that is).
A checklist approach destroys a lot of information and thereby misleads about relative importance, which is what you are trying to evaluate. Because some outcomes you are trying to alleviate are much worse than others, some are much more likely than others, some are much more tractable to attract than others, etc., a lot of this operates on variables that implicitly varying by orders of magnitudes across interventions and a checklist approach will massively under-represent the differences and is thus unlikely to point at the highest impact interventions.
Thanks for the clarification. Now I understand your point. Speaking for myself, because it’s easier, and I’m not consulting with my team...
I disagree that quantification always produces more reliable judgments and think it’s context dependent, but that’s a very fundamental disagreement which perhaps we can discuss in person one day. I’ll just say that in this particular case, we are very transparent about our systematic approach to collecting and aggregating the data we have to draw conclusions (incorporating probability and impact estimates) and will be providing all of the data for others to scrutinize. As I said in our post, we are also happy to present it right away in webinar form, so that others can draw their own conclusions or produce quantitative aggregations of them if they’d like.
And in fact our data—on the harm and containment of threats and the effectiveness and practicality of tactical response—has and will continue to inform the BOTECs that others in the EA space are already doing (in many cases many are currently doing this without the underlying evidence, because it takes time to conceptualize, sift through, and digest). And even without quantifying, it can directly help grantmakers or practitioners make decisions about whether to invest in a specific tactic to counter a specific threat, which is a decision those folks have to make every day. For example, say a statewide organization is anticipating a specific threat, like security forces at the polls and is trying to organize a program in response. Our table and analysis can help them understand what tactics exist to counter those threats and whether any of them are likely more effective than others.
Perhaps a part of the misunderstanding can be boiled down to a title issue. I shouldn’t have used a superlative. I should have just said: “what are high-leverage tactics,” which is actually what I meant.