Hi Derek, hope you are doing well. Thank you for sharing your views on this analysis that you completed while you were at Rethink Priorities.
The difference between your estimates and Hauke’s certainly made our work more interesting.
A few points that may be of general interest:
For both analysts we used 3 estimates, an ‘optimistic guess’, ‘best guess’ and ‘pessimistic guess’.
For users from middle-income countries we doubled the impact estimates. Without reviewing our report/notes in detail, I don’t recall the rationale for the specific value of this multiplier. The basic idea is that high-income countries are better served, more competitive markets, so apps are more likely to find users with worse counterfactuals in middle income countries.
The estimates were meant to be conditional on Mind Ease achieving some degree of success. We simply assumed the impact of failure scenarios is 0. Hauke’s analysis seems to have made more clear use of this aspect. Not only is Hauke’s reading of the literature more optimistic, but he is more optimistic about how much more effective a successful Mind Ease will be relative to the competition.
Indeed the values we used for Derek’s analysis, for high income countries, were all less than 0.01. We simplified the 3 estimates, doing a weighted average across the two types of countries, into the single value of 0.01 for Derek’s analysis after rounding up (I think the true number may be more like 0.006). The calculations in the post use rounded values so it is easier for a reader to follow. Nevertheless, the results are in line with our more detailed calculations in the original report.
Similar to this point of rounding, we simplified the explanation of the robustness tilt we applied. It wasn’t just about Derek vs Hauke. It was also along the dimensions of the business analysis (e.g. success probabilities). We simplified the framing of the robustness tilt both here and in a ‘Fermi Estimate’ section of the original report because we believed that it is conceptually clearer to only talk about the one dimension.
What would I suggest to someone who would like to penalize the estimate more or less for all the uncertainty? Adjust the impact return.
How can you adjust the impact return in a consistent way? Of course, to make analyses like this useful you would want to do them in a consistent fashion. There isn’t a golden standard for how to control the strength of the robustness tilts we used. But you can think of the tilt we applied (in the original report) as being like being told a coin is fair (50/50) and then assuming it is biased to 80% heads (if heads is the side you don’t want). This is an expression of how different our tilted probability distribution was from the distribution in the base model (the effect on the impact estimate was more severe; 1-0.02/(0.25/2+0.01/2)=85%). There is a way of assessing this “degree of coin-equivalent tilt” for any tilt of any model. So if you felt another startup had the same level of uncertainty as Mind Ease you could tilt your model of it until you get the same degree of tilt. This would give you some consistency and not make the tilts based purely in analyst intuition (though of course there is basically no way to avoid some bias). If a much better way to consistently manage these tilts was developed, we would happily use it.
Overall, this analysis is just one example of how one might deal with all the things that make such assessments difficult including impact uncertainty, business uncertainty, and analyst disagreement. The key point really being a need to summarize all the uncertainty in a way that is useful to busy, non-technical decision makers who aren’t going to look at the underlying distributions. We look forward to seeing how techniques in this regard evolve as more and more impact assessments are done and shared publicly.
Hi Derek, hope you are doing well. Thank you for sharing your views on this analysis that you completed while you were at Rethink Priorities.
The difference between your estimates and Hauke’s certainly made our work more interesting.
A few points that may be of general interest:
For both analysts we used 3 estimates, an ‘optimistic guess’, ‘best guess’ and ‘pessimistic guess’.
For users from middle-income countries we doubled the impact estimates. Without reviewing our report/notes in detail, I don’t recall the rationale for the specific value of this multiplier. The basic idea is that high-income countries are better served, more competitive markets, so apps are more likely to find users with worse counterfactuals in middle income countries.
The estimates were meant to be conditional on Mind Ease achieving some degree of success. We simply assumed the impact of failure scenarios is 0. Hauke’s analysis seems to have made more clear use of this aspect. Not only is Hauke’s reading of the literature more optimistic, but he is more optimistic about how much more effective a successful Mind Ease will be relative to the competition.
Indeed the values we used for Derek’s analysis, for high income countries, were all less than 0.01. We simplified the 3 estimates, doing a weighted average across the two types of countries, into the single value of 0.01 for Derek’s analysis after rounding up (I think the true number may be more like 0.006). The calculations in the post use rounded values so it is easier for a reader to follow. Nevertheless, the results are in line with our more detailed calculations in the original report.
Similar to this point of rounding, we simplified the explanation of the robustness tilt we applied. It wasn’t just about Derek vs Hauke. It was also along the dimensions of the business analysis (e.g. success probabilities). We simplified the framing of the robustness tilt both here and in a ‘Fermi Estimate’ section of the original report because we believed that it is conceptually clearer to only talk about the one dimension.
What would I suggest to someone who would like to penalize the estimate more or less for all the uncertainty? Adjust the impact return.
How can you adjust the impact return in a consistent way? Of course, to make analyses like this useful you would want to do them in a consistent fashion. There isn’t a golden standard for how to control the strength of the robustness tilts we used. But you can think of the tilt we applied (in the original report) as being like being told a coin is fair (50/50) and then assuming it is biased to 80% heads (if heads is the side you don’t want). This is an expression of how different our tilted probability distribution was from the distribution in the base model (the effect on the impact estimate was more severe; 1-0.02/(0.25/2+0.01/2)=85%). There is a way of assessing this “degree of coin-equivalent tilt” for any tilt of any model. So if you felt another startup had the same level of uncertainty as Mind Ease you could tilt your model of it until you get the same degree of tilt. This would give you some consistency and not make the tilts based purely in analyst intuition (though of course there is basically no way to avoid some bias). If a much better way to consistently manage these tilts was developed, we would happily use it.
Overall, this analysis is just one example of how one might deal with all the things that make such assessments difficult including impact uncertainty, business uncertainty, and analyst disagreement. The key point really being a need to summarize all the uncertainty in a way that is useful to busy, non-technical decision makers who aren’t going to look at the underlying distributions. We look forward to seeing how techniques in this regard evolve as more and more impact assessments are done and shared publicly.