There’s been a few posts recently about how there should be more EA failures, since we’re trying a bunch of high-risk, high-reward projects, and some of them should fail or we’re not being ambitious enough.
I think this is a misunderstanding of what high-EV bets look like. Most projects do not either produce wild success or abject failure, there’s usually a continuity of outcomes in between, and that’s what you hit. This doesn’t look like “failure”, it looks like moderate success.
For example, consider the MineRL BASALT competition that I organized. The low-probability, high-value outcome would have had hundreds or thousands of entries to the competition, several papers produced as a result, and the establishment of BASALT as a standard benchmark and competition in the field.
What actually happened was that we got ~11 submissions, of which maybe ~5 were serious, made decent progress on the problem, produced a couple of vaguely interesting papers, some people in the field have heard about the benchmark and occasionally use it, and we built enough excitement in the team that the competition will (very likely) run again this year.
Is this failure? It certainly isn’t what normally comes to mind from the normal meaning of “failure”. But it was:
Below my median expectation for what the competition would accomplish
Not something I would have put time into if someone had told me in advance exactly what it would accomplish so far, and the time cost needed to get it.
One hopes that roughly 50% of the things I do meet the first criterion, and probably 90% of the things I’d do would meet the second. But also maybe 90% of the work I do is something people would say was “successful” even ex post.
If you are actually seeing failures for relatively large projects that look like “failures” in the normal English sense of the word, where basically nothing was accomplished at all, I’d be a lot more worried that actually your project was not in fact high-EV even ex ante, and you should be updating a lot more on your failure, and it is a good sign that we don’t see that many EA “failures” in this sense.
(One exception to this is earning-to-give entrepreneurship, where “we had to shut the company down and made ~no money after a year of effort” seems reasonably likely and it still would plausibly be high-EV ex ante.)
There’s been a few posts recently about how there should be more EA failures, since we’re trying a bunch of high-risk, high-reward projects, and some of them should fail or we’re not being ambitious enough.
I think this is a misunderstanding of what high-EV bets look like. Most projects do not either produce wild success or abject failure, there’s usually a continuity of outcomes in between, and that’s what you hit. This doesn’t look like “failure”, it looks like moderate success.
For example, consider the MineRL BASALT competition that I organized. The low-probability, high-value outcome would have had hundreds or thousands of entries to the competition, several papers produced as a result, and the establishment of BASALT as a standard benchmark and competition in the field.
What actually happened was that we got ~11 submissions, of which maybe ~5 were serious, made decent progress on the problem, produced a couple of vaguely interesting papers, some people in the field have heard about the benchmark and occasionally use it, and we built enough excitement in the team that the competition will (very likely) run again this year.
Is this failure? It certainly isn’t what normally comes to mind from the normal meaning of “failure”. But it was:
Below my median expectation for what the competition would accomplish
Not something I would have put time into if someone had told me in advance exactly what it would accomplish so far, and the time cost needed to get it.
One hopes that roughly 50% of the things I do meet the first criterion, and probably 90% of the things I’d do would meet the second. But also maybe 90% of the work I do is something people would say was “successful” even ex post.
If you are actually seeing failures for relatively large projects that look like “failures” in the normal English sense of the word, where basically nothing was accomplished at all, I’d be a lot more worried that actually your project was not in fact high-EV even ex ante, and you should be updating a lot more on your failure, and it is a good sign that we don’t see that many EA “failures” in this sense.
(One exception to this is earning-to-give entrepreneurship, where “we had to shut the company down and made ~no money after a year of effort” seems reasonably likely and it still would plausibly be high-EV ex ante.)