Note: This message came out of a conversation with u/AppliedDivinityStudies and therefore contains a mix of opinions from the two of us, even though I use “I” throughout. All mistakes can be attributed to me (An1lam) though.
Really appreciate you all running this program and writing this up! That said, I disagree with a number of the conclusions in the write-up and worry that if neither I nor anyone else speak up with our criticisms, people will get the (in my opinion) wrong idea about bottlenecks to more longtermist entrepreneurship.
At a high level, many of my criticisms stem from my sense that the program didn’t lean in to the “entrepreneurship” component that hard and as a result ended up looking a lot like typical EA activities (nothing wrong with typical EA activities).
First, I strongly disagree with the implicit conclusion that fostering a LE requires lots of existing LE entrepreneurs, specifically:
Hundreds of people expressed interest in doing LE, but a very small number of these (1-3 dozen) had backgrounds in both longtermism and entrepreneurship. There were few people that we thought could pull off very ambitious projects.
And also:
Talent pool is larger than expected, but less senior.
If there existed a large pool of LE entrepreneurs with the right skills, there’d be a less pressing need for this sort of program. I get that you’re wary of analogies to tech startups due to downside risk but to the degree one wants to foster an ecosystem, taking a risk on at least some more junior people seems pretty necessary. Even within the EA ecosystem, my sense is that people who founded successful orgs often hadn’t done this before. E.g., as far as I know Nick Bostrom hadn’t founded an FHI 0.0 before founding the current instantiation of FHI. Same for GiveWell, CEA, etc. Given that, the notion that doing LE entrepreneurship requires “backgrounds in both longtermism and entrepeneurship” seems like too restrictive a filter.
Second, without examples it’s a little hard to discuss, but I feel like the concern about downside risk is real but overblown. It’s definitely an important difference between LE entrepeneurship and traditional startups to be mindful of but I question whether it’s being used to justify an extreme form of the precautionary principle that says funders shouldn’t fund ideas with downside risks instead of the, more reasonable IMO, principle of funding +EV things or trying to ensure the portfolio of projects has +EV.
Third, I think some of the assumptions about what types of activities should take precedence for LE entrepreneurship deserve re-examining. As I alluded to above, it seems like the activities you say matter most for LE entrepeneurship, “research, strategic thinking, forecasting long-run consequences, and introspection [rather] than finding product-market fit” are suspiciously similar to “typical EA activities”. From my perspective, it could instead be interesting to try and take some of the startup gospel around iteration, getting things out into the wild sooner rather than later, etc. seriously and adapt them to LE entrepreneurship rather than starting from the (appearance of the) assumption that you have very little to learn from that world. This isn’t fully charitable, but I have the sense that EA has a lot of people who gravitate towards talking/strategizing/coordinating and getting other people to do things but sometimes shy away from “actually doing things” themselves. I view an LE entrepreneurship incubator as an opportunity to reward or push more people towards the “actually doing things” part. Part of this may also be that I’m a bit confused about where the boundary between normal and LE entrepreneurship lies. In my mind, SpaceX, fusion startups, psychedelics research would all qualify as examples of LE entrepreneurship with limited downside risk or at least not existential downside risks. Would you agree that these qualify as good examples?
Fourth, you mention advisors but only say a few by name. I’m 1) curious whether any of these advisors were experienced entrepreneurs and 2) interested in whether you considered getting advisors only adjacent to EA but very experienced entrepreneurs. As an example, at least one founder of Wave is an EA-aligned successful entrepreneurs who I can only imagine has wisdom to impart about entrepreneurship. I don’t live in the Bay Area but I have the sense that there are quite a few other EA-adjacent founders there who might also be interested in advising a program like this.
Fifth, this is more low-level but I still don’t really understand the skepticism of a YC-like incubator for LE entrepreneurship. It seems like your arguments boil down to 1) the current pool is small and 2) the requirements are different. But on 1, when YC started, the pool of entrepreneurs was smaller too! Such a program can help to increase the size of that pool. On 2, I agree that a literal copy of YC would have the issues you describe but I’d imagine a YC-like program blending the two community’s thinking styles in a way that gets most of the benefits of each while avoiding the downsides. As an aside, we are also very supportive of longtermists doing YC but for slightly different reasons. This may also be related to the confusion about what qualifies as LEE.
Summarizing, my goal in writing this comment is not to just criticize the program. Instead, I worry that by highlighting the need for experience and the overwhelming risk of harm, the write-up as-is might discourage would-be LE entrepreneurs from trying something . I hope that my comment can help provide a counterweight to that.
FWIW, as someone who previously warned about risk of accidental harm, I personally mostly agree with this comment. I think what I care more is “option value to shut projects down if they turn out to be harmful” than preventing damage in the first place (with the exception of projects that have very large negative effects from the very beginning).
I think offering funding & advice causes more people to work with you, and the closer they are working with you, the larger the influence your opinion is likely to have on the question of whether they should shut down their project.
Thanks for the comment An1lam, and apologies for the delay! Really appreciate the engagement.
Some thoughts in response:
On (1), I agree that fostering an ecosystem doesn’t require a load of experienced people; you’re very much right that that’s the whole problem that we’re trying to solve in the first place! What we mostly mean to say in terms of pointing to the lack of requisite experience is that we found that there weren’t that many individuals who are ready to hit the ground running with founding a substantial longtermist start-up right now; hence a lot of LE activities that we would have been excited about pursuing instead (and which are mentioned in the “what we’d be excited about” section) are targeted at making up for this lack of experience by e.g. fostering a community, bringing folks in-house in a foundry.
On (2), as you say, it’s difficult to talk about this in the abstract, and certainly I feel sympathetic to the worry that we might be being too cautious (that was one of the motivations for me starting this project in the first place). With that said, I do think thinking thoroughly through downside risk considerations and treating them more seriously than is the norm among founders writ large is something that I want to encourage of all longtermist entrepreneurs, and I’d worry about not doing it enough more than I’d worry about doing it too much given what’s at stake. The thing that I’m interested in advocating for here, though, is something like demonstration of thought, care, and consideration, rather than avoiding projects with downside risk altogether (I think the two often are conflated); for example, if it turns out that after a reasonable amount of thinking there is some amount of downside risk, but the EV of the project is still high & the founder has identified reasonable things they could do to mitigate this downside risk, I (and I imagine most EA funders) would be supportive of that project going ahead.
On (3), seems right that EAs in general, and particularly folks trying to start new projects, have a fair amount that they could learn from ‘traditional’ start-up best practice (that was a big part of the framing that we took in designing curricula materials for the fellowship, for example). I’m not sure where we disagree here; I do also think that strategic thinking and introspection are important, and perhaps uniquely so for start-ups which don’t follow the usual product-fit dynamic (although I think several do, even something like a new research org), or where there are very slow feedback loops to use in the interim, hence why it might be unusually important for longtermist entrepreneurs to have this trait. But I don’t think thinking this is important is mutually exclusive to thinking that learning from more mainstream start-up practice is useful.
On (4), advisors were a combination of experienced entrepreneurs (some of whom also happened to be quite engaged with EA, some less so) and domain experts.
On (5), basically Jonas’ comment captures the main thing—that given our epistemic state in longtermist cause areas at the moment, it just seems quite difficult to come up with start-up ideas that fit the typical YC model (find product-market fit, then scale), which means that the idea generation process I think needs to be quite a bit more involved, be advised by domain experts, etc. and the build-iterate cycle necessarily needs to be tweaked from the usual dynamic expected for e.g. software products. I.e. mostly leaning on (2) of ‘the requirements are different’ rather than the pool being small.
Two additional high level things that come to mind in response:
1 - We definitely don’t want to discourage promising longtermist entrepreneurs who are excited about starting something in this space, the intent was to clarify for future potential incubator founders what we tried and learned.
2 - In terms of your overall concern that the program wasn’t entrepreneurial enough, it’s a bit hard to comment. At the end of the day, we do think that simply replicating traditional entrepreneurship in this space won’t quite work, but also do think that the EA community could learn a ton from the entrepreneurship community (and designed our program with both of those in mind). It’s possible you disagree, or possible that the tone and seriousness with which we centered entrepreneurship didn’t come through in this post.
Thanks so much for your reply. This actually really helped clarify things for me. I think we may still have some different priors about (2) but overall your comment made me think we agree much more than we disagree (and than I’d previously thought we’d disagreed).
I again just want to note that I’m grateful you ran the program and engaged so productively with my comment.
Regarding a YC incubator model, I think the main issue is just that people rarely generate sufficiently well-targeted and ambitious startup ideas. I really don’t think we need another dozen donation apps or fundraising orgs, but that’s what people often come up with. I think we’d want something that does more to help people develop better ideas. (Perhaps that’s what you had in mind as well.)
A quick thought on having a YC-style programme and taking risks on more junior talent:
Domain expertise is important—I think YC would agree on this. If taking on a deep tech startup they would look for someone on the team who had domain expertise in the field.
I think early YC Internet startups like Dropbox or Airbnb make it look like domain expertise is less important and it’s more about just getting stuck in. The difference is that when Dropbox started there was no expert in “files on the internet” so the founders could basically become the world experts just by getting stuck in and working on it.
The difference with Longtermist areas like AI and Bio is you can’t just become the expert by working on it and (yes, you guessed it) the downside risk means we don’t want to take bets on people to just go for it and try it out (unlike Dropbox where it doesn’t really matter if it fails catastrophically).
Note: This message came out of a conversation with u/AppliedDivinityStudies and therefore contains a mix of opinions from the two of us, even though I use “I” throughout. All mistakes can be attributed to me (An1lam) though.
Really appreciate you all running this program and writing this up! That said, I disagree with a number of the conclusions in the write-up and worry that if neither I nor anyone else speak up with our criticisms, people will get the (in my opinion) wrong idea about bottlenecks to more longtermist entrepreneurship.
At a high level, many of my criticisms stem from my sense that the program didn’t lean in to the “entrepreneurship” component that hard and as a result ended up looking a lot like typical EA activities (nothing wrong with typical EA activities).
First, I strongly disagree with the implicit conclusion that fostering a LE requires lots of existing LE entrepreneurs, specifically:
And also:
If there existed a large pool of LE entrepreneurs with the right skills, there’d be a less pressing need for this sort of program. I get that you’re wary of analogies to tech startups due to downside risk but to the degree one wants to foster an ecosystem, taking a risk on at least some more junior people seems pretty necessary. Even within the EA ecosystem, my sense is that people who founded successful orgs often hadn’t done this before. E.g., as far as I know Nick Bostrom hadn’t founded an FHI 0.0 before founding the current instantiation of FHI. Same for GiveWell, CEA, etc. Given that, the notion that doing LE entrepreneurship requires “backgrounds in both longtermism and entrepeneurship” seems like too restrictive a filter.
Second, without examples it’s a little hard to discuss, but I feel like the concern about downside risk is real but overblown. It’s definitely an important difference between LE entrepeneurship and traditional startups to be mindful of but I question whether it’s being used to justify an extreme form of the precautionary principle that says funders shouldn’t fund ideas with downside risks instead of the, more reasonable IMO, principle of funding +EV things or trying to ensure the portfolio of projects has +EV.
Third, I think some of the assumptions about what types of activities should take precedence for LE entrepreneurship deserve re-examining. As I alluded to above, it seems like the activities you say matter most for LE entrepeneurship, “research, strategic thinking, forecasting long-run consequences, and introspection [rather] than finding product-market fit” are suspiciously similar to “typical EA activities”. From my perspective, it could instead be interesting to try and take some of the startup gospel around iteration, getting things out into the wild sooner rather than later, etc. seriously and adapt them to LE entrepreneurship rather than starting from the (appearance of the) assumption that you have very little to learn from that world. This isn’t fully charitable, but I have the sense that EA has a lot of people who gravitate towards talking/strategizing/coordinating and getting other people to do things but sometimes shy away from “actually doing things” themselves. I view an LE entrepreneurship incubator as an opportunity to reward or push more people towards the “actually doing things” part. Part of this may also be that I’m a bit confused about where the boundary between normal and LE entrepreneurship lies. In my mind, SpaceX, fusion startups, psychedelics research would all qualify as examples of LE entrepreneurship with limited downside risk or at least not existential downside risks. Would you agree that these qualify as good examples?
Fourth, you mention advisors but only say a few by name. I’m 1) curious whether any of these advisors were experienced entrepreneurs and 2) interested in whether you considered getting advisors only adjacent to EA but very experienced entrepreneurs. As an example, at least one founder of Wave is an EA-aligned successful entrepreneurs who I can only imagine has wisdom to impart about entrepreneurship. I don’t live in the Bay Area but I have the sense that there are quite a few other EA-adjacent founders there who might also be interested in advising a program like this.
Fifth, this is more low-level but I still don’t really understand the skepticism of a YC-like incubator for LE entrepreneurship. It seems like your arguments boil down to 1) the current pool is small and 2) the requirements are different. But on 1, when YC started, the pool of entrepreneurs was smaller too! Such a program can help to increase the size of that pool. On 2, I agree that a literal copy of YC would have the issues you describe but I’d imagine a YC-like program blending the two community’s thinking styles in a way that gets most of the benefits of each while avoiding the downsides. As an aside, we are also very supportive of longtermists doing YC but for slightly different reasons. This may also be related to the confusion about what qualifies as LEE.
Summarizing, my goal in writing this comment is not to just criticize the program. Instead, I worry that by highlighting the need for experience and the overwhelming risk of harm, the write-up as-is might discourage would-be LE entrepreneurs from trying something . I hope that my comment can help provide a counterweight to that.
FWIW, as someone who previously warned about risk of accidental harm, I personally mostly agree with this comment. I think what I care more is “option value to shut projects down if they turn out to be harmful” than preventing damage in the first place (with the exception of projects that have very large negative effects from the very beginning).
I think offering funding & advice causes more people to work with you, and the closer they are working with you, the larger the influence your opinion is likely to have on the question of whether they should shut down their project.
Thanks for the comment An1lam, and apologies for the delay! Really appreciate the engagement.
Some thoughts in response:
On (1), I agree that fostering an ecosystem doesn’t require a load of experienced people; you’re very much right that that’s the whole problem that we’re trying to solve in the first place! What we mostly mean to say in terms of pointing to the lack of requisite experience is that we found that there weren’t that many individuals who are ready to hit the ground running with founding a substantial longtermist start-up right now; hence a lot of LE activities that we would have been excited about pursuing instead (and which are mentioned in the “what we’d be excited about” section) are targeted at making up for this lack of experience by e.g. fostering a community, bringing folks in-house in a foundry.
On (2), as you say, it’s difficult to talk about this in the abstract, and certainly I feel sympathetic to the worry that we might be being too cautious (that was one of the motivations for me starting this project in the first place). With that said, I do think thinking thoroughly through downside risk considerations and treating them more seriously than is the norm among founders writ large is something that I want to encourage of all longtermist entrepreneurs, and I’d worry about not doing it enough more than I’d worry about doing it too much given what’s at stake. The thing that I’m interested in advocating for here, though, is something like demonstration of thought, care, and consideration, rather than avoiding projects with downside risk altogether (I think the two often are conflated); for example, if it turns out that after a reasonable amount of thinking there is some amount of downside risk, but the EV of the project is still high & the founder has identified reasonable things they could do to mitigate this downside risk, I (and I imagine most EA funders) would be supportive of that project going ahead.
On (3), seems right that EAs in general, and particularly folks trying to start new projects, have a fair amount that they could learn from ‘traditional’ start-up best practice (that was a big part of the framing that we took in designing curricula materials for the fellowship, for example). I’m not sure where we disagree here; I do also think that strategic thinking and introspection are important, and perhaps uniquely so for start-ups which don’t follow the usual product-fit dynamic (although I think several do, even something like a new research org), or where there are very slow feedback loops to use in the interim, hence why it might be unusually important for longtermist entrepreneurs to have this trait. But I don’t think thinking this is important is mutually exclusive to thinking that learning from more mainstream start-up practice is useful.
On (4), advisors were a combination of experienced entrepreneurs (some of whom also happened to be quite engaged with EA, some less so) and domain experts.
On (5), basically Jonas’ comment captures the main thing—that given our epistemic state in longtermist cause areas at the moment, it just seems quite difficult to come up with start-up ideas that fit the typical YC model (find product-market fit, then scale), which means that the idea generation process I think needs to be quite a bit more involved, be advised by domain experts, etc. and the build-iterate cycle necessarily needs to be tweaked from the usual dynamic expected for e.g. software products. I.e. mostly leaning on (2) of ‘the requirements are different’ rather than the pool being small.
Two additional high level things that come to mind in response:
1 - We definitely don’t want to discourage promising longtermist entrepreneurs who are excited about starting something in this space, the intent was to clarify for future potential incubator founders what we tried and learned.
2 - In terms of your overall concern that the program wasn’t entrepreneurial enough, it’s a bit hard to comment. At the end of the day, we do think that simply replicating traditional entrepreneurship in this space won’t quite work, but also do think that the EA community could learn a ton from the entrepreneurship community (and designed our program with both of those in mind). It’s possible you disagree, or possible that the tone and seriousness with which we centered entrepreneurship didn’t come through in this post.
Hey Jade,
Thanks so much for your reply. This actually really helped clarify things for me. I think we may still have some different priors about (2) but overall your comment made me think we agree much more than we disagree (and than I’d previously thought we’d disagreed).
I again just want to note that I’m grateful you ran the program and engaged so productively with my comment.
Really glad to hear, and in turn, really appreciate your engagement with the post!
Regarding a YC incubator model, I think the main issue is just that people rarely generate sufficiently well-targeted and ambitious startup ideas. I really don’t think we need another dozen donation apps or fundraising orgs, but that’s what people often come up with. I think we’d want something that does more to help people develop better ideas. (Perhaps that’s what you had in mind as well.)
Honestly I wasn’t too sure what the biggest issue was but what you described seems reasonable to me!
A quick thought on having a YC-style programme and taking risks on more junior talent:
Domain expertise is important—I think YC would agree on this. If taking on a deep tech startup they would look for someone on the team who had domain expertise in the field.
I think early YC Internet startups like Dropbox or Airbnb make it look like domain expertise is less important and it’s more about just getting stuck in. The difference is that when Dropbox started there was no expert in “files on the internet” so the founders could basically become the world experts just by getting stuck in and working on it.
The difference with Longtermist areas like AI and Bio is you can’t just become the expert by working on it and (yes, you guessed it) the downside risk means we don’t want to take bets on people to just go for it and try it out (unlike Dropbox where it doesn’t really matter if it fails catastrophically).