I currently lead EA funds.
Before that, I worked on improving epistemics in the EA community at CEA (as a contractor), as a research assistant at the Global Priorities Institute, on community building, and Global Health Policy.
Unless explicitly stated otherwise, opinions are my own, not my employer’s.
You can give me positive and negative feedback here.
Yeah that’s fair. I’m a lot more bullish on getting AI systems that satisfy the linked question’s definition than my own one.
Did you look at the metaculus resolution criteria? They seem extremely weak to me, would be intersted to know which critiera you think o3 (or whatever the best OAI model is) is furthest away from.
AGI by 2028 is more likely than not
Most of my uncertainty is from potentially not understanding the criteria. They seem extremely weak to me:
* Able to reliably pass a Turing test of the type that would win the Loebner Silver Prize.
Able to score 90% or more on a robust version of the Winograd Schema Challenge, e.g. the “Winogrande” challenge or comparable data set for which human performance is at 90+%
Be able to score 75th percentile (as compared to the corresponding year’s human students; this was a score of 600 in 2016) on all the full mathematics section of a circa-2015-2020 standard SAT exam, using just images of the exam pages.
Be able to learn the classic Atari game “Montezuma’s revenge” (based on just visual inputs and standard controls) and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play (see closely-related question.)
I wouldn’t be surprised if we’ve already passed this.
Bioweapons are an existential risk
Note that imo almost all the x-risk from bio routes through AI, and is better thought of as an AI-risk threat model.
You might believe future GPU hours are currently underpriced (e.g. maybe we’ll soon develop AI systems that can automate valuable scientific research). In such a scenario, GPU hours would become much more valuable, while standard compute credits (which iiuc are essentially just money designated for computing resources) would not increase in value. Buying the underlying asset directly might be a straightforward way to invest in GPU hours now before their value increases dramatically.
Maybe there are cleverer ways to bet on price of GPU hours dramatically increasing that are conceptually simpler than nvidia share prices increasing, idk.
I think that “moods” should be a property of the whole discourse, as opposed to specific posts. I find it a bit annoying when commenters say a specific post has a missing mood—most posts don’t aim to represent the whole discourse.
I think this kind of investigation would be valuable, but I’m not sure what concrete questions you’d imagine someone answering to figure this out.
Yeah, I think starting research nonprofits is an especially different skillset. Good point.
Fwiw, I am a fan of open-source public goods. I think more nonprofits should try to build products rather than papers (or at least be in that mindset). I am a little worried that doing research is just more accepted than producing products that people might use and that creates unhelpful barriers. Relatedly, I quite like the move towards essay websites more care goes towards thinking about the UX and trying to actually interact with users.
I mostly agree with this. I wasn’t really aiming to give a balanced take here on whether people should start for-profit instead of nonprofit—I just meant to list a few (imo) underrated by EAs features of for-profits.
I’m more confused about the poor personal fit point. I suspect that many EAs are also a bad fit (at least initially) for starting nonprofits, but the EA ecosystem makes it somewhat easier for nonprofits to survive (which imo is probably a good thing overall).
The version of your claim I most agree with is:
* An EAs comparative advantage is identifying the most overlooked and big deal moral considerations, and access to minimal viable nonprofit capital
* This selects for highly neglected things where you might be able to have an impact at a small scale, with not particularly high standards
* So even if EAs aren’t sufficiently competent to build high-growth companies, they can still have an outsized impact via founding nonprofits
Makes sense. Sorry I didn’t say this before but I really appreciate your comments on this post and the high effort modelling that you did when working out what to work on. I think it’s a great example to set for the community and shows how seriously you can take these decisions (if you want to).
I don’t quite understand why you’re comparing donated money from people earning to give to money consumed by a charity.
I think a better comparison could be to just convert everything into impact adjusted dollars imagining that you’re able to sell off your impact equity. In this scheme it’s clearer that taking more money from value aligned funders is bad, whilst taking money from non-aligned funders is roughly neutral and donating a bunch of money is very good. The charity has to essentially “pay back” in impact the value aligned funder to get itself out of a hole, whereas the for profit doesn’t need to worry about that (on impact grounds).
To be clear, I think many nonprofits do a lot of good and are well worth funding—I spend most of my time trying to fund them—but it is harder to work out net positive if you’re consuming a bunch of fungible and value aligned resources.
Ok, I did also write:
* Nonprofits have significantly weaker feedback mechanisms compared to for-profits
* Few people are going to complain that you provided bad service when it didn’t cost them anything.
* Most nonprofits are not very ambitious, despite having large moral ambitions.
* It’s challenging to find talented people willing to accept a substantial pay cut to work with you.
* For-profits are considerably more likely to create something that people actually want.
Which I think are all differentially more useful in short timelines than in long timelines worlds as you don’t have less time to mess around, and you instead need to very quickly work on a useful thing. If you disagree with this maybe we’re talking past other and I misunderstand your perspective.
Fwiw I’m sympathetic to nonprofits being better on net than for profit in short-timelines for reasons that aren’t discussed in this post e.g. greater freedom to focus narrowly on useful work, particularly in cases where there isn’t a viable business model.
I’m not sure why you wouldn’t care about standards if timelines are short. I feel like you should care more about people actually using your work, which might be more likely if you are forced to get product-market fit.
Similarly, I would expect a great deal of for-profits to make things no one wants and burn a lot of capital in the process of going out of business, though that may not be a majority of the for-profits started by your average EA.
There are strong reasons to believe that profitable companies are producing goods or services that people want (which is not to say they are morally worthwhile). In most cases, the end user will simply not pay if the service they provide is bad. This just isn’t true in non-profits. I don’t think being an EA changes the calculus very much on the for-profit side, but importantly for profits have a strong kill function.
I’m not sure I follow your analysis—what does these numbers represent?
CE incubatee
p5: ~$100k
Median: ~$500K
Mean: ~$6M
p95: ~$23M
I do generally believe that at the point you’re big enough to have a significant impact, it’s quite likely that your investors will pressure you to squeeze money out of it in a way that will likely ruin said impact.
I think this could be a crux. Overall, I’m not very convinced that this effect is particularly strong relative to incentives that apply in the nonprofit case. Some rough heuristics for why I think it’s often overstated:
* Once a company is operating at scale (e.g. Series B) it’s rare to see large pivots, and if a change isn’t large why would it cost a lot of impact?
* Most incubators want people to work on things they are excited about, there’s only so much you can push people to change directions before the founder gets fed up.
* I suspect that investors might place a lot of pressure on companies to change the way they operate—but overall (relative to nonprofits) I think that’s probably a good thing.
I think I’m also less convinced than you are that the skillsets for for-profit and non-profit founders are highly transferable. I guess it depends a lot on the particular organisations.
This is great, thanks for sharing. I hadn’t come across it, but it matches my intuitions well.
Yeah, I think that program is cool. Although I’d probably be even more excited to see EAs participate in YC (or Entrepreneur First if they don’t have a cofounder).
The focus of the article was on reasons to start a for-profit, but I’m not sure that the risk of failure for for-profits is significantly higher than that of non-profits (ambition adjusted). Why do you think it’s so much higher?
I think the mainline plan looks more like use the best agents/model internally and release significantly less capable general agents/models, very capable but narrow agents/models, or AI generated products.