I currently lead EA funds.
Before that, I worked on improving epistemics in the EA community at CEA (as a contractor), as a research assistant at the Global Priorities Institute, on community building, and Global Health Policy.
Unless explicitly stated otherwise, opinions are my own, not my employer’s.
You can give me positive and negative feedback here.
I’m not sure why you wouldn’t care about standards if timelines are short. I feel like you should care more about people actually using your work, which might be more likely if you are forced to get product-market fit.
Similarly, I would expect a great deal of for-profits to make things no one wants and burn a lot of capital in the process of going out of business, though that may not be a majority of the for-profits started by your average EA.
There are strong reasons to believe that profitable companies are producing goods or services that people want (which is not to say they are morally worthwhile). In most cases, the end user will simply not pay if the service they provide is bad. This just isn’t true in non-profits. I don’t think being an EA changes the calculus very much on the for-profit side, but importantly for profits have a strong kill function.
I’m not sure I follow your analysis—what does these numbers represent?
CE incubatee
p5: ~$100k
Median: ~$500K
Mean: ~$6M
p95: ~$23M
I do generally believe that at the point you’re big enough to have a significant impact, it’s quite likely that your investors will pressure you to squeeze money out of it in a way that will likely ruin said impact.
I think this could be a crux. Overall, I’m not very convinced that this effect is particularly strong relative to incentives that apply in the nonprofit case. Some rough heuristics for why I think it’s often overstated:
* Once a company is operating at scale (e.g. Series B) it’s rare to see large pivots, and if a change isn’t large why would it cost a lot of impact?
* Most incubators want people to work on things they are excited about, there’s only so much you can push people to change directions before the founder gets fed up.
* I suspect that investors might place a lot of pressure on companies to change the way they operate—but overall (relative to nonprofits) I think that’s probably a good thing.
I think I’m also less convinced than you are that the skillsets for for-profit and non-profit founders are highly transferable. I guess it depends a lot on the particular organisations.
This is great, thanks for sharing. I hadn’t come across it, but it matches my intuitions well.
Yeah, I think that program is cool. Although I’d probably be even more excited to see EAs participate in YC (or Entrepreneur First if they don’t have a cofounder).
The focus of the article was on reasons to start a for-profit, but I’m not sure that the risk of failure for for-profits is significantly higher than that of non-profits (ambition adjusted). Why do you think it’s so much higher?
To be clear, I’m open to building broad coalitions and think that a good documentary could/would feature content on low-stakes risks; but, I believe people should be transparent about their motivations and avoid conflating non-GCR stuff with GCR stuff.
Do you have a list of research questions that you think could easily be sped up with AI systems? I suspect that I’m more pessimistic than you are due to concerns around scheming AI agents doing intentional research sabotage and think that the affordances of AI agents might make some currently intractable agendas more tractable.
Thank you for replying—it’s great that someone within the industry shared their perspective!
I don’t really understand why that would make the US building DCs in allied countries destabilising. The short answer for why it might be stabilising is:
* It gives non-US actors more leverage, making deals where benefits are shared more likely.
* It’s harder for the US to defect on commitments to develop models safely and not misuse them if it’s easy for their allies to spy on them (or they have made commitments for DC use to be monitored)
* It keeps the Western democracies ahead of the CCP.
I think that allied countries themselves building DCs might be comparably stabilising—it gives more leverage to allied countries, at the cost of baking in less coordination and affordances to make deals around how AI is used and developed.
Some quick takes in a personal capacity:
I agree that a good documentary about AI risk could be very valuable. I’m excited about broad AI risk outreach and few others seem to be stepping up. The proposal seem ambitious and exciting.
I suspect that a misleading documentary would be mildly net-negative, and it’s easy to be misleading. So far, a significant fraction of public communications from the AI safety community has been fairly misleading (definitely not all—there is some great work out there as well).
In particular, equivocating between harms like deepfakes and GCRs seems pretty bad. I think it’s fine to mention non-catastrophic harms, but often, the benefits of AI systems seem likely to dwarf them. More cooperative (and, in my view, effective) discourse should try to mention the upsides and transparently point to the scale of different harms.
In the past, team members have worked on (or at least in the same organisation) comms efforts that seemed low integrity and fairly net-negative to me (e.g., some of their work on deepfakes, and adversarial mobile billboards around the UK AI Safety summit). Idk if these specific team members were involved in those efforts.
The team seems very agentic and more likely to succeed than most “field-building” AIS teams.
Their plan seems pretty good to me (though I am not an expert in the area). I’m pretty into people just trying things. Seems like there are too few similar efforts, and like we could regret not making more stuff like this happen, particularly if your timelines are short.
I’m a bit confused. Some donors should be very excited about this, and others should be much more on the fence or think it’s somewhat net-negative. Overall, I think it’s probably pretty promising.
Sorry, I agree this message is somewhat misleading—I’ll ask our ops team to review this.
Thanks. We should probably try to display this on our website properly. We have been able to fund for-profits in the past, but it is pretty difficult. I don’t think the only reason we passed on your application was that it’s for-profit, but that did make our bar much higher (this is a consequence of US/UK charity law and isn’t a reflection on the impact of non-profits/for-profits).
By the way, I personally think that your project should probably be a for-profit, as it will be easier to raise funding, users will hold you to higher standards, and your team seems quite value-aligned.
Some AI research projects that (afaik) haven’t had much work done on them and would be pretty interesting:
If the US were to co-build secure data centres in allied countries, would that be geopolitically stabilising or destabilising?
What AI safety research agendas could be massively sped up by AI agents? What properties do they have (e.g. easily checkable, engineering > conceptual …)?
What will the first non-AI R&D uses of powerful and general AI systems be?
Are there ways to leverage cheap (e.g. 100x lower than present-day cost) intelligence or manual labour to massively increase the US’s electricity supply?
What kinds of regulation might make it easier to navigate an intelligence explosion (e.g. establishing quick pathways to implement policy informed by AI experts, or establishing zones where compute facilities can be quickly built without navigating a load of red tape)?
Given that they’ve made a public Manifund application, it seems fine to share that there has been quite a lot of discussion about this project on the LTFF internally. I don’t think we are in a great place to share our impressions right now, but if Connor would like me to, I’d be happy to share some of my takes in a personal capacity.
I think the main reasons that EAs are working on AI stuff over bio stuff is that there aren’t many good routes into worst case bio work afaict largely due to infohazard concerns from field building, and the x-risk case for biorisk not being very compelling (maybe due to infohazard concerns around threat models).
I’m interested in figuring out “what skill profiles are most leveraged for altruistic work after we get the first competent AI agents”?
I think this might be one of the most important questions for field-building orgs to work on (in any cause area but particularly AI safety). I think 80k and other AIS-motivated careers groups should try to think hard about this question and what it means for their strategy going forward.
I’m optimistic about productivity on the question of “what skill profiles are most leveraged after we get the first competent AI-agents” because:
few people have actually tried to think about what managing an AI workforce would look like
this doesn’t feel conceptually abstract (e.g. some people have already adopted LMs in their day-to-day workflows)
something like drop-in replacements for human remote workers are more likely than AIs less analogous to human workers if timelines are short (and if timelines are short, this work is more urgent than if timelines are long)
One important consideration is whether human managers are going to be good proxies for AI-agent managers. It’s plausible to me that the majority of object-level AI safety work that has ever be done will be done by AI agents. It could be that the current status quo persists, where people’s leverage routes through things like leadership and management skills, or it could be that ~everyone will become more leveraged proportional to their access to AI models (roughly their budget) due to AI agents having different affordances than humans, or something else.
For example, suppose that AI agents are much more tolerant, hardworking, and understanding of user intent for short tasks than human workers. I expect that having good strategic takes would become much more leveraged and having good internal stakeholder management skills would become much less leveraged (relative to today). If 80k (for example) thought that was likely, then maybe they should double down on finding people with great strategy takes and care less about typical management skills.
Maybe “junior” roles will be automated before “senior” roles almost uniformly across the economy. In that case, it’s even more valuable than it is right now to focus on getting “senior” people.
Or maybe coding will be automated before everything else, and for some reason people’s relative leverage inside the company stays pretty fixed, or it’s too hard to say which kinds of jobs will be most leveraged at software companies. Then “coding” still becomes much more leveraged per dollar than today, and it’s probably useful to find more value-aligned coders.
It would be surprising to me if the price of certain kinds of intellectual labour decreased by 100x, and this had little impact on people’s relative leverage.
(disclaimer: I’m not sure whether the above stories actually go through, they are meant to just be illustrative of the kind of thinking that seems undersupplied)
I thought that David meant “intellectual areas” as opposed to physical/digital venues. Seems like lots of discussion in those venues isn’t particularly decision-relevant, and timelines aren’t really an unusually decision-irrelevant case.
I’m not directly working on an AI pause, but if I were, I would think that timelines were very strategically relevant:
How soon will we see warning shots, and will they arrive before society is disempowered?
What kinds of preparation should I do before we see a warning shot
Will progress be sufficiently slow such that we get frog-boiled?
How quickly should we aim to spend our limited human capital and financial resources?
How “in the water” will (or could) AI be over the next few years?
Discussion of AI timelines is helpful for answering many of the questions above, as are the endline forecasts. I suspect that I believe that working on AI pauses is much less robust to timelines than you think it is, and discussion of timelines is much more helpful for developing models of AI development and risk (and those models are highly decision-relevant).
Ok, I did also write:
* Nonprofits have significantly weaker feedback mechanisms compared to for-profits
* Few people are going to complain that you provided bad service when it didn’t cost them anything.
* Most nonprofits are not very ambitious, despite having large moral ambitions.
* It’s challenging to find talented people willing to accept a substantial pay cut to work with you.
* For-profits are considerably more likely to create something that people actually want.
Which I think are all differentially more useful in short timelines than in long timelines worlds as you don’t have less time to mess around, and you instead need to very quickly work on a useful thing. If you disagree with this maybe we’re talking past other and I misunderstand your perspective.
Fwiw I’m sympathetic to nonprofits being better on net than for profit in short-timelines for reasons that aren’t discussed in this post e.g. greater freedom to focus narrowly on useful work, particularly in cases where there isn’t a viable business model.