I currently lead EA funds.
Before that, I worked on improving epistemics in the EA community at CEA (as a contractor), as a research assistant at the Global Priorities Institute, on community building, and Global Health Policy.
Unless explicitly stated otherwise, opinions are my own, not my employer’s.
You can give me positive and negative feedback here.
Do you have a list of research questions that you think could easily be sped up with AI systems? I suspect that I’m more pessimistic than you are due to concerns around scheming AI agents doing intentional research sabotage and think that the affordances of AI agents might make some currently intractable agendas more tractable.
Thank you for replying—it’s great that someone within the industry shared their perspective!
I don’t really understand why that would make the US building DCs in allied countries destabilising. The short answer for why it might be stabilising is:
* It gives non-US actors more leverage, making deals where benefits are shared more likely.
* It’s harder for the US to defect on commitments to develop models safely and not misuse them if it’s easy for their allies to spy on them (or they have made commitments for DC use to be monitored)
* It keeps the Western democracies ahead of the CCP.
I think that allied countries themselves building DCs might be comparably stabilising—it gives more leverage to allied countries, at the cost of baking in less coordination and affordances to make deals around how AI is used and developed.
Some quick takes in a personal capacity:
I agree that a good documentary about AI risk could be very valuable. I’m excited about broad AI risk outreach and few others seem to be stepping up. The proposal seem ambitious and exciting.
I suspect that a misleading documentary would be mildly net-negative, and it’s easy to be misleading. So far, a significant fraction of public communications from the AI safety community has been fairly misleading (definitely not all—there is some great work out there as well).
In particular, equivocating between harms like deepfakes and GCRs seems pretty bad. I think it’s fine to mention non-catastrophic harms, but often, the benefits of AI systems seem likely to dwarf them. More cooperative (and, in my view, effective) discourse should try to mention the upsides and transparently point to the scale of different harms.
In the past, team members have worked on (or at least in the same organisation) comms efforts that seemed low integrity and fairly net-negative to me (e.g., some of their work on deepfakes, and adversarial mobile billboards around the UK AI Safety summit). Idk if these specific team members were involved in those efforts.
The team seems very agentic and more likely to succeed than most “field-building” AIS teams.
Their plan seems pretty good to me (though I am not an expert in the area). I’m pretty into people just trying things. Seems like there are too few similar efforts, and like we could regret not making more stuff like this happen, particularly if your timelines are short.
I’m a bit confused. Some donors should be very excited about this, and others should be much more on the fence or think it’s somewhat net-negative. Overall, I think it’s probably pretty promising.
Sorry, I agree this message is somewhat misleading—I’ll ask our ops team to review this.
Thanks. We should probably try to display this on our website properly. We have been able to fund for-profits in the past, but it is pretty difficult. I don’t think the only reason we passed on your application was that it’s for-profit, but that did make our bar much higher (this is a consequence of US/UK charity law and isn’t a reflection on the impact of non-profits/for-profits).
By the way, I personally think that your project should probably be a for-profit, as it will be easier to raise funding, users will hold you to higher standards, and your team seems quite value-aligned.
Some AI research projects that (afaik) haven’t had much work done on them and would be pretty interesting:
If the US were to co-build secure data centres in allied countries, would that be geopolitically stabilising or destabilising?
What AI safety research agendas could be massively sped up by AI agents? What properties do they have (e.g. easily checkable, engineering > conceptual …)?
What will the first non-AI R&D uses of powerful and general AI systems be?
Are there ways to leverage cheap (e.g. 100x lower than present-day cost) intelligence or manual labour to massively increase the US’s electricity supply?
What kinds of regulation might make it easier to navigate an intelligence explosion (e.g. establishing quick pathways to implement policy informed by AI experts, or establishing zones where compute facilities can be quickly built without navigating a load of red tape)?
Given that they’ve made a public Manifund application, it seems fine to share that there has been quite a lot of discussion about this project on the LTFF internally. I don’t think we are in a great place to share our impressions right now, but if Connor would like me to, I’d be happy to share some of my takes in a personal capacity.
I think the main reasons that EAs are working on AI stuff over bio stuff is that there aren’t many good routes into worst case bio work afaict largely due to infohazard concerns from field building, and the x-risk case for biorisk not being very compelling (maybe due to infohazard concerns around threat models).
I’m interested in figuring out “what skill profiles are most leveraged for altruistic work after we get the first competent AI agents”?
I think this might be one of the most important questions for field-building orgs to work on (in any cause area but particularly AI safety). I think 80k and other AIS-motivated careers groups should try to think hard about this question and what it means for their strategy going forward.
I’m optimistic about productivity on the question of “what skill profiles are most leveraged after we get the first competent AI-agents” because:
few people have actually tried to think about what managing an AI workforce would look like
this doesn’t feel conceptually abstract (e.g. some people have already adopted LMs in their day-to-day workflows)
something like drop-in replacements for human remote workers are more likely than AIs less analogous to human workers if timelines are short (and if timelines are short, this work is more urgent than if timelines are long)
One important consideration is whether human managers are going to be good proxies for AI-agent managers. It’s plausible to me that the majority of object-level AI safety work that has ever be done will be done by AI agents. It could be that the current status quo persists, where people’s leverage routes through things like leadership and management skills, or it could be that ~everyone will become more leveraged proportional to their access to AI models (roughly their budget) due to AI agents having different affordances than humans, or something else.
For example, suppose that AI agents are much more tolerant, hardworking, and understanding of user intent for short tasks than human workers. I expect that having good strategic takes would become much more leveraged and having good internal stakeholder management skills would become much less leveraged (relative to today). If 80k (for example) thought that was likely, then maybe they should double down on finding people with great strategy takes and care less about typical management skills.
Maybe “junior” roles will be automated before “senior” roles almost uniformly across the economy. In that case, it’s even more valuable than it is right now to focus on getting “senior” people.
Or maybe coding will be automated before everything else, and for some reason people’s relative leverage inside the company stays pretty fixed, or it’s too hard to say which kinds of jobs will be most leveraged at software companies. Then “coding” still becomes much more leveraged per dollar than today, and it’s probably useful to find more value-aligned coders.
It would be surprising to me if the price of certain kinds of intellectual labour decreased by 100x, and this had little impact on people’s relative leverage.
(disclaimer: I’m not sure whether the above stories actually go through, they are meant to just be illustrative of the kind of thinking that seems undersupplied)
I thought that David meant “intellectual areas” as opposed to physical/digital venues. Seems like lots of discussion in those venues isn’t particularly decision-relevant, and timelines aren’t really an unusually decision-irrelevant case.
I’m not directly working on an AI pause, but if I were, I would think that timelines were very strategically relevant:
How soon will we see warning shots, and will they arrive before society is disempowered?
What kinds of preparation should I do before we see a warning shot
Will progress be sufficiently slow such that we get frog-boiled?
How quickly should we aim to spend our limited human capital and financial resources?
How “in the water” will (or could) AI be over the next few years?
Discussion of AI timelines is helpful for answering many of the questions above, as are the endline forecasts. I suspect that I believe that working on AI pauses is much less robust to timelines than you think it is, and discussion of timelines is much more helpful for developing models of AI development and risk (and those models are highly decision-relevant).
In what places do timelines get a lot of attention despite not being very decision-relevant?
Thanks for the link, I haven’t come across that report before.
I think Yann has pretty atypical views for people working on LMs. For example, if you take the reference classes of AI-related Turing award winners or Chief scientist types at AI labs, most are far more bullish on LMs (e.g., Hinton, Bengio, Ilya, Jared Kaplan, Schulman).
Why should it matter whether new models have been released after the reveal of ARC-AGI-2? If models have to be specifically fine-tuned for these tasks, doesn’t that show they are lacking in the capability to generalize to novel problems?
The main reason is that the benchmark has been pretty adversarially selected, so it’s not clear that it’s pointing at a significant lack in LM capabilities. I agree that it’s weak evidence that they can’t generalise to novel problems, but basically all of the update is priced in from just interacting with systems and noticing that they are better in some domains than others.
For one, it tells you that current frontier models lack the general intelligence or “fluid intelligence” to solve simple puzzles that pretty much any person can solve. Why is that? Isn’t that interesting?
I disagree that ARC-AGI is strong evidence against LMs not having “fluid intelligence”—I agree that was the intention of the benchmark, and I think it’s weak evidence.
Another “benchmark” I mused about is the ability of AI systems to generate profit for their users by displacing human labour. It seems like improvement on that “benchmark” has been much, much slower than Moore’s law, but, then again, I don’t know if anyone’s been able to accurately measure that.
Has this been a lot slower than Moore’s law? I think OpenAI revenue is, on average, more aggressive than Moore’s law. I’d guess that LM ability to automate intellectual work is more aggressive than Moore’s law, too, but it started from a very low baseline, so it’s hard to see. Subjectively, LMs feel like they should be having a larger impact on the economy than they currently are. I think this is more related to horizon length than fluid intelligence, but 🤷♂️.
The bigger picture is that LLMs have extremely meagre capabilities in many cognitive domains and I haven’t seen signs of anything but modest improvement over the last ~2.5 years. I also don’t see many people trying to quantify those things.
I’m curious for examples here—particularly if they are the kinds of things that LMs have affordances for, are intellectual tasks, and are at least moderately economically valuable (so that someone has actually tried to solve).
In my view, there are many good reasons to work at an AI company, including:
* productively steering an AI lab during crunch time
* doing well-resourced AI safety research
* increasing the ability for safety-conscious people to blow the whistle to governments
* learning about the AI frontier from the best people in the field
* giving to effective charities
* influencing the views of other employees
* influencing how powerful AI systems are deployed and what they are used for during deployment
I don’t think these necessarily outweigh the costs of working at an AI company, but the altruistic benefits are sometimes large, and it seems good for people to consider the option thoughtfully.
My impression is that ARC-AGI (1) is close to being solved, which is why they brought our ARC-AGI-2 a few weeks ago.
Benchmarks are often adversarially selected so they take longer to be saturated, so I don’t think little progress on ARC-AGI-2 a few weeks after release (and iirc after any major model release) tells us much at all.
-Most AI experts are skeptical that scaling up LLMs could lead to AGI.
I don’t think this is true. Do you have a source? My guess is that I wouldn’t consider many of the people “experts”.
-It seems like there are deep, fundamental scientific discoveries and breakthroughs that would need to be made for building AGI to become possible. There is no evidence we’re on the cusp of those happening and it seems like they could easily take many decades.
I think this is a pretty strange take. It seems like basically all progress on AI has involved approximately 0 “deep, fundamental scientific discoveries”, so I think you need some argument for why the trend will change. Alternatively, if you think we have made lots of discoveries and that explains AI progress so far, then you need an argument for why these discoveries will stop. Or, if you think we have made little AI progress since ~2010 then I think most readers would strongly disagree with you.
-If you look at easy benchmarks like ARC-AGI and ARC-AGI-2 that are easy for humans to solve and intentionally designed to be a low bar for AI to clear, the weaknesses of frontier AI models are starkly revealed.
I don’t think they are designed to be a low bar to clear. They seem very adversarially selected, though I agree that LMs do poorly on them relative to subjectively more difficult tasks like coding. It seems pretty hard to make a timelines update from ARC-AGI unless you are very confident in the importance of abstract shape rotation problems for much more concrete problems, or you care about some notion of “intelligence” much more than automating intellectual labour.
I feel quite confused about the case where someone earns much less than their earning potential in another altruistically motivated but less impactful career doing work that uses a similar skillset (e.g. joining a think tank after working on policy at an AI company). This seems somewhere between A and B.
To be clear, I’m open to building broad coalitions and think that a good documentary could/would feature content on low-stakes risks; but, I believe people should be transparent about their motivations and avoid conflating non-GCR stuff with GCR stuff.