Possible claim 1: “We want people in academia to be doing lots of good AGI-x-risk-mitigating research.” Yes, I don’t think this is controversial.
Possible claim 2: “We should stop giving independent researchers and nonprofits money to do AGI-x-risk-mitigating research, because academia is better.” You didn’t exactly say this, but sorta imply it. I disagree. Academia has strengths and weaknesses, and certain types of projects and people might or might not be suited to academia, and I think we shouldn’t make a priori blanket declarations about academia being appropriate for everything versus nothing. My wish-list of AGI safety research projects (blog post is forthcoming UPDATE: here it is) has a bunch of items that are clearly well-suited to academia and a bunch of others that are equally clearly a terrible fit to academia. Likewise, some people who might work on AGI safety are in a great position to do so within academia (e.g. because they’re already faculty) and some are in a terrible position to do so within academia (e.g. because they lack relevant credentials). Let’s just have everyone do what makes sense for them!
Possible claim 3: “We should do field-building to make good AGI-x-risk-mitigating research more common, and better, within academia.” The goal seems uncontroversially good. Whether any specific plan will accomplish that goal is a different question. For example, a proposal to fund a particular project led by such-and-such professor (say, Jacob Steinhardt or Stuart Russell) is very different from a proposal to endow a university professorship in AGI safety. In the latter case, I would suspect that universities will happily take the money and spend it on whatever their professors would have been doing anyway, and they’ll just shoehorn the words “AGI safety” into the relevant press releases and CVs. Whereas in the former case, it’s just another project, and we can evaluate it on its merits, including comparing it to possible projects done outside academia.
Possible claim 4: “We should turn AGI safety into a paradigmatic field with well-defined widely-accepted research problems and approaches which contribute meaningfully towards x-risk reduction, and also would be legible to journals, NSF grant applications, etc.” Yes that would be great (and is already true to a certain extent), but you can’t just wish that into existence! Nobody wants the field to be preparadigmatic! It’s preparadigmatic not by choice, but rather to the extent that we are still searching for the right paradigms.
> the EA/LW community has a comparative advantage at stating the right problem to solve and grantmaking, the academic community has a comparative advantage at solving sufficiently defined problems
I think this is fairly uncontroversial, and roughly right—I will probably be thinking in these terms more often in the future.
Implications are that the most important output the community can hope to produce is research agendas, benchmarks, idealized solutions and problem statements, and leave ML research, practical solutions and the actual task of building an aligned AGI to academics.
(the picture gets more complicated because experiments are a good way of bashing our heads against the problem and gaining intuitions useful for making the problems sharper)
In slogan form: the role of the community should be to create the ImageNet Challenge of AI Alignment, and plan to leave the task of building the AlexNet of AI Alignment to academics.
It’s not obvious to me that “the academic community has a comparative advantage at solving sufficiently defined problems”. For example, mechanistic interpretability has been a well-defined problem for the past two years at least, but it seems that a disproportionate amount of progress on it has been made outside of academia, by Chris Olah & collaborators at OpenAI & Anthropic. There are various concrete problems here but it seems that more progress is being made by independent researchers (e.g. Vanessa Kosoy, John Wentworth) and researchers at nonprofits (MIRI) than by anyone in academia. In other domains, I tend to think of big challenging technical projects as being done more often by the private or public sector—for example, academic groups are not building rocket ships, or ultra-precise telescope mirrors, etc., instead companies and governments are. Yet another example: In the domain of AI capabilities research, DeepMind and OpenAI and FAIR and Microsoft Research etc. give academic labs a run for their money in solving concrete problems. Also, quasi-independent-researcher Jeremy Howard beat a bunch of ML benchmarks while arguably kicking off the pre-trained-language-model revolution here.
My perspective is: academia has a bunch of (1) talent and (2) resources. I think it’s worth trying to coax that talent and resources towards solving important problems like AI alignment, instead of the various less-important and less-time-sensitive things that they do.
However, I think it’s MUCH less clear that any particular Person X would be more productive as a grad student than as a nonprofit employee, or more productive as a professor than as a nonprofit technical co-founder. In fact, I strongly expect the reverse.
And in that case, we should really be framing it as “There are tons of talented people in academia, and we should be trying to convince them that AGI x-risk is a problem they should work on. And likewise, there are tons of resources in academia, and we should be trying to direct those resources towards AGI x-risk research.” Note the difference: in this framing, we’re not pre-supposing that pushing people and projects from outside academia to inside academia is a good thing. It might or might not be, depending on the details.
Fair points. In particular, I think my response should have focused more on the role of academia + industry.
a disproportionate amount of progress on [mechanistic interpretability] has been made outside of academia, by Chris Olah & collaborators at OpenAI & Anthropic
Not entirely fair: if you open the field just a bit to “interpretability” in general you will see that most important advances in the field (eg SHAP and LIME) were done inside academia.
I would also not be too surprised to find people within academia who are doing great mechanistic interpretability, simply because of the sheer number of people researching interpretability.
There are various concrete problems here but it seems that more progress is being made by independent researchers (e.g. Vanessa Kosoy, John Wentworth) and researchers at nonprofits (MIRI) than by anyone in academia.
Granted, but part of the problem is convincing academics to engage. I think that the math community would be x100 times better at solving these problems if they ever become popular enough in academia.
However, I think it’s MUCH less clear that any particular Person X would be more productive as a grad student than as a nonprofit employee, or more productive as a professor than as a nonprofit technical co-founder. In fact, I strongly expect the reverse.
Matches my intuition as well (though I might be biased here). I’d add that I expect grad students will get better mentorship on average in academia than in non profits / doing independent research (but mostly work on irrelevant problems while in academia).
One important intuition that I have is that I think academia + industry scales to “crappy but at the end advancing in the object level” despite having lots of mediocre people involved, while I think that all cool things happening in EA+LW are due to some exceptionally talented people, and if we tried to scale them up we would end with “horrible potcrackery”.
If you are saying academia has a good track record, then I must say (1) wrong for stuff like ML, where in recent years much (arguably most) relevant progress is made outside of academia, and (2) it may have a good track record for the long history of science, and when you say it’s good at solving problems, sure I think it might solve alignment in 100 years, but we need it in 10, and academia is slow. (E.g. read Yudkowsky’s sequence on science, if you don’t think that academia is slow.)
Do you have some reason why you think that a person can make more progress in academia than elsewhere? I agree that academia has people, and it’s good to get those people, but academia has badly shaped incentives, like (from my other comment): “Academia doesn’t have good incentives to make that kind of important progress: You are supposed to publish papers, so you (1) focus on what you can do with current ML systems, instead of focusing on more uncertain longer-term work, and (2) goodhart on some subproblems that don’t take that long to solve, instead of actually focusing on understanding the core difficulties and how one might address them.” So I expect a person can make more progress outside of academia. Much more, in fact.
Some important parts of the AI safety problem seem to me like they don’t fit well into academia work. There are of course exceptions, people in academia who can make useful progress here, but they are rare. I am not that confident in this, as my understanding of AI safety isn’t that deep, but I’m not just making this up. (EDIT: This mostly overlaps with the first two points I made, that academia is slow and that there are bad incentives, and maybe some other minor considerations about why excellent people (e.g. John Wentworth) may rather choose to not work in academia. What I’m saying is that I think that AI safety is a problem where those obstacles are big obstacles, whereas there might be other fields where those obstacles aren’t thaaat bad.)
Possible claim 2: “We should stop giving independent researchers and nonprofits money to do AGI-x-risk-mitigating research, because academia is better.” You didn’t exactly say this, but sorta imply it. I disagree.
I don’t agree with possible claim 2. I just say that we should promote academic careers more than independent researching, not that we should stop giving them money. I don’t think money is the issue.
OK, thanks for clarifying. So my proposal would be: if a person wants to do / found / fund an AGI-x-risk-mitigating research project, they should consider their background, their situation, the specific nature of the research project, etc., and decide on a case-by-case basis whether the best home for that research project is academia (e.g. CHAI) versus industry (e.g. DeepMind, Anthropic) versus nonprofits (e.g. MIRI) versus independent research. And a priori, it could be any of those. Do you agree with that?
There are a few possible claims mixed up here:
Possible claim 1: “We want people in academia to be doing lots of good AGI-x-risk-mitigating research.” Yes, I don’t think this is controversial.
Possible claim 2: “We should stop giving independent researchers and nonprofits money to do AGI-x-risk-mitigating research, because academia is better.” You didn’t exactly say this, but sorta imply it. I disagree. Academia has strengths and weaknesses, and certain types of projects and people might or might not be suited to academia, and I think we shouldn’t make a priori blanket declarations about academia being appropriate for everything versus nothing. My wish-list of AGI safety research projects (blog post is forthcoming UPDATE: here it is) has a bunch of items that are clearly well-suited to academia and a bunch of others that are equally clearly a terrible fit to academia. Likewise, some people who might work on AGI safety are in a great position to do so within academia (e.g. because they’re already faculty) and some are in a terrible position to do so within academia (e.g. because they lack relevant credentials). Let’s just have everyone do what makes sense for them!
Possible claim 3: “We should do field-building to make good AGI-x-risk-mitigating research more common, and better, within academia.” The goal seems uncontroversially good. Whether any specific plan will accomplish that goal is a different question. For example, a proposal to fund a particular project led by such-and-such professor (say, Jacob Steinhardt or Stuart Russell) is very different from a proposal to endow a university professorship in AGI safety. In the latter case, I would suspect that universities will happily take the money and spend it on whatever their professors would have been doing anyway, and they’ll just shoehorn the words “AGI safety” into the relevant press releases and CVs. Whereas in the former case, it’s just another project, and we can evaluate it on its merits, including comparing it to possible projects done outside academia.
Possible claim 4: “We should turn AGI safety into a paradigmatic field with well-defined widely-accepted research problems and approaches which contribute meaningfully towards x-risk reduction, and also would be legible to journals, NSF grant applications, etc.” Yes that would be great (and is already true to a certain extent), but you can’t just wish that into existence! Nobody wants the field to be preparadigmatic! It’s preparadigmatic not by choice, but rather to the extent that we are still searching for the right paradigms.
(COI note.)
Here my personal interpretation of the post:
> the EA/LW community has a comparative advantage at stating the right problem to solve and grantmaking, the academic community has a comparative advantage at solving sufficiently defined problems
I think this is fairly uncontroversial, and roughly right—I will probably be thinking in these terms more often in the future.
Implications are that the most important output the community can hope to produce is research agendas, benchmarks, idealized solutions and problem statements, and leave ML research, practical solutions and the actual task of building an aligned AGI to academics.
(the picture gets more complicated because experiments are a good way of bashing our heads against the problem and gaining intuitions useful for making the problems sharper)
In slogan form: the role of the community should be to create the ImageNet Challenge of AI Alignment, and plan to leave the task of building the AlexNet of AI Alignment to academics.
It’s not obvious to me that “the academic community has a comparative advantage at solving sufficiently defined problems”. For example, mechanistic interpretability has been a well-defined problem for the past two years at least, but it seems that a disproportionate amount of progress on it has been made outside of academia, by Chris Olah & collaborators at OpenAI & Anthropic. There are various concrete problems here but it seems that more progress is being made by independent researchers (e.g. Vanessa Kosoy, John Wentworth) and researchers at nonprofits (MIRI) than by anyone in academia. In other domains, I tend to think of big challenging technical projects as being done more often by the private or public sector—for example, academic groups are not building rocket ships, or ultra-precise telescope mirrors, etc., instead companies and governments are. Yet another example: In the domain of AI capabilities research, DeepMind and OpenAI and FAIR and Microsoft Research etc. give academic labs a run for their money in solving concrete problems. Also, quasi-independent-researcher Jeremy Howard beat a bunch of ML benchmarks while arguably kicking off the pre-trained-language-model revolution here.
My perspective is: academia has a bunch of (1) talent and (2) resources. I think it’s worth trying to coax that talent and resources towards solving important problems like AI alignment, instead of the various less-important and less-time-sensitive things that they do.
However, I think it’s MUCH less clear that any particular Person X would be more productive as a grad student than as a nonprofit employee, or more productive as a professor than as a nonprofit technical co-founder. In fact, I strongly expect the reverse.
And in that case, we should really be framing it as “There are tons of talented people in academia, and we should be trying to convince them that AGI x-risk is a problem they should work on. And likewise, there are tons of resources in academia, and we should be trying to direct those resources towards AGI x-risk research.” Note the difference: in this framing, we’re not pre-supposing that pushing people and projects from outside academia to inside academia is a good thing. It might or might not be, depending on the details.
Fair points. In particular, I think my response should have focused more on the role of academia + industry.
Not entirely fair: if you open the field just a bit to “interpretability” in general you will see that most important advances in the field (eg SHAP and LIME) were done inside academia.
I would also not be too surprised to find people within academia who are doing great mechanistic interpretability, simply because of the sheer number of people researching interpretability.
Granted, but part of the problem is convincing academics to engage. I think that the math community would be x100 times better at solving these problems if they ever become popular enough in academia.
Matches my intuition as well (though I might be biased here). I’d add that I expect grad students will get better mentorship on average in academia than in non profits / doing independent research (but mostly work on irrelevant problems while in academia).
One important intuition that I have is that I think academia + industry scales to “crappy but at the end advancing in the object level” despite having lots of mediocre people involved, while I think that all cool things happening in EA+LW are due to some exceptionally talented people, and if we tried to scale them up we would end with “horrible potcrackery”.
But I’d be delighted to be proven wrong!
I must say I strongly agree with Steven.
If you are saying academia has a good track record, then I must say (1) wrong for stuff like ML, where in recent years much (arguably most) relevant progress is made outside of academia, and (2) it may have a good track record for the long history of science, and when you say it’s good at solving problems, sure I think it might solve alignment in 100 years, but we need it in 10, and academia is slow. (E.g. read Yudkowsky’s sequence on science, if you don’t think that academia is slow.)
Do you have some reason why you think that a person can make more progress in academia than elsewhere? I agree that academia has people, and it’s good to get those people, but academia has badly shaped incentives, like (from my other comment): “Academia doesn’t have good incentives to make that kind of important progress: You are supposed to publish papers, so you (1) focus on what you can do with current ML systems, instead of focusing on more uncertain longer-term work, and (2) goodhart on some subproblems that don’t take that long to solve, instead of actually focusing on understanding the core difficulties and how one might address them.” So I expect a person can make more progress outside of academia. Much more, in fact.
Some important parts of the AI safety problem seem to me like they don’t fit well into academia work. There are of course exceptions, people in academia who can make useful progress here, but they are rare. I am not that confident in this, as my understanding of AI safety isn’t that deep, but I’m not just making this up. (EDIT: This mostly overlaps with the first two points I made, that academia is slow and that there are bad incentives, and maybe some other minor considerations about why excellent people (e.g. John Wentworth) may rather choose to not work in academia. What I’m saying is that I think that AI safety is a problem where those obstacles are big obstacles, whereas there might be other fields where those obstacles aren’t thaaat bad.)
Hi Steven,
I don’t agree with possible claim 2. I just say that we should promote academic careers more than independent researching, not that we should stop giving them money. I don’t think money is the issue.
Thanks
OK, thanks for clarifying. So my proposal would be: if a person wants to do / found / fund an AGI-x-risk-mitigating research project, they should consider their background, their situation, the specific nature of the research project, etc., and decide on a case-by-case basis whether the best home for that research project is academia (e.g. CHAI) versus industry (e.g. DeepMind, Anthropic) versus nonprofits (e.g. MIRI) versus independent research. And a priori, it could be any of those. Do you agree with that?
Yes, I do indeed :)
You can frame it if you want as: founders should aim to expand the range of academic opportunities, and engage more with academics.