ML safety researcher, working on NN interpretability.
Adrià Garriga Alonso
Good insight, thank you for writing this post! I agree with it. Now that you point it out, I find striking how knowlege has compounded, even more impressively than money.
I would like to add another contestant: influence, within or out of mainstream institutions. As a movement, social capital and influence on other people (especially politicians) could prove very useful to be able to have a large impact when the time is right. I’m thinking especially of the Mont Pelerin society: how they spread in economics academia by convincing people and placing people in positions of (mostly academic) influence; and how they eventually became orthodox economic policy.
The EA community also seems to be very aware of the MPS. What I’m pointing out is that, under your framework, community building is also an intervention for patient longtermism.
I think even among such selected crowd, Anita would stand out like a bright star. The average top-university PhD student doesn’t end up holding a top faculty job. (This may seem elitist, but it is important: becoming a trainer of mediocre PhD students is likely not more effective than non-profit work). A first-author Nature paper in undergrad (!) is quite rare too.
Thank you for the write-up. I wish I had this advice, and (more crucially) kept reminding myself of it, during my PhD. As you say, academic incentives did poison my brain, and I forgot about my original reasons for entering the programme. I only realised one month ago that it had been happening slowly; my brain is likely still poisoned, but I’m working on it.
I’m curious about your theory of change, if you have time to briefly write about it. You wrote that
addressing these risks goes substantially through EAs taking on a lot more object level work— founding organizations, engineering systems, making scientific progress— than I expect is the median view
and that you don’t think gunning for a faculty position is a good thing. What kind of job is the right one to “make scientific progress”, then? I thought that the best way to do that is to run a lab, managing a bunch of smart PhD students and postdocs, and steering them towards useful research directions.
My impression is that PIs manage the same or more people than the equivalent seniority position in industry, at least in machine learning; but that they have freedom to set research priorities, instead of having to follow a boss. (On the flipside, they have to pander to grant givers, but that seems to give more freedom in research direction).
In summary, what do you think is the kind of job where you can make the most scientific progress?
I don’t see how this is a counterargument. Do you mean to say that, once you are on track to tenure, you can already start doing the high-impact research?
It seems to me that, if this research is too diverged from the academic incentives, then our hypothetical subject may become one of these rare cases of CS tenure-track faculty that does not get tenure.
You can get research taste by doing research at all, it doesn’t have to be a PhD. You may argue that PIs have very good research taste that you can learn from. But their taste is geared towards satisfying academic incentives! It might not be good taste for what you care about. As Chris Olah points out, “Your taste is likely very influenced by your research cluster”.
So, if I understand correctly, the central claim is that: if naturalism is true and we make a “Scientist AI” whose initial goal is to gain knowledge and which can change its goals, then the AI will be aligned. Is that accurate?
I think this is dangerously wrong. Even if the AI comes to gain perfect knowledge of morality for humans (either because naturalism is true, or because it reads about it on human-written books), there is no guarantee that it will then try to act as it is moral. Why does the orthogonality thesis not apply? Why would the AI not disregard morality and act in its self-interest, as many humans actually do?
(EDIT: from further reading, it seems that moral realism does reject the orthogonality thesis. To this I say: what about psychopaths?)
It is extremely implausible that an AI that can discover moral facts will be aligned by default, given the existence of so many humans that are simply not. That is still, assuming that moral realism (which I’m assuming is similar to naturalism) is true.
My impression is that it might be easy to miss some amino acid types if you’re not careful (e.g. tryptophan is almost exclusively found in meat/dairy and is the only way your body can make serotonin
I am pretty confident that this particular impression is incorrect. The essential amino-acid profiles of the protein of most plant sources is very close to human requirements. See in particular Figure 14 of the WHO report on amino-acid requirements. (https://apps.who.int/iris/bitstream/handle/10665/43411/WHO_TRS_935_eng.pdf?sequence=1&isAllowed=y, page 165 of the PDF). It compares the human percentage amino-acid requirements with the content of various animal and vegetal sources. They are incredibly similar, and also the percentage of tryptophan required is larger than the human pattern in all plant sources (except perhaps maize if we scale down the bars).
That said, thank you for the post! I am now 70% confident that I am in fact stressed; but I don’t see a way to stop it, the work just keeps on piling up.
What does the WANBAM acronym (assuming it is one) stand for? Presumably Women And Non Binary… Altruism Movement?
(apologies if the question is irrelevant but I’m very curious, and I couldn’t find this in the post or the website)
It is unclear to me if this is a good idea. Sci-hub is great, but whoever does this would face a good amount of legal risk. If EA organisations (eg American ones) are known to be funding this, they face the risk of lawsuits and reputational damage.
I think at least this post should not be publicized too widely. Maybe nobody else commented on this post for precisely this reason?
[for policy makers]
It is a mistake to assume that AI researchers are driven by the positive consequences of their work. Geoffrey Hinton, winner of a Turing Award for his enormous contribution to deep neural networks, is not optimistic about the effects of advanced AI, or whether humans can decide what it does. In a 2015 meeting of the Royal Society, he stated that “there is not a good track record of less intelligent things controlling things of greater intelligence”, and that “political systems will use [AI] to terrorize people”. Nevertheless, he presses on with his research, because “the prospect of discovery is too sweet”.
(source for the quotes: https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom )
[for any audience]
The chief asset of the human species is our intelligence: with it, we have settled all over and transformed the world. Most machine learning researchers expect AI to surpass human intelligence in all areas within a lifetime (source ). When that happens, humanity will find ourselves in the same place as chimpanzees: with our fate at the mercy of the most intelligent species. As deep learning Geoffrey Hinton noted, “there is not a good track record of less intelligent things controlling things of greater intelligence”.
In 1951, Alan Turing argued that at some point computers would probably exceed the intellectual capacity of their inventors, and that “therefore we should have to expect the machines to take control.” Whether that is a good or a bad thing depends on whether the machines are benevolent towards us or not. (Partial source: https://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom )
Literally everyone knows he was the Masculine Mongoose. Superheros don’t even try to hide their identity any more.
thank machine doggo
Yes.
No, getting rid of factory farming (“fiat iustitia”) won’t increase X-risk (“pereat mundus”).
Or are you implying that resources are in competition for the two? (Perhaps weakly true)
Very cool, I didn’t actually believe that other regulatory regimes emulated the EU, but I believe it a little bit now. The large number of GDPR emulations surprised me.
One thing I don’t quite get
This complicated architecture has also had a 5.2% growth rate in all its bodies combined, with most of the staff being highly educated (usually possessing a master’s degree).
This is a growth in the number of staff?
Both of these factors resulted in a signal of competence to other countries in the world, which results in a higher degree of trust in the EU’s decisions and policies.
Why does having more staff signal competence? Is it because more budget implies the agencies are being taken seriously?
No, I think your table is substantially better than chatgpt’s because it factors out the two alignment dimensions into two spatial dimensions.
Great thanks, I’ve set up a recurring donation!
EDIT: apparently they’re very time-constrained, so I’ll give $13.3k as a lump sum instead.
Work trials (paid, obviously) are awesome for better hiring, especially if you’re seeking to get good candidates that don’t fulfill the traditional criteria (e.g. coming from an elite US/UK university). Many job seekers don’t have a current employment.
Living with other EAs or your coworkers is mostly fine too, especially if you’re in a normal living situation, like most EA group houses are.
These suggestions aren’t great. I agree with the “Don’t date” ones, but these were already argued for before.
That’s one way to see it, but I thought that ideally you’re supposed to keep considering all the possible “interventions” you can personally do to help moral patients. That is, if the most effective cause that matches your skills (and is neglected, etc etc) changes, you’re supposed to switch.
In practice that does not happen much, because skills and experience in one area are most useful in the same area, and because re-thinking your career constantly is tiring and even depressing; but it could be that way.
If it was that way, people who have decided on their cause area (for the next say, 5 years) should still call themselves EAs.