Software engineer interested in AI safety.
Stephen McAleese
Karma: 378
Edits based on feedback from LessWrong and the EA Forum:
EDITS:
- Added new ‘Definitions’ section to introduction to explain definitions such as ‘AI safety’, ‘researcher’ and the difference between technical and non-technical research.UPDATED ESTIMATES (lower bound, estimate, upper bound):
TECHNICAL
- CHAI: 10-30-60 → 5-25-50
- FHI: 10-10-40 → 5-10-30
- MIRI: 10-15-30 → 5-10-20NON-TECHNICAL
- CSER: 5-5-10 → 2-5-15
- Delete BERI from the list of non-technical research organizations
- Delete SERI from the list of non-technical research organizations
- Levelhume Centre: 5-10-70 (Low confidence) → 2-5-15 (Medium confidence)
- FLI: 5-5-20 → 3-5-15
- Add OpenPhil: 2-5-15
- Epoch: 5-10-15 → 2-4-10
- Add ‘Other’: 5-10-50
Thanks for the information! Your estimate seems more accurate than mine.
In the case of Epoch, I would count every part-time employee as roughly half a full-time employee to avoid underestimating their productivity.
Estimating the Current and Future Number of AI Safety Researchers
It’s true that the universe B might never fully catch up because 99% of a single generation was lost. But over 1 billion years, we would expect about 40 million generations to live. Even if a few generations were lost, if there is a recovery the total loss won’t be high.
“The catastrophe that takes place in scenario B removes 99% of all humans alive, which in turn removes around 99% of all humans that could have lived at the end of time.”
That would only happen if the population never recovered. But since I would expect the world to rapidly repopulate, I therefore would expect the long-term difference to be insignificant.
I agree that the total number of humans who will ever live at the end of time is similar in A and B. Therefore I think there is almost no difference between A and B in the long term.
I think the argument for longtermism is pretty straightforward: if we have a long future then most people who will ever exist will live in the future. If we value all people across all times equally, then we should care far more about the future than the present.
Also, what do you mean by ‘knowledge explosion’?
Interesting post. I like how it introduced the idea that kindness can improve relationships which seems important and beneficial. It’s great that people perform acts of kindness to improve their relationships.
“These findings are consistent with the theory that the function of altruism is to foster cooperative relationships with others.”
However, I think there are reasons for being altruistic other than improving relationships. For example, charities such as the Against Malaria Foundation are saving lives.
Also, if we only give money to people we know, we won’t be benefitting the people in the world who need resources such as money the most. People we don’t know in poor countries are often the most in need and are probably too poor to reciprocate.
How Do AI Timelines Affect Existential Risk?
I see some problems with the claims made in the section named “Human potential starts in current lives”.
“Therefore, losing 99% of humanity today means losing 99% of the total human potential, which would be roughly 99 times worse than losing the remaining 1%.”
Similarly, it seems like you’re making the claim that a 100% loss of all lives is only slightly worse than 99% of lives because each 1% of people today contributes 1% to the final population of humanity.
But I think this claim rests on the assumption that 99% of humans dying would reduce the final population by 99%.
You mentioned that if 99% of humans died, the remaining 1% could repopulate the world by having a higher birth rate but then went on to say that this possibility didn’t affect your point much.
But I think it would have a huge effect. If humanity lasts 1 billion years and 99% of humans died at some point, even if it took 1000 years to repopulate the earth, that would only be 1/1000 of all of history and the population wouldn’t change much in the long term. Although the death of 99% of the population, might affect the genes of future people, I think the effect on the population size would be negligible. Therefore, I think the assumption is false.
If the assumption were correct, 100% of humanity dying would only be slightly worse than 99% dying. But since the 1% would probably rapidly repopulate the world, 99% dying would probably have a negligible impact on the total long-term population. Meanwhile, if 100% died the entire future population would be lost. Therefore 100% is far worse than 99%.
My understanding is that the people in the repugnant conclusion have lives that are slightly net positive which means those lives could have suffering as long as the good slightly outweighs the suffering. For example, a life could have slightly net positive value if it had 11 positive experiences for every 10 negative experiences.
It seems like there is a quality and quantity trade-off where you could grow EA faster by expecting less engagement or commitment. I think there’s a lot of value in thinking about how to make EA massively scale. For example, if we wanted to grow EA to millions of people maybe we could lower the barrier to entry somehow by having a small number of core ideas or advertising low-commitment actions such as earning to give. I think scaling up the number of people massively would benefit the most scalable charities such as GiveDirectly.
The counterargument is that impact per person tends to be long-tailed. For example, the net worth of Sam Bankman Fried is ~100,000 higher than a typical person. Therefore, who is in EA might matter as much or more as how many EAs there are.
My guess is that quality matters more in AI safety because I think the talent necessary to have a big positive impact is high. AI safety impact also seems long-tailed.
It’s not clear to me whether quality or quantity is more important because some of the benefits are hard to quantify. One easily measurable metric is donations: adding a sufficiently large number of average donators should have the same financial value as adding a single billionaire.
I really like this post and it reminds of Peter Singer’s thought experiment of getting your shoes wet to save a child from drowning in a shallow pond.
The idea is that altruism is right when the gain to someone else is large and the personal sacrifice is relatively small.
So although small sacrifices for the sake of benefiting other people are often worth doing, large sacrifices that significantly decrease your wellbeing probably aren’t.
Great post. Although there seems to be quite a lot of interest in AI safety in the EA community, the number of technical AI safety researchers is still surprisingly small. I spent some time estimating the number of researchers and I think the 100-200 number sounds right to me as an order of magnitude estimate.
Given AI safety technical research is so important and neglected, it seems like a high-impact field to get into. The field is still relatively immature and therefore there could be many low-hanging insights to discover.
Thanks for the summary. I think the dath ilan world is a really interesting concept and useful for thinking about how to maximize the chance of the alignment problem being solved.
Maybe someone will write an essay like this about aging in a few decades when we find a solution for it.
Great post. I think it explains the important concept of expected value in a simple way.
I re-estimated counts for many of the non-technical organizations and here are my conclusions:
I didn’t change the CSET estimate (10) because there seems to be a core group of about 5 researchers there and many others (20-30). Their productivity also seems to be high: I counted over 20 publications so far this year though it seems like only about half of them are related to AI governance (list of publications).
I deleted BERI and SERI from the list because they don’t seem to have any full-time researchers.
Epoch: decreased estimate from 10 to 4.
Good AI seems to be more technical than non-technical (todo).