Software engineer interested in AI safety.
Stephen McAleese
Karma: 378
I think microgrants are a great idea! Because they’re small, you can make lots of investments to different people with relatively little risk and cost.
One way of doing automated AI safety research is for AI safety researchers to create AI safety ideas on aisafetyideas.com and then use the titles as prompts for a language model. Here is GPT-3 generating a response to one of the ideas:
This question would have been way easier if just I estimated the number of AI safety researchers in my city (1?) instead of the whole world.
Here is a model that involves taking thousands of trials of the product of six variables randomly set between 10% and 90% (e.g. 0.5^6 = 0.015 = 1.5%).
As other people have noted, conjunctive models tend to produce low probabilities (<5%).
Great post. This is possibly the best explanation of the relationship between capabilities and safety I’ve seen so far.
The whole section on Price’s Law has been replaced with a section on Lotka’s Law.
Great talk. I think it breaks down the problem of AI alignment well. It also reminds me of the more recent breakdown by Dan Hendryks which decomposes ML safety into three problems: robustness, monitoring and alignment.
I’ve noticed that a lot of good ideas seem to come from talks. For example, Richard Hamming’s famous talk on working on important problems. Maybe there should be more of them.
Thanks, I think you’re right. I’ll have to edit that section.
I went through all the authors from the Alignment Forum from the past ~6 months, manually researched each person and came up with a new estimate named ‘Other’ of about 80 people which includes independent researchers, other people in academia and people in programs such as SERI MATS.
More edits:
- DeepMind: 5 → 10.
- OpenAI: 5 → 10.
- Moved GoodAI from the non-technical to technical table.
- Added technical research organization: Algorithmic Alignment Group (MIT): 4-7.
- Merged ‘other’ and ‘independent researchers’ into one group named ‘other’ with new manually created (accurate) estimate.
Great point. The decline of religion has arguably left a cultural vacuum that new organizations can fill.
Edit: updated OpenAI from 5 to 10.
From their website, AI Impacts currently has 2 researchers and 2 support staff (the current total estimate is 3).
The current estimate for Epoch is 4 which is similar to most estimates here.
I’m trying to come up with a more accurate estimate for independent researchers and ‘Other’ researchers.
New estimates:
CSER: 2-5-10 → 2-3-7
FLI: 5-5-20 → 3-4-6
Levelhume: 2-5-15 → 3-4-10
Edit: added Rethink Priorities to the list of non-technical organizations.
I re-estimated the number of researchers in these organizations and the edits are shown in the ‘EDITS’ comment below.
Copied from the EDITS comment:
- CSER: 5-5-10 → 2-5-15
- FLI: 5-5-20 → 3-5-15
- Levelhume Centre: 5-10-70 (Low confidence) → 2-5-15 (Medium confidence)My counts for CSER:
- full-time researchers: 3
- research affiliates: 4FLI: counted 5 people working on AI policy and governance.
Levelhume Centre:
- 7 senior research fellows
- 14 research fellowsMany of them work at other organizations. I think 5 is a good conservative estimate.
New footnote for the ‘Other’ row in the non-technical list of researchers (estimate is 10):
“There are about 45 research profile on Google Scholar with the ‘AI governance’ tag. I counted about 8 researchers who weren’t at the other organizations listed.”
I re-estimated counts for many of the non-technical organizations and here are my conclusions:
I didn’t change the CSET estimate (10) because there seems to be a core group of about 5 researchers there and many others (20-30). Their productivity also seems to be high: I counted over 20 publications so far this year though it seems like only about half of them are related to AI governance (list of publications).
I deleted BERI and SERI from the list because they don’t seem to have any full-time researchers.
Epoch: decreased estimate from 10 to 4.
Good AI seems to be more technical than non-technical (todo).
Edits based on feedback from LessWrong and the EA Forum:
EDITS:
- Added new ‘Definitions’ section to introduction to explain definitions such as ‘AI safety’, ‘researcher’ and the difference between technical and non-technical research.UPDATED ESTIMATES (lower bound, estimate, upper bound):
TECHNICAL
- CHAI: 10-30-60 → 5-25-50
- FHI: 10-10-40 → 5-10-30
- MIRI: 10-15-30 → 5-10-20NON-TECHNICAL
- CSER: 5-5-10 → 2-5-15
- Delete BERI from the list of non-technical research organizations
- Delete SERI from the list of non-technical research organizations
- Levelhume Centre: 5-10-70 (Low confidence) → 2-5-15 (Medium confidence)
- FLI: 5-5-20 → 3-5-15
- Add OpenPhil: 2-5-15
- Epoch: 5-10-15 → 2-4-10
- Add ‘Other’: 5-10-50
Thanks for the information! Your estimate seems more accurate than mine.
In the case of Epoch, I would count every part-time employee as roughly half a full-time employee to avoid underestimating their productivity.
It’s true that the universe B might never fully catch up because 99% of a single generation was lost. But over 1 billion years, we would expect about 40 million generations to live. Even if a few generations were lost, if there is a recovery the total loss won’t be high.
This reminds of Nick Bostrom’s story, “The Fable of the Dragon-Tyrant”. Maybe somebody will write a story like this about ageing instead of smallpox in the future.