Software engineer interested in AI safety.
Stephen McAleese
Karma: 379
Thanks! I used that format because it was easy for me to write. I’m glad to see that it improves the reading experience too.
AGI as a Black Swan Event
I really like this post and I think it’s now my favorite post so far on the recent collapse of FTX.
Many recent posts on this subject have focused on topics such as Sam Bankman Fried’s character, what happened at FTX and how it reflects on EA as a whole.
While these are interesting subjects, I got the impression that a lot of the posts were too backward-looking and not constructive enough.
I was looking for a post that was more reflective and less sensational and focused on what we can learn from the experience and how we can adjust the strategy of EA going forward and I think this post meets these criteria better than most of the previous posts.
This reminds of Nick Bostrom’s story, “The Fable of the Dragon-Tyrant”. Maybe somebody will write a story like this about ageing instead of smallpox in the future.
I think microgrants are a great idea! Because they’re small, you can make lots of investments to different people with relatively little risk and cost.
One way of doing automated AI safety research is for AI safety researchers to create AI safety ideas on aisafetyideas.com and then use the titles as prompts for a language model. Here is GPT-3 generating a response to one of the ideas:
This question would have been way easier if just I estimated the number of AI safety researchers in my city (1?) instead of the whole world.
Here is a model that involves taking thousands of trials of the product of six variables randomly set between 10% and 90% (e.g. 0.5^6 = 0.015 = 1.5%).
As other people have noted, conjunctive models tend to produce low probabilities (<5%).
Great post. This is possibly the best explanation of the relationship between capabilities and safety I’ve seen so far.
The whole section on Price’s Law has been replaced with a section on Lotka’s Law.
Great talk. I think it breaks down the problem of AI alignment well. It also reminds me of the more recent breakdown by Dan Hendryks which decomposes ML safety into three problems: robustness, monitoring and alignment.
I’ve noticed that a lot of good ideas seem to come from talks. For example, Richard Hamming’s famous talk on working on important problems. Maybe there should be more of them.
Thanks, I think you’re right. I’ll have to edit that section.
I went through all the authors from the Alignment Forum from the past ~6 months, manually researched each person and came up with a new estimate named ‘Other’ of about 80 people which includes independent researchers, other people in academia and people in programs such as SERI MATS.
More edits:
- DeepMind: 5 → 10.
- OpenAI: 5 → 10.
- Moved GoodAI from the non-technical to technical table.
- Added technical research organization: Algorithmic Alignment Group (MIT): 4-7.
- Merged ‘other’ and ‘independent researchers’ into one group named ‘other’ with new manually created (accurate) estimate.
Great point. The decline of religion has arguably left a cultural vacuum that new organizations can fill.
Edit: updated OpenAI from 5 to 10.
From their website, AI Impacts currently has 2 researchers and 2 support staff (the current total estimate is 3).
The current estimate for Epoch is 4 which is similar to most estimates here.
I’m trying to come up with a more accurate estimate for independent researchers and ‘Other’ researchers.
New estimates:
CSER: 2-5-10 → 2-3-7
FLI: 5-5-20 → 3-4-6
Levelhume: 2-5-15 → 3-4-10
Edit: added Rethink Priorities to the list of non-technical organizations.
I re-estimated the number of researchers in these organizations and the edits are shown in the ‘EDITS’ comment below.
Copied from the EDITS comment:
- CSER: 5-5-10 → 2-5-15
- FLI: 5-5-20 → 3-5-15
- Levelhume Centre: 5-10-70 (Low confidence) → 2-5-15 (Medium confidence)My counts for CSER:
- full-time researchers: 3
- research affiliates: 4FLI: counted 5 people working on AI policy and governance.
Levelhume Centre:
- 7 senior research fellows
- 14 research fellowsMany of them work at other organizations. I think 5 is a good conservative estimate.
New footnote for the ‘Other’ row in the non-technical list of researchers (estimate is 10):
“There are about 45 research profile on Google Scholar with the ‘AI governance’ tag. I counted about 8 researchers who weren’t at the other organizations listed.”
I disagree because I think these kinds of post hoc explanations are invalidated by the hindsight fallacy. I think the FTX crash was a typical black swan because it seems much more foreseeable in retrospect than it was before the event.
To use another example, the 2008 financial crisis made sense in retrospect, but the Big Short movie shows that, before the event, even the characters shorting the mortgage bonds had strong doubts about whether they were right and most other people were completely oblivious.
Although the FTX crisis makes sense in retrospect, I have to admit that I had absolutely no idea that it was about to happen before the event.