Former AI safety research engineer, now AI governance researcher at OpenAI. Blog: thinkingcomplete.blogspot.com
richard_ngo
Karma: 6,787
As a followup to byanyothername’s questions: Could you say a little about what distinguishes your coaching from something like a CFAR workshop?
A few doubts:
It seems like MSR requires a multiverse large enough to have many well-correlated agents, but not large enough to run into the problems involved with infinite ethics. Most of my credence is on no multiverse or infinite multiverse, although I’m not particularly well-read on this issue.
My broad intuition is something like “Insofar as we can know about the values of other civilisations, they’re probably similar to our own. Insofar as we can’t, MSR isn’t relevant.” There are probably exceptions, though (e.g. we could guess the direction in which an r-selected civilisation’s values would vary from our own).
I worry that MSR is susceptible to self-mugging of some sort. I don’t have a particular example, but the general idea is that you’re correlated with other agents even if you’re being very irrational. And so you might end up doing things which seem arbitrarily irrational. But this is just a half-fledged thought, not a proper objection.
And lastly, I would have much more confidence in FDT and superrationality in general if there were a sensible metric of similarity between agents, apart from correlation (because if you always cooperate in prisoner’s dilemmas, then your choices are perfectly correlated with CooperateBot, but intuitively it’d still be more rational to defect against CooperateBot, because your decision algorithm isn’t similar to CooperateBot in the same way that it’s similar to your psychological twin). I guess this requires a solution to logical uncertainty, though.
Happy to discuss this more with you in person. Also, I suggest you cross-post to Less Wrong.
My default position would be that IKEA have an equal obligation, but that it’s much more difficult and less efficient to try and make IKEA fulfill that obligation.
I think it’s a mischaracterisation to think of virtue ethics in terms of choosing the most virtuous actions (in fact, one common objection to virtue ethics is that it doesn’t help very much in choosing actions). I think virtue ethics is probably more about being the most virtuous, and making decisions for virtuous reasons. There’s a difference: e.g. you’re probably not virtuous if you choose normally-virtuous actions for the wrong reasons.
For similar reasons, I disagree with cole_haus that virtue ethicists choose actions to produce the most virtuous outcomes (although there is at least one school of virtue ethics which seems vaguely consequentialist, the eudaimonists. See https://plato.stanford.edu/entries/ethics-virtue). Note however that I haven’t actually looked into virtue ethics in much detail.
Edit: contractarianism is a fourth approach which doesn’t fit neatly into either division
It’s a clever explanation, but I’m not sure how much to believe it without analysing other hypotheses. E.g. maybe tax-deductibility is a major factor, or maybe it’s just much harder to give away large amounts of money quickly.
Posts on the new Forum are split into two categories:
Frontpage posts are timeless content covering the ideas of effective altruism. They should be useful or interesting even to readers who only know the basic concepts of EA and aren’t very active within the community.
I’m a little confused about this description. I feel like intellectual progress often requires presupposition of fairly advanced ideas which build on each other, and which are therefore inaccessible to “readers who only know the basic concepts”. Suppose that I wrote a post outlining views on AI safety aimed at people who already know the basics of machine learning, or a post discussing a particular counter-argument to an unusual philosophical position. Would those not qualify as frontpage posts? If not, where would they go? And where do personal blogs fit into this taxonomy?
CBT-I is also recommended in Why We Sleep (see my summary of the book).
Nitpick: “The former two have diminishing returns, but the latter does not.” It definitely does—I think getting 12 or 13 hours sleep is actively worse for you than getting 9 hours.
Some cruxes on impactful alternatives to AI policy work
Your points seem plausible to me. While I don’t remember exactly what I intended by the claim above, I think that one influence was some material I’d read referencing the original “productivity paradox” of the 70s and 80s. I wasn’t aware that there was a significant uptick in the 90s, so I’ll retract my claim (which, in any case, wasn’t a great way to make the overall point I was trying to convey).
How democracy ends: a review and reevaluation
I like “science-aligned” better than “secular”, since the former implies the latter as well as a bunch of other important concepts.
Also, it’s worth noting that “everyone’s welfare is to count equally” in Will’s account is approximately equivalent to “effective altruism values all people equally” in Ozymandias’ account, but neither of them imply the following paraphrase: “from the effective altruism perspective, saving the life of a baby in Africa is exactly as good as saving the life of a baby in America, which is exactly as good as saving the life of Ozy’s baby specifically.” I understand the intention of that phrase, but actually I’d save whichever baby would grow up to have the best life. Is there any better concrete description of what impartiality actually implies?
Disentangling arguments for the importance of AI safety
I agree that it’s not too concerning, which is why I consider it weak evidence. Nevertheless, there are some changes which don’t fit the patterns you described. For example, it seems to me that newer AI safety researchers tend to consider intelligence explosions less likely, despite them being a key component of argument 1. For more details along these lines, check out the exchange between me and Wei Dai in the comments on the version of this post on the alignment forum.
I’ve actually spent a fair while thinking about CAIS, and written up my thoughts here. Overall I’m skeptical about the framework, but if it turns out to be accurate I think that would heavily mitigate arguments 1 and 2, somewhat mitigate 3, and not affect the others very much. Insofar as 4 and 5 describe AGI as an agent, that’s mostly because it’s linguistically natural to do so—I’ve now edited some of those phrases. 6b does describe AI as a species, but it’s unclear whether that conflicts with CAIS, insofar as the claim that AI will never be agentlike is a very strong one, and I’m not sure whether Drexler makes it explicitly (I discuss this point in the blog post I linked above).
“returns that can later be deployed to greater altruistic effect as AI research progresses”
This is hiding an important premise, which is that you’ll actually be able to deploy those increased resources well enough to make up for the opportunities you forego now. E.g. Paul thinks that (as an operationalisation of slow takeoff) the economy will double in 4 years before the first 1 year doubling period starts. So after that 4 year period you might end up with twice as much money but only 1 or 2 years to spend it on AI safety.
If AI research companies aren’t currently undervalued, then your Premise 4 (being an investor in such companies will generate outsized returns on the road to slow-takeoff AGI) is incorrect, because the market will have anticipated those outsized returns and priced them in to the current share price.
I agree that the extent to which individual humans are rational agents is often overstated. Nevertheless, there are many examples of humans who spend decades striving towards distant and abstract goals, who learn whatever skills and perform whatever tasks are required to reach them, and who strategically plan around or manipulate the actions of other people. If AGI is anywhere near as agentlike as humans in the sense of possessing the long-term goal-directedness I just described, that’s cause for significant concern.
(edited) I just saw your link above about growth vs value investing. I don’t think that’s a helpful distinction in this case, and when people talk about a company being undervalued I think that typically includes both unrecognised growth potential and unrecognised current value. (Maybe that’s less true for startups, but we’re talking about already-listed companies here).
I do think the core claim of “if AGI will be as big a deal as we think it’ll be, then the markets are systematically undervaluing AI companies” is a reasonable one, but the arguments you’ve given here aren’t precise enough to justify confidence, especially given the aforementioned need for caution. For example, premise 4 doesn’t actually follow directly from premise 3 because the returns could be large but not outsized compared with other investments. I think you can shore that link up, but not without contradicting your other point:
I’m not claiming that investing in AI companies will generate higher-than-average returns in the long run.
Which means (under the definition I’ve been using) that you’re not claiming that they’re undervalued.
Can you give some examples of “more responsible” ways?
I agree that in general calculating your own random digits feels a lot like rolling your own crypto. (Edit: I misunderstood the method and thought there was an easy exploit, which I was wrong about. Nevertheless at least 1⁄3 of the digits in the API response are predictable, maybe more, and the whole thing is quite small, so it might be possible to increase your probability of winning slightly by brute force calculating possibilities, assuming you get to pick your own contiguous ticket number range. My preliminary calculations suggest that this method would be too difficult, but I’m not an expert, there may be more sophisticated hacks).
Kudos for doing this. The main piece of advice which comes to mind is to make sure to push this via university EA groups. I don’t think you explicitly identified students as a target demographic in your post, but current students and new grads have the three traits which make the hotel such an attractive proposition: they’re unusually time-rich, cash-poor, and willing to relocate.