Most of my stuff (even the stuff of interest to EAs) can be found on LessWrong: https://www.lesswrong.com/users/daniel-kokotajlo
kokotajlod
Karma: 2,437
Great, thank you!
Thanks for posting this! GiveWell was a big part of how I got involved with EA back in the day & I’m glad you are still going strong.
Off the top of your head, how would your recommendations change if you only cared about the next 5 years? (alternatively, if you had roughly a 10%/year discount rate) This may influence my donation.
Since when was he lawful good? I’ve never thought of him as anything but chaotic.
The situation is much worse than that. It’s more like: They are worried about the possibility that the first ever nuclear explosion will ignite the upper atmosphere and set the whole earth ablaze. (You’ve probably read, this is a real concern they had). Except in this hypothetical the preliminary calculations are turning up the answer of Yes no matter how they run them. So they are continuing to massage the calculations and make the modelling software more realistic in the hopes of a No answer, and also advocating for changes to the design of the bomb that’ll hopefully mitigate the risk, and also advocating for the whole project to slow down before it’s too late, but the higher-ups have go fever & so it looks like in a few years the whole world will be on fire. Meanwhile, some other people are talking to the handful of Los Alamos physicists and saying “but even if the atmosphere doesn’t catch on fire, what about arms races, accidents, terrorism, etc.?” and the physicists are like “lol yeah that’s gonna be a whole big problem if we manage so survive the first test, which unfortunately we probably won’t. We’d be working on that problem if this one didn’t take priority.”
OK, sure.
Instrumental reasons supporting this dogma: The dogma helps us all stay sane and focused on the mission instead of fighting each other, so we have reason to promote it that is independent of whether or not it is true. (By contrast, an epistemic reason supporting the dogma would be a reason to think it is true, rather than merely a reason to think it is helpful/useful/etc.)
Inner alignment problem: Well, it’s generally considered to be an open unsolved problem. We don’t know how to make the goals/values/etc of the hypothetical superhuman AGI correspond in any predictable way to the reward signal or training setup—I mean, yeah, no doubt there is a correspondence, but we don’t understand it well enough to say “Given such-and-such a training environment and reward signal, the eventual goals/values/etc of the eventual AGI will be so-and-so.” So we can’t make the learning process zero in on even fairly simple goals like “maximize the amount of diamond in the universe.” For an example of an attempt to do so, a proposal that maaaybe might work, see https://www.lesswrong.com/posts/k4AQqboXz8iE5TNXK/a-shot-at-the-diamond-alignment-problem Though actually this isn’t even a proposal to get that, it’s a proposal to get the much weaker thing of an AGI that makes a lot of diamond eventually.
I think that it is unfair to characterize it as something that hasn’t been questioned. It has in fact been argued for at length. See e.g. the literature on the inner alignment problem. I agree there are also instrumental reasons supporting this dogma, but even if there weren’t, I’d still believe it and most alignment researchers would still believe it, because it is a pretty straightforward inference to make if you understand the alignment literature.
IMO: Most of the difficulty in technical alignment is figuring out how to robustly align to any particular values whatsoever. Mine, yours, humanitys, all sentient life on earth, etc. All roughly equally difficult. “Human values” is probably the catchphrase mostly for instrumental reasons—we are talking to other humans, after all, and in particular to liberal egalitarian humans who are concerned about some people being left out and especially concerned about an individual or small group hoarding power. Insofar as lots of humans were super concerned that animals would be left out too, we’d be saying humans+animals. The hard part isn’t deciding who to align to, it’s figuring out how to align to anything at all.
A counter to that second thing is: Well we can always just give to GiveDirectly or something like that.
Huh, now that you mention it, I think the third voice thinks that much higher spending would be negative, not just a lot smaller or zero. So maybe that’s what’s going on: The third voice intuits that there are backfire risks along the lines of “EA gets a reputation for being ridiculously profligate” that the model doesn’t model?
Maybe another thing that’s going on is that maybe we literally are funding all the opportunities that seem all-things-net-positive to us. The model assumes an infinite supply of opportunities, of diminishing quality, but in fact maybe there are literally only finitely many and we’ve exhausted them all.
Further, for all results, we begin maximum crunching after the modal AGI arrival date, which is understandable while the rate of growth of the movement is greater than the rate of decrease in probability of AGI (times the discount factor).
Huh, I find that super counterintuitive. Is this maybe a reductio against the model?
A third voice is saying “Well, clearly lowering our bar for spending that much would be a terrible idea, therefore something about this model must be wrong—but which part specifically?”
In most cases, we find that the optimal spending schedule is between 5% and 15% better than the ‘default’ strategy of just spending the interest one accrues and generally between 5% to 35% better than a naive projection of the community’s current spending rate
I am curious about the right way to think about this. One voice in my head is saying “OMG here is One Simple Trick to get 5-35% extra lifetime impact for the entire EA community—just lower our bar for spending until we are spending X% per year as the schedule indicates!” Another voice in my head is saying “Huh, 35% extra lifetime impact is small potatoes actually, there’s probably all sorts of mistakes we are making that, if corrected, would yield similar effects. Maybe I should be trying to identify and correct those mistakes instead.”
I lean more towards the first voice than the second currently.
Thanks this is great!
My intuition strongly suggests that it would take less than a thousand years to rebuild industrial society, even starting from hunter-gatherer levels.
Argument to articulate my intuition:
--It would take at most something like 100 years to get to something like ancient Athens, a primitive agrarian economy but rich enough that some people could afford to think and learn etc., and with a strong culture of science/discovery/engineering. The italicized part is the hard part, I think—it happened in Athens and various other city-states but not in most of them. However, 100 years after the collapse the cultural knowledge of the Before Times would still be present, so I think the average city-state 100 years after collapse would have levels of science/discovery/engineering culture that would be at least as high as Athens.
--Ancient Athens, if it had loads of artefacts from an industrial civilization and diagrams lying around for how to build them, probably could have figured out to get there in 100 years or so. The bottleneck historically was NOT the sheer clock time it takes to go through the various stages (iteratively improving a set of tools and processes and accumulating resources which you use to build the next tech level, which you then improve to build the tech level after that, etc.) but rather the clock time it took to invent the new technologies. (A major restriction was that people didn’t generally think inventing things was a worthwhile pursuit; the governments of the time thought that building huge monuments was a better use of resources. Well, if they thought they could be FIRING CANNONS in 50 years and FLYING IN AIRPLANES BOMBING THEIR ENEMIES in 100 years, if only they put loads of money and effort into science and tech, they surely would!)
--Conclusion: Estimated ~200 year timeline to rebuild civilization starting from industrial era.
This estimate is way more optimistic than yours, I wonder why. Which of the premises above would you disagree with? I’m guessing the second one?
Fair enough. I think my comment came across as hostile to to the OP, which was not my intention but on reflection I should have anticipated. Sorry.
That sounds like a useful technique. “Outside view” would be a good term for it if it wasn’t already being used to mean so many other things. :/ How about “Neutral observer” or “friend’s advice” or “hypothetical friend?”
Oops, accidentally voted twice on this. Didn’t occur to me that the LW and EAF versions were the same underlying poll.
In addition to that, it’s important not just that you actually have high integrity but that people believe you do. And people will be rightly hesitant to believe that you do if you are going around saying that the morally correct thing to do is maximize expected utility but don’t worry it’s always and everywhere true that the way to maximize expected utility is to act as if you have high integrity. There are two strategies available, then: Actually have high integrity, which means not being 100% a utilitarian/consequentialist, or carry out an extremely convincing deception campaign to fool people into thinking you have high integrity. I recommend the former & if you attempt the latter, fuck you.
In practice, many of the utilitarians/consequentialists don’t see the negative outcomes themselves, or at least sufficiently many of them don’t that things will go to shit pretty quickly. (Relatedly, see the Unilateralists’ Curse, the Epistemic Prisoner’s Dilemma, and pretty much the entire literature of game theory, all those collective action problems...).
I’m happy to see OpenAI and OpenAI Alignment Team get recognition/credit for having a plan and making it public. Well deserved I’d say. (ETA: To be clear, like the OP I don’t currently expect the plan to work as stated; I expect us to need to pivot eventually & hope a better plan comes along before then!)