Update: The draft I mentioned is now a post!
kokotajlod
Karma: 1,122
I wonder if you think the EA community is too slow to update their strategies here. It feels like what is coming is easily among the most difficult things humanity ever has to get right and we could be doing much more if we all took current TAI forecasts more into account.
You guessed it—I believe that most of EA’s best and brightest will end up having approximately zero impact (compared to what they could have had) because they are planning for business-as-usual. The twenties are going to take a lot of people by surprise, I think. Hopefully EAs working their way up the academic hierarchy will at least be able to redirect prestige/status towards those who have been building up expertise in AI safety and AI governance, when the time comes.
Fun with +12 OOMs of Compute
I think that if I were going on outside-view economic arguments I’d probably be <50% singularity by 2100.
To what extent is this a repudiation of Roodman’s outside-view projection? My guess is you’d say something like “This new paper is more detailed and trustworthy than Roodman’s simple model, so I’m assigning it more weight, but still putting a decent amount of weight on Roodman’s being roughly correct and that’s why I said <50% instead of <10%.”
Thanks! How can an org give ops staff more freedom and involvement-if-they-want-it? What are some classic mistakes to avoid?
Thanks! I wonder if some sort of two-tiered system would work, where there’s a value-aligned staff member who is part of the core team and has lots of money and flexibility and so forth, and then they have a blank check to hire contractors who aren’t value-aligned to do various things. That might help the value-aligned staff member from becoming overworked. Idk though, I have no idea what I’m talking about. What do you think?
Do you think, on the margin, that EA orgs could get more and better ops work/people by paying substantially larger salaries?
One point in favor of 1984 and Animal Farm is that Orwell was intimitely familiar with real-life totalitarian regimes, having fought for the communists in Spain etc. His writing is more credible IMO because he’s criticizing the side he fought for rather than the side he fought against. (I mean, he’s criticizing both, for sure—his critiques apply equally to fascism—but most authors who warn us of dystopian futures are warning us against their outgroup, so to speak, whereas Orwell is warning us against what used to be his ingroup.)
Thanks, this was a surprisingly helpful answer, and I had high expectations!
This is updating me somewhat towards doing more blog posts of the sort that I’ve been doing. As it happens, I have a draft of one that is very much Category 3, let me know if you are interested in giving comments!
Your sense of why we disagree is pretty accurate, I think. The only thing I’d add is that I do think we should update downwards on low-end compute scenarios because of market efficiency considerations, just not as strongly as you perhaps, and moreover I also think that we should update upwards for various reasons (the surprising recent sucesses of deep learning, the fact that big corporations are investing heavily-by-historical-standards in AI, the fact that various experts think they are close to achieving AGI) and the upwards update mostly cancels out the downwards update IMO.
Yep, my current median is something like 2032. It fluctuates depending on how I estimate it, sometimes I adjust it up or down a bit based on how I’m feeling in the moment and recent updates, etc.
Hi Ajeya! I”m a huge fan of your timelines report, it’s by far the best thing out there on the topic as far as I know. Whenever people ask me to explain my timelines, I say “It’s like Ajeya’s, except...”
My question is, how important do you think it is for someone like me to do timelines research, compared to other kinds of research (e.g. takeoff speeds, alignment, acausal trade...)
I sometimes think that even if I managed to convince everyone to shift from median 2050 to median 2032 (an obviously unlikely scenario!), it still wouldn’t matter much because people’s decisions about what to work on are mostly driven by considerations of tractability, neglectedness, personal fit, importance, etc. and even that timelines difference would be a relatively minor consideration. On the other hand, intuitively it does feel like the difference between 2050 and 2032 is a big deal and that people who believe one when the other is true will probably make big strategic mistakes.Bonus question: Murphyjitsu: Conditional on TAI being built in 2025, what happened? (i.e. how was it built, what parts of your model were wrong, what do the next 5 years look like, what do the 5 years after 2025 look like?)
Well said. I agree that that is a path to impact for the sort of work QRI is doing, it just seems lower-priority to me than other things like working on AI alignment or AI governance. Not to mention the tractability / neglectedness concerns (philosophy is famously intractable, and there’s an entire academic discipline for it already)
Is emotional valence a particularly confused and particularly high-leverage topic, and one that might plausibly be particularly conductive getting clarity on? I think it would be hard to argue in the negative on the first two questions. Resolving the third question might be harder, but I’d point to our outputs and increasing momentum. I.e. one can levy your skepticism on literally any cause, and I think we hold up excellently in a relative sense. We may have to jump to the object-level to say more.
I don’t think I follow. Getting more clarity on emotional valence does not seem particularly high-leverage to me. What’s the argument that it is?
To your second concern, I think a lot about AI and ‘order of operations’. … But might there be path-dependencies here such that the best futures happen if we gain more clarity on consciousness, emotional valence, the human nervous system, the nature of human preferences, and so on, before we reach certain critical thresholds in superintelligence development and capacity? Also — certainly.
Certainly? I’m much less sure. I actually used to think something like this; in particular, I thought that if we didn’t program our AI to be good at philosophy, it would come to some wrong philosophical view about what consciousness is (e.g. physicalism, which I think is probably wrong) and then kill us all while thinking it was doing us a favor by uploading us (for example).
But now I think that programming our AI to be good at philosophy should be tackled directly, rather than indirectly by first solving philosophical problems ourselves and then programming the AI to know the solutions. For one thing, it’s really hard to solve millenia-old philosophical problems in a decade or two. For another, there are many such problems to solve. Finally, our AI safety schemes probably won’t involve feeding answers into the AI, so much as trying to get the AI to learn our reasoning methods and so forth, e.g. by imitating us.Widening the lens a bit, qualia research is many things, and one of these things is an investment in the human-improvement ecosystem, which I think is a lot harder to invest effectively in (yet also arguably more default-safe) than the AI improvement ecosystem. Another ‘thing’ qualia research can be thought of as being is an investment in Schelling point exploration, and this is a particularly valuable thing for AI coordination.
I don’t buy these claims yet. I guess I buy that qualia research might help improve humanity, but so would a lot of other things, e.g. exercise and nutrition. As for the Schelling point exploration thing, what does that mean in this context?
I’m confident that, even if we grant that the majority of humanity’s future trajectory will be determined by AGI trajectory — which seems plausible to me — I think it’s also reasonable to argue that qualia research is one of the highest-leverage areas for positively influencing AGI trajectory and/or the overall AGI safety landscape.
I’m interested to hear those arguments!
Thanks for this detailed and well-written report! As a philosopher (and fan of the cyberpunk aesthetic :) ) your project sounds really interesting and exciting to me. I hope I get to meet you one day and learn more. However, I currently don’t see the case for prioritising your project:
Isn’t it perplexing that we’re trying to reduce the amount of suffering and increase the amount of happiness in the world, yet we don’t have a precise definition for either suffering or happiness?
As a human collective, we want to create good futures. This encompasses helping humans have happier lives, preventing intense suffering wherever it may exist, creating safe AI, and improving animals’ lives too, both in farms and in the wild.
But what is happiness? And what is suffering?
Until we can talk about these things objectively, let alone measure and quantify them reliably, we’ll always be standing in murky water.
It seems like you could make this argument about pretty much any major philosophical question, e.g. “We’re trying to reduce the amount of suffering and increase the amount of happiness in the world, yet we don’t have a precise definition of the world, or of we, or of trying, and we haven’t rigorously established that this is what we should be doing anyway, and what does should mean anyway?”
Meanwhile, here’s my argument for why QRI’s project shouldn’t be prioritized:
--Crazy AI stuff will probably be happening in the next few decades, and if it doesn’t go well, the impact of QRI’s research will be (relatively) small or even negative.
--If it does go well, QRI’s impact will still be small, because the sort of research QRI is doing would have been done anyway after AI stuff goes well. If other people don’t do it, the current QRI researchers could do it, and probably do it even better thanks to advanced AI assistance.
Thanks! Makes sense. (To be clear, I wasn’t saying that tight control by a single political faction would be a good thing… only that it would fix the polarization problem.) I think the Civil War era was probably more polarized than today, but that’s not very comforting given what happened then. Ideally we’d be able to point to an era with greater-than-today polarization that didn’t lead to mass bloodshed. I don’t know much about the Jefferson-Adams thing but I’d be surprised if it was as bad as today.
Birds, Brains, Planes, and AI: Against Appeals to the Complexity/Mysteriousness/Efficiency of the Brain
For personal fit stuff: I agree that for intellectual work, personal fit is very important. It’s just that I have discovered, almost by accident, that I have more personal fit than I realized for things I wasn’t trained in. (You may have made a similar discovery?) Had I prioritized personal fit less early on, I would have explored more. I still wonder what sorts of things I could be doing by now if I had tried to reskill instead of continuing in philosophy. Yeah, maybe I would have discovered that I didn’t like it and gone back to philosophy, but maybe I would have discovered that I loved it. I guess this isn’t against prioritizing personal fit per se, but against how past-me interpreted the advice to prioritize personal fit.
For engaging with people outside EA: I went to a philosophy PhD program and climbed the conventional academic hierarchy for a few years. I learned a bunch of useful stuff, but I also learned a bunch of useless stuff, and a bunch of stuff which is useful but plausibly not as useful as what I would have learned working for an EA org. When I look back on what I accomplished over the last five years, almost all of the best stuff seems to be things I did on the side, extracurricular from my academic work. (e.g. doing internships at CEA etc.) I also made a bunch of friends outside EA, which I agree is nice in several ways (e.g. the ones you mention) but to my dismay I found it really hard to get people to lift a finger in the direction of helping the world, even if I could intellectually convince them that e.g. AI risk is worth taking seriously, or that the critiques and stereotypes of EA they heard were incorrect. As a counterpoint, I did have interactions with several dozen people probably, and maybe I caused more positive change than I could see, especially since the world’s not over yet and there is still time for the effects of my conversations to grow. Still though: I missed out on several year’s worth of EA work and learning by going to grad school; that’s a high opportunity cost.
As for learning things myself: I heard a lot of critiques of EA, learned a lot about other perspectives on the world, etc. but ultimately I don’t think I would be any worse off in this regard if I had just gone into an EA org for the past five years instead of grad school.
Thanks for this! I think my own experience has led to different lessons in some cases (e.g. I think I should have prioritised personal fit less and engaged less with people outside the EA community), but I nevertheless very much approve of this sort of public reflection.
Good question. Yeah, how about views of the average post from 2020 in 2020? And ditto for 90th percentile.
Whoa, Lesswrong beats SSC? That surprises me.