RobBensinger

• RobBensingerApr 9, 2023, 3:56 PM

  14 points

  0 ∶ 0

  on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down

  (Meta note: The TIME piece was previously discussed here. I’ve cross-posted the contents because the TIME version is paywalled in some countries, and is plastered with ads. This version adds some clarifying notes that Eliezer wrote on Twitter regarding the article.)

• RobBensingerMar 31, 2023, 7:43 PM

  9 points

  1 ∶ 2

  in reply to: RobBensinger’s comment on: RobBens­inger’s Shortform

  Gordon Worley adds:

  yeah been complaining about this for a while. I’m not sure exactly when things started to fall apart, but it’s been in about the last year. the quality of discussion there has fallen off a cliff because it now seems to be full of folks unfamiliar with the basics of rationality or even ea thought. ea has always not been exactly rationality, but historically there was enough overlap to make eaf a cool place. now it’s full of people who don’t share a common desire to understand the world.

  (obviously still good folks on the forum, just enough others to make it less fun and productive to post there)

• RobBensingerMar 31, 2023, 5:49 PM

  5 points

  2 ∶ 0

  in reply to: Evan_Gaensbauer’s comment on: RobBens­inger’s Shortform

  I posted it there and on Twitter. :) Honestly, it plausibly deserves a top-level EA Forum post as well; I don’t usually think memes are the best way to carry out discourse, but in this case I feel like it would be healthy for EA to be more self-aware and explicit about the fact that this dynamic is going on, and have a serious larger conversation about it.

  (And if people nit-pick some of the specific factual claims implied by my meme, all the better!)

• RobBensingerMar 31, 2023, 5:41 PM

  18 points

  3 ∶ 2

  on: RobBens­inger’s Shortform

  Some comments Duncan made in a private social media conversation:

  (Resharing because I think it’s useful for EAs to be tracking why rad people are bouncing off EA as a community, not because I share Duncan’s feeling—though I think I see where he’s coming from!)

  I have found that the EA forum is more like a “search for traitors” place, now, than like a “allies in awesomeness” place.

  Went there to see what’s up just now, and the first thing in recent comments is:

  

  Which, like. If I had different preconceptions, might land as somebody being like “Oh! Huh! What cool thing was happening here, that I don’t know about?”

  But in the current atmosphere, with my current preconceptions, feels like a knife sliding from a sheath.

  That seemed like a potential warning sign to me of cultural unhealth on the EA Forum, especially given that others shared Duncan’s sentiments.

  I asked if Duncan would like to share his impression on the EA Forum so EAs could respond and talk it out, and he said:

  Not worth it at this point! Feels like jumping into a pool of piranhas and saying “I don’t like that you’re piranhas.”

  More charitably: the EA forum has made it clear that it’s not for people like me, and doesn’t want people like me around; I can respect that and will only show up for rare and important things where my being on a different wavelength is a cost worth the benefit.

  (He was willing to let me cross-post it myself, however.)

• RobBensingerMar 31, 2023, 5:25 PM

  5 points

  1 ∶ 1

  in reply to: justsaying’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  I don’t think this is insane, and I think <20% is probably too low a threshold to carry the case—a 15% risk of extinction from AGI would mean we should be drastically, drastically more scared of AGI than of nuclear war.

• RobBensingerMar 31, 2023, 5:20 PM

  5 points

  1 ∶ 0

  in reply to: Greg_Colbourn ⏸️ ’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  Sounds right!

• RobBensingerMar 31, 2023, 1:36 AM

  9 points

  2 ∶ 0

  in reply to: Daniel_Eth’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  including, if it escalated to that level, military response (plausibly including nuclear).

  I don’t see where Eliezer has said “plausibly including nuclear”. The point of mentioning nuclear was to highlight the scale of the risk on Eliezer’s model (‘this is bad enough that even a nuclear confrontation would be preferable’), not to predict nuclear confrontation.

• RobBensingerMar 31, 2023, 1:33 AM

  11 points

  1 ∶ 0

  in reply to: TeddyW’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  I believe you that you’re honestly speaking for your own views, and for the views of lots of other people in ML. From experience, I know that there are also lots of people in ML who do think AGI is likely to kill us all, and choose to work on advancing capabilities anyway. (With the justification Eliezer highlighted, and in many cases with other justifications, though I don’t think these are adequate.)

  I’d be interested to hear your views about this, and why you don’t think superintelligence risk is a reason to pause scaling today. I can imagine a variety of reasons someone might think this, but I have no idea what your reason is, and I think conversation about this is often quite productive.

• RobBensingerMar 31, 2023, 1:28 AM

  11 points

  5 ∶ 1

  in reply to: Daniel_Eth’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  Yep, I +1 this response. I don’t think Eliezer is proposing anything unusual (given the belief that AGI is more dangerous than nukes, which is a very common belief in EA, though not universally shared). I think the unusual aspect is mostly just that Eliezer is being frank and honest about what treating AGI development and proliferation like nuclear proliferation looks like in the real world.

  He explained his reasons for doing that here:

  • https://​​twitter.com/​​ESYudkowsky/​​status/​​1641452620081668098

  • https://​​www.lesswrong.com/​​posts/​​Lz64L3yJEtYGkzMzu/​​rationality-and-the-english-language

• RobBensingerMar 30, 2023, 9:48 AM

  23 points

  9 ∶ 3

  in reply to: David Mathers🔸’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  Agreed! I think the policy proposal is a good one that makes a lot of sense, and I also think this is a good time to remind people that “international treaties with teeth are plausibly necessary here” doesn’t mean it’s open season on terrible naively consequentialist ideas that sound “similarly extreme”. See the Death With Dignity FAQ.

• RobBensingerMar 22, 2023, 3:25 AM

  25 points

  4 ∶ 0

  on: Where I’m at with AI risk: con­vinced of dan­ger but not (yet) of doom

  Naively, it seems as if killing everyone would earn AI a massive penalty in training: why would it develop aims that are consistent with doing that?

  An AI killing everyone wouldn’t earn a massive penalty in training, because there won’t be humans alive in that scenario to assign the penalty. Cf. point 10 in AGI Ruin:

  10. You can’t train alignment by running lethally dangerous cognitions, observing whether the outputs kill or deceive or corrupt the operators, assigning a loss, and doing supervised learning. On anything like the standard ML paradigm, you would need to somehow generalize optimization-for-alignment you did in safe conditions, across a big distributional shift to dangerous conditions. (Some generalization of this seems like it would have to be true even outside that paradigm; you wouldn’t be working on a live unaligned superintelligence to align it.) This alone is a point that is sufficient to kill a lot of naive proposals from people who never did or could concretely sketch out any specific scenario of what training they’d do, in order to align what output—which is why, of course, they never concretely sketch anything like that. Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far out-of-distribution from safer building/​training operations that didn’t kill you. This is where a huge amount of lethality comes from on anything remotely resembling the present paradigm. Unaligned operation at a dangerous level of intelligence*capability will kill you; so, if you’re starting with an unaligned system and labeling outputs in order to get it to learn alignment, the training regime or building regime must be operating at some lower level of intelligence*capability that is passively safe, where its currently-unaligned operation does not pose any threat. (Note that anything substantially smarter than you poses a threat given any realistic level of capability. Eg, “being able to produce outputs that humans look at” is probably sufficient for a generally much-smarter-than-human AGI to navigate its way out of the causal systems that are humans, especially in the real world where somebody trained the system on terabytes of Internet text, rather than somehow keeping it ignorant of the latent causes of its source code and training environments.)

  A related topic is Distinguishing Test From Training.

  But if future AIs are trained like current ones are—by being given vast amounts of human-derived data—I’d naively expect AI goals to have the human-like property of being fuzzy, complex and constrained

  I think you need to say more about what the system is being trained for (and how we train it for that). Just saying “facts about humans are in the data” doesn’t provide a causal mechanism by which the AI acts in human-like ways, any more than “facts about clouds are in the data” provides a mechanism by which the AI role-plays being a cloud.

  If there’s a lot more human data than meteorological data on clouds in the training data, then I guess I’d consider it more likely that the AI has goals that are somehow human-related and/​or imitative-of-humans, than that the AI has goals that are cloud-related or imitative-of-clouds? But I don’t expect the resultant minds to be all that human-like or all that cloud-like, and if there are resemblances, they could be bad ones and not just good ones.

  “Constrained” is a particularly hard target to hit because it actively pushes against producing impressive SotA-advancing results. I’d expect the first AGI systems to be built by labs that are pushing full steam ahead on making crazy impressive things happen ASAP, which means you’re actively optimizing against minds that are trying to limit their impact, intelligence, or power.

  (Also, I think training AGI systems by giving them vast amounts of human-derived data is a terrible idea, and cuts out many of the most promising tactics for aligning AGI systems. But that’s maybe a topic to save for after we agree about whether you just get human-ish values for free by exposing alien minds or mind-building-processes to lots of facts about humans.)

  So there still seems an inferential leap from ‘existing systems are sometimes misaligned’ to ‘superintelligent AI will most likely be catastrophically misaligned’.

  Note that there’s a difference between ‘how much goal overlap is there’ and ‘how catastrophic is the non-overlap’. You gave an argument that human goals overlap some with the goals of evolution, but you didn’t give an argument that humans are non-catastrophic from the (pseudo-)perspective of evolution. That would depend on whether humans will produce lots of copies of human DNA in the future.

  Power-seeking, wealth-seeking, and self-protection are all instrumentally useful unless your goals include not having power, not having wealth, and not resisting human interference.

  Yep! The orthogonality doesn’t just show that unfriendly goals are possible; it shows that friendly goals are possible too.

  power and survival are instrumentally convergent for humans too, but not all humans maximally seek these things (even if they can). What will be different about AI?

  Some good discussion of this in Superintelligent AI is necessary for an amazing future, but far from sufficient, and in Niceness is unnatural.

Yud­kowsky on AGI risk on the Ban­kless podcast

• RobBensingerMar 8, 2023, 12:55 AM

  3 points

  0 ∶ 0

  in reply to: O Carciente’s comment on: On Philos­o­phy Tube’s Video on Effec­tive Altruism

  The word “epistemic” is philosophy jargon. The phrase “epistemic status” in the link you gave isn’t a separate piece of jargon, it’s just the normal word “status” modified by the word “epistemic”.

  The original comment I was replying to said:

  “A minor gripe I have about LW, and EA by extension, is that words with a specific meaning in philosophy are misused and therefore take on a different meaning – take “epistemic status”, which has grown out of its original intended meaning of how confident one is in one’s claim and is now used more to describe someone’s background and raise general caveats and flags for where someone might have blind spots.”

  If the claim is that rationalists are misusing the word “epistemic”, not some specific unfamiliar-to-me new piece of jargon (“epistemic status”), then the claim is based on a misunderstanding of the word “epistemic”. Epistemic in philosophy means “pertaining to knowledge (belief justification, reliability, accuracy, reasonableness, warrant, etc.)”, not “pertaining to confidence level”.

  Someone’s “epistemic status” includes what they believe and how strongly they believe it, but it also includes anything that’s relevant to how justified, reasonable, supported, based-on-reliable-processes, etc. your beliefs are. Like, “epistemic status: I wrote this whole hungry, which often makes people irritable and causes them to have more brain farts, which reduces the expected reliability and justifiedness of the stuff I wrote” is totally legit. And if people have the background knowledge to understand why you might want to flag that you were hungry, it’s completely fine to write “epistemic status: written while hungry” as a shorthand.

  (I do think rationalists sometimes put other stuff under “epistemic status” as a joke, but “rationalists joke too much” is a different criticism than “rationalists have their own nonstandard meaning for the word ‘epistemic’”.)

• RobBensingerFeb 28, 2023, 2:43 AM

  3 points

  1 ∶ 0

  in reply to: Jessica Wen’s comment on: On Philos­o­phy Tube’s Video on Effec­tive Altruism

  A minor gripe I have about LW, and EA by extension, is that words with a specific meaning in philosophy are misused and therefore take on a different meaning

  The version of “rationalist” you’re talking about is a common usage, but:

  • The oldest meaning of “rationalist” is about truth, science, inquiry, and good epistemics rather than about “observation matters less than abstract thought”.

  • Rationalists’ conception of “rationality” isn’t our invention: we’re just using the standard conception from cognitive science.

  • Lots of groups have called themselves “rationalist” in a LW-like sense prior to LessWrong. It’s one of the more common terms humanists, secularists, atheists, materialists, etc. historically used to distinguish themselves from religionists, purveyors of pseudoscience, and the like.

  Also, the rationalist vs. empiricist debate in philosophy is mostly of historical interest; it’s not clear to me that it should matter much to non-historians nowadays.

  take “epistemic status”

  “Epistemic status” isn’t philosophy jargon, is it?

  I took it to be riffing on early LiveJournal posts that began with text like “status: bored” or “current mood: busy”, adding the qualifier “epistemic” as a cute variation.

• RobBensingerFeb 26, 2023, 10:34 AM

  26 points

  15 ∶ 1

  on: On Philos­o­phy Tube’s Video on Effec­tive Altruism

  EA heavily draws from Rationalism, which views reason as the chief source of knowledge; specifically, EA heavily prioritises quantitative analysis over qualitative analysis.

  This misunderstands what rationalism is (in the context of EA, LW, etc.).

  You’re thinking “reason, rationality, deliberation, explicit thought, etc. as opposed to emotion, intuition, implicit thought, etc.” LessWrong-style “rationality” is instead about truth as opposed to falsehood (“epistemic rationality”) and succeeding in your goals as opposed to failing (“instrumental rationality”).

  Classic LW content like What Do We Mean By “Rationality”?, The Straw Vulcan, Feeling Rational, and When (Not) To Use Probabilities talk about how different these concepts are: using deliberative, explicit, quantitative, etc. thought is rational (in the LW sense) if it helps you better understand things or helps you succeed in life, but emotions, hunches, intuitions, etc. are also a crucial part of understanding the world and achieving your goals.

• RobBensingerFeb 26, 2023, 10:12 AM

  4 points

  1 ∶ 0

  in reply to: FJehn’s comment on: On Philos­o­phy Tube’s Video on Effec­tive Altruism

  “There is not objective source of truth that tells you that for example quality adjusted life years are the best objective measure.”

  There’s no objective source of truth telling humans to value what we value; on some level it’s just a brute fact that we have certain values. But given a set of values, some metrics will do better vs. worse at describing the values.

  Or in other words: Facts about how much people prefer one thing relative to other things are “subjective” in the weak sense that all psychological facts are subjective: they’re about subjects /​ minds. But psychology facts aren’t “subjective” in a sense like “there are no facts of the matter about minds”. Minds are just as real a part of the world as chairs, electrons, and zebras.

  Consider, for example, a measure that says “a sunburn is ¹⁄₂ as bad as a migraine” versus one that says “a sunburn is a billion times as bad as a migraine”. We can decompose this into a factual claim about the relative preferences of some group of agents, plus a normative claim that calls the things that group dislikes “bad”.

  For practical purposes, the important contribution of welfare metrics isn’t “telling us that the things we dislike are bad”; realists are already happy to run with this, and anti-realists are happy to play along with the basic behavioral take-aways in practice.

  Instead, the important contribution is the factual claim about what a group prefers, which is as objective/​subjective as any other psych claim. Viewed through that lens, even if neither of the claims above is perfectly accurate, it seems clear that the “1/​2 as bad” claim is a lot closer to the psychological truth.

• RobBensingerFeb 14, 2023, 5:01 PM

  4 points

  2 ∶ 1

  in reply to: titotal’s comment on: Ele­ments of Ra­tion­al­ist Discourse

  Perhaps the real argument I’m making is that “don’t use weird jargon (outside of lesswrong)” should be another principle.

  Seems like an obviously bad rule to me. “Don’t use weird jargon anywhere in the world except LessWrong” is a way stronger claim than “Don’t use weird jargon in an adversarial debate where you’re trying to rhetorically out-manipulate a dishonest creationist”.

  (This proposal also strikes me as weirdly minor compared to the other rules. Partly because it’s covered to some degree by “Reducibility” already, which encourages people to only use jargon if they’re willing and able to paraphrase it away or explain it on request.)

  “On my model of EA and of the larger world, trying out stuff like this is one of the best ways for EA to increase the probability it has a positive impact.”

  to

  “”I believe that trying out these techniques will probably improve the effectiveness of EA”.”

  Seems like a bad paraphrase to me, in a few ways:

  • “On my model of EA and of the larger world” is actually doing some important work here. The thing I’m trying to concisely gesture at is that I have a ton of complicated background beliefs about the world, and also about how EA should interface with the wider world, that make me much more confident that guidelines like the one in the OP are good ones.
    
    I actually want to signpost all of that pretty clearly, so people know they can follow up and argue with me about the world and about EA if they have different beliefs/​models about how EA can do the most good.

  • “X will probably improve Y” is a lot weaker than “X is one of the best ways to improve Y”.

  • “Improve the effectiveness of EA” is very vague, and (to my eye) makes it sound like I think these guidelines are useful for things like “making EAs more productive at doing the things they’re already trying to do”.
    
    I do think the guidelines would have that effect, but I also think that they’d help people pick better cause areas and interventions to work on, by making people’s reasoning processes and discussions clearer, more substantive, and more cruxy. You could say that this is also increasing our “effectiveness” (especially in EA settings, where “effective” takes on some vague jargoniness of its own), but connotationally it would still be misleading, especially for EAs who are using “effective” in the normal colloquial sense.

  I think overly-jargony, needlessly complicated text is bad. But if “On my model of EA and of the larger world, trying out stuff like this is one of the best ways for EA to increase the probability it has a positive impact.” crosses your bar for “too jargony” and “too complicated”, I think you’re setting your bar waaaay too low for the EA Forum audience.

• RobBensingerFeb 14, 2023, 4:43 PM

  4 points

  3 ∶ 0

  in reply to: Guy Raveh’s comment on: Ele­ments of Ra­tion­al­ist Discourse

  My subjective feeling is that all of the terms on this list make conversations less clear, more exhausting, and broadly unpleasant.

  Why? Picking an example that seems especially innocuous to me: why do you feel like the word “probability” (used to refer to degrees of belief strength) makes conversations “less clear”? What are the specific ways you think it makes for more-exhausting or more-unpleasant conversations?

  You could say that’s unsurprising, coming from a person who deliberately avoids LessWrong.

  I think people who dislike LW should also steal useful terms and habits of thought like these, if any seem useful. In general, a pretty core mental motion in my experience is: if someone you dislike does a thing that works, steal that technique from them and get value from it yourself.

  Don’t handicap yourself by cutting out all useful ways of thinking, ideas, arguments, etc. that come from a source you dislike. Say “fuck the source” and then grab whatever’s useful and ditch the rest.

  If the only problem were “this concept is good but I don’t want to use a word that LessWrong uses”, I’d just suggest coming up with a new label for the same concept and using that. (The labels aren’t the important part.)

• RobBensingerFeb 14, 2023, 12:07 PM

  3 points

  3 ∶ 0

  in reply to: RobBensinger’s comment on: Ele­ments of Ra­tion­al­ist Discourse

  On my model of EA and of the larger world, trying out stuff like this is one of the best ways for EA to increase the probability it has a positive impact.

  “Adopt weird jargon”, notably, isn’t one of the items on the list.

  I liked Nate’s argument for weird jargon enough that I included it in footnote 6 (while mainly looking for the explanation of “my model is...”), but IMO you can follow all ten items on the list without using any weird jargon. Though giving probabilities to things while trying to be calibrated does inherently have a lot of the properties of jargon: people who are used to “90% confidence” meaning something a lot fuzzier (that turns out to be wrong quite regularly) may be confused initially when they realize that you literally mean there’s a 9-in-10 chance you’re right.

  The jargon from this post I especially think EAs should use routinely is:

  • symmetric vs. asymmetric weapons

  • crux

  • (rationalist) taboo

  • local validity

  • inferential gap /​ inferential distance

  • Ideological Turing Test (ITT)

  • Typical Mind Fallacy

  • terms related to Bayesian probability: probability, priors, retrodict, etc.

  Where some of these (“crux”, “probability”) are common English words, but I’m proposing using them in a narrower and more precise way.

• RobBensingerFeb 14, 2023, 12:05 PM

  16 points

  8 ∶ 1

  in reply to: titotal’s comment on: Ele­ments of Ra­tion­al­ist Discourse

  The main disadvantage of rationalist discourse is that it can do pretty badly when talking to non-rationalists.

  I think posts on LessWrong often do badly on that metric, because they aren’t trying to appeal to (e.g.) the voting public. Which seems to me like the right call on LW’s part.

  I think the norms themselves will actually help you do better at communicating with people outside your bubble, because a lot of them are common-sense (but not commonly applied!) ideas for bridging the gap between yourself and people who don’t have the same background as you.

  • Be internally clear about what your primary conversational goals are, and focus on the things that you expect to help most with those goals, rather than getting side-tracked every time a shiny tangent catches your eye.

  • Treat people you’re talking to with respect, err on the side of assuming good faith, and try to understand their perspective.

  • Try to use simple, concrete language.

  • Make your goals and preferences explicit in relatively chill descriptive language, rather than relying on outrage, shaming, etc. to try to pressure people into agreeing with you.

  • Flag when there are ways to test a claim, and consider testing it if it’s easy. (E.g., “oh wait, let’s google that.”)

  • Etc.

  I.e., these norms play well with multicultural, diverse groups of people.

  I do think they’re better for cooperative dynamics than for heavily adversarial ones: if you’re trying to understand the perspective of someone else, have them better understand your perspective, treat the other person like a peer, respect their autonomy and agency, learn more about the world in collaboration with them, etc., then I think these norms are spot-on.

  If you’re trying to manipulate them or sneak ideas past their defenses, then I don’t think these norms are ideal (though I think that’s generally a bad thing to do, and I think EA will do a lot better and be a healthier environment if it moves heavily away from that approach to discourse).

  If you’re interacting with someone else who’s acting adversarially toward you, then I think these norms aren’t bad but they have their emphasis in the wrong place. Like, “Goodwill” leaves room for noticing bad actors and responding accordingly (see footnote 5), but if I were specifically giving people advice for dealing with bad actors, I don’t think any version of “Goodwill” would go on my top ten list of tips and tricks to employ.

  Instead, these norms are aimed at a target more like “a healthy intellectual community that’s trying to collaboratively figure out what’s true (that can also respond well when bad actors show up in its spaces, but that’s more like a top-ten desideratum rather than being the #1 desideratum)”.

  “Trick an audience of laypeople into believing your views faster than a creationist can trick that audience into believing their views” is definitely not what these discourse norms are optimized for helping with, and I think that’s to their credit. Basically zero EAs should be focusing on a goal like that, IMO, and if it did make sense for a rare EA to skill up in that, they definitely shouldn’t import those norms and habits into discussions with other EAs.