RobBensinger

• RobBensinger 31 Mar 2023 1:36 UTC

  2 points

  0

  in reply to: Daniel_Eth’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  including, if it escalated to that level, military response (plausibly including nuclear).

  I don’t see where Eliezer has said “plausibly including nuclear”. The point of mentioning nuclear was to highlight the scale of the risk on Eliezer’s model (‘this is bad enough that even a nuclear confrontation would be preferable’), not to predict nuclear confrontation.

• RobBensinger 31 Mar 2023 1:33 UTC

  4 points

  0

  in reply to: TeddyW’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  I believe you that you’re honestly speaking for your own views, and for the views of lots of other people in ML. From experience, I know that there are also lots of people in ML who do think AGI is likely to kill us all, and choose to work on advancing capabilities anyway. (With the justification Eliezer highlighted, and in many cases with other justifications, though I don’t think these are adequate.)

  I’d be interested to hear your views about this, and why you don’t think superintelligence risk is a reason to pause scaling today. I can imagine a variety of reasons someone might think this, but I have no idea what your reason is, and I think conversation about this is often quite productive.

• RobBensinger 31 Mar 2023 1:28 UTC

  7 points

  2

  in reply to: Daniel_Eth’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  Yep, I +1 this response. I don’t think Eliezer is proposing anything unusual (given the belief that AGI is more dangerous than nukes, which is a very common belief in EA, though not universally shared). I think the unusual aspect is mostly just that Eliezer is being frank and honest about what treating AGI development and proliferation like nuclear proliferation looks like in the real world.

  He explained his reasons for doing that here:

  • https://​​twitter.com/​​ESYudkowsky/​​status/​​1641452620081668098

  • https://​​www.lesswrong.com/​​posts/​​Lz64L3yJEtYGkzMzu/​​rationality-and-the-english-language

• RobBensinger 30 Mar 2023 9:48 UTC

  17 points

  12

  in reply to: Dr. David Mathers’s comment on: Paus­ing AI Devel­op­ments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky

  Agreed! I think the policy proposal is a good one that makes a lot of sense, and I also think this is a good time to remind people that “international treaties with teeth are plausibly necessary here” doesn’t mean it’s open season on terrible naively consequentialist ideas that sound “similarly extreme”. See the Death With Dignity FAQ.

• RobBensinger 22 Mar 2023 3:25 UTC

  18 points

  8

  on: Where I’m at with AI risk: con­vinced of dan­ger but not (yet) of doom

  Naively, it seems as if killing everyone would earn AI a massive penalty in training: why would it develop aims that are consistent with doing that?

  An AI killing everyone wouldn’t earn a massive penalty in training, because there won’t be humans alive in that scenario to assign the penalty. Cf. point 10 in AGI Ruin:

  10. You can’t train alignment by running lethally dangerous cognitions, observing whether the outputs kill or deceive or corrupt the operators, assigning a loss, and doing supervised learning. On anything like the standard ML paradigm, you would need to somehow generalize optimization-for-alignment you did in safe conditions, across a big distributional shift to dangerous conditions. (Some generalization of this seems like it would have to be true even outside that paradigm; you wouldn’t be working on a live unaligned superintelligence to align it.) This alone is a point that is sufficient to kill a lot of naive proposals from people who never did or could concretely sketch out any specific scenario of what training they’d do, in order to align what output—which is why, of course, they never concretely sketch anything like that. Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far out-of-distribution from safer building/​training operations that didn’t kill you. This is where a huge amount of lethality comes from on anything remotely resembling the present paradigm. Unaligned operation at a dangerous level of intelligence*capability will kill you; so, if you’re starting with an unaligned system and labeling outputs in order to get it to learn alignment, the training regime or building regime must be operating at some lower level of intelligence*capability that is passively safe, where its currently-unaligned operation does not pose any threat. (Note that anything substantially smarter than you poses a threat given any realistic level of capability. Eg, “being able to produce outputs that humans look at” is probably sufficient for a generally much-smarter-than-human AGI to navigate its way out of the causal systems that are humans, especially in the real world where somebody trained the system on terabytes of Internet text, rather than somehow keeping it ignorant of the latent causes of its source code and training environments.)

  A related topic is Distinguishing Test From Training.

  But if future AIs are trained like current ones are—by being given vast amounts of human-derived data—I’d naively expect AI goals to have the human-like property of being fuzzy, complex and constrained

  I think you need to say more about what the system is being trained for (and how we train it for that). Just saying “facts about humans are in the data” doesn’t provide a causal mechanism by which the AI acts in human-like ways, any more than “facts about clouds are in the data” provides a mechanism by which the AI role-plays being a cloud.

  If there’s a lot more human data than meteorological data on clouds in the training data, then I guess I’d consider it more likely that the AI has goals that are somehow human-related and/​or imitative-of-humans, than that the AI has goals that are cloud-related or imitative-of-clouds? But I don’t expect the resultant minds to be all that human-like or all that cloud-like, and if there are resemblances, they could be bad ones and not just good ones.

  “Constrained” is a particularly hard target to hit because it actively pushes against producing impressive SotA-advancing results. I’d expect the first AGI systems to be built by labs that are pushing full steam ahead on making crazy impressive things happen ASAP, which means you’re actively optimizing against minds that are trying to limit their impact, intelligence, or power.

  (Also, I think training AGI systems by giving them vast amounts of human-derived data is a terrible idea, and cuts out many of the most promising tactics for aligning AGI systems. But that’s maybe a topic to save for after we agree about whether you just get human-ish values for free by exposing alien minds or mind-building-processes to lots of facts about humans.)

  So there still seems an inferential leap from ‘existing systems are sometimes misaligned’ to ‘superintelligent AI will most likely be catastrophically misaligned’.

  Note that there’s a difference between ‘how much goal overlap is there’ and ‘how catastrophic is the non-overlap’. You gave an argument that human goals overlap some with the goals of evolution, but you didn’t give an argument that humans are non-catastrophic from the (pseudo-)perspective of evolution. That would depend on whether humans will produce lots of copies of human DNA in the future.

  Power-seeking, wealth-seeking, and self-protection are all instrumentally useful unless your goals include not having power, not having wealth, and not resisting human interference.

  Yep! The orthogonality doesn’t just show that unfriendly goals are possible; it shows that friendly goals are possible too.

  power and survival are instrumentally convergent for humans too, but not all humans maximally seek these things (even if they can). What will be different about AI?

  Some good discussion of this in Superintelligent AI is necessary for an amazing future, but far from sufficient, and in Niceness is unnatural.

Yud­kowsky on AGI risk on the Ban­kless podcast

• RobBensinger 8 Mar 2023 0:55 UTC

  3 points

  0

  in reply to: O Carciente’s comment on: On Philos­o­phy Tube’s Video on Effec­tive Altruism

  The word “epistemic” is philosophy jargon. The phrase “epistemic status” in the link you gave isn’t a separate piece of jargon, it’s just the normal word “status” modified by the word “epistemic”.

  The original comment I was replying to said:

  “A minor gripe I have about LW, and EA by extension, is that words with a specific meaning in philosophy are misused and therefore take on a different meaning – take “epistemic status”, which has grown out of its original intended meaning of how confident one is in one’s claim and is now used more to describe someone’s background and raise general caveats and flags for where someone might have blind spots.”

  If the claim is that rationalists are misusing the word “epistemic”, not some specific unfamiliar-to-me new piece of jargon (“epistemic status”), then the claim is based on a misunderstanding of the word “epistemic”. Epistemic in philosophy means “pertaining to knowledge (belief justification, reliability, accuracy, reasonableness, warrant, etc.)”, not “pertaining to confidence level”.

  Someone’s “epistemic status” includes what they believe and how strongly they believe it, but it also includes anything that’s relevant to how justified, reasonable, supported, based-on-reliable-processes, etc. your beliefs are. Like, “epistemic status: I wrote this whole hungry, which often makes people irritable and causes them to have more brain farts, which reduces the expected reliability and justifiedness of the stuff I wrote” is totally legit. And if people have the background knowledge to understand why you might want to flag that you were hungry, it’s completely fine to write “epistemic status: written while hungry” as a shorthand.

  (I do think rationalists sometimes put other stuff under “epistemic status” as a joke, but “rationalists joke too much” is a different criticism than “rationalists have their own nonstandard meaning for the word ‘epistemic’”.)

• RobBensinger 28 Feb 2023 2:43 UTC

  3 points

  1

  in reply to: Jessica Wen’s comment on: On Philos­o­phy Tube’s Video on Effec­tive Altruism

  A minor gripe I have about LW, and EA by extension, is that words with a specific meaning in philosophy are misused and therefore take on a different meaning

  The version of “rationalist” you’re talking about is a common usage, but:

  • The oldest meaning of “rationalist” is about truth, science, inquiry, and good epistemics rather than about “observation matters less than abstract thought”.

  • Rationalists’ conception of “rationality” isn’t our invention: we’re just using the standard conception from cognitive science.

  • Lots of groups have called themselves “rationalist” in a LW-like sense prior to LessWrong. It’s one of the more common terms humanists, secularists, atheists, materialists, etc. historically used to distinguish themselves from religionists, purveyors of pseudoscience, and the like.

  Also, the rationalist vs. empiricist debate in philosophy is mostly of historical interest; it’s not clear to me that it should matter much to non-historians nowadays.

  take “epistemic status”

  “Epistemic status” isn’t philosophy jargon, is it?

  I took it to be riffing on early LiveJournal posts that began with text like “status: bored” or “current mood: busy”, adding the qualifier “epistemic” as a cute variation.

• RobBensinger 26 Feb 2023 10:34 UTC

  28 points

  43

  on: On Philos­o­phy Tube’s Video on Effec­tive Altruism

  EA heavily draws from Rationalism, which views reason as the chief source of knowledge; specifically, EA heavily prioritises quantitative analysis over qualitative analysis.

  This misunderstands what rationalism is (in the context of EA, LW, etc.).

  You’re thinking “reason, rationality, deliberation, explicit thought, etc. as opposed to emotion, intuition, implicit thought, etc.” LessWrong-style “rationality” is instead about truth as opposed to falsehood (“epistemic rationality”) and succeeding in your goals as opposed to failing (“instrumental rationality”).

  Classic LW content like What Do We Mean By “Rationality”?, The Straw Vulcan, Feeling Rational, and When (Not) To Use Probabilities talk about how different these concepts are: using deliberative, explicit, quantitative, etc. thought is rational (in the LW sense) if it helps you better understand things or helps you succeed in life, but emotions, hunches, intuitions, etc. are also a crucial part of understanding the world and achieving your goals.

• RobBensinger 26 Feb 2023 10:12 UTC

  4 points

  2

  in reply to: FJehn’s comment on: On Philos­o­phy Tube’s Video on Effec­tive Altruism

  “There is not objective source of truth that tells you that for example quality adjusted life years are the best objective measure.”

  There’s no objective source of truth telling humans to value what we value; on some level it’s just a brute fact that we have certain values. But given a set of values, some metrics will do better vs. worse at describing the values.

  Or in other words: Facts about how much people prefer one thing relative to other things are “subjective” in the weak sense that all psychological facts are subjective: they’re about subjects /​ minds. But psychology facts aren’t “subjective” in a sense like “there are no facts of the matter about minds”. Minds are just as real a part of the world as chairs, electrons, and zebras.

  Consider, for example, a measure that says “a sunburn is ¹⁄₂ as bad as a migraine” versus one that says “a sunburn is a billion times as bad as a migraine”. We can decompose this into a factual claim about the relative preferences of some group of agents, plus a normative claim that calls the things that group dislikes “bad”.

  For practical purposes, the important contribution of welfare metrics isn’t “telling us that the things we dislike are bad”; realists are already happy to run with this, and anti-realists are happy to play along with the basic behavioral take-aways in practice.

  Instead, the important contribution is the factual claim about what a group prefers, which is as objective/​subjective as any other psych claim. Viewed through that lens, even if neither of the claims above is perfectly accurate, it seems clear that the “1/​2 as bad” claim is a lot closer to the psychological truth.

• RobBensinger 14 Feb 2023 17:01 UTC

  4 points

  2

  in reply to: titotal’s comment on: Ele­ments of Ra­tion­al­ist Discourse

  Perhaps the real argument I’m making is that “don’t use weird jargon (outside of lesswrong)” should be another principle.

  Seems like an obviously bad rule to me. “Don’t use weird jargon anywhere in the world except LessWrong” is a way stronger claim than “Don’t use weird jargon in an adversarial debate where you’re trying to rhetorically out-manipulate a dishonest creationist”.

  (This proposal also strikes me as weirdly minor compared to the other rules. Partly because it’s covered to some degree by “Reducibility” already, which encourages people to only use jargon if they’re willing and able to paraphrase it away or explain it on request.)

  “On my model of EA and of the larger world, trying out stuff like this is one of the best ways for EA to increase the probability it has a positive impact.”

  to

  “”I believe that trying out these techniques will probably improve the effectiveness of EA”.”

  Seems like a bad paraphrase to me, in a few ways:

  • “On my model of EA and of the larger world” is actually doing some important work here. The thing I’m trying to concisely gesture at is that I have a ton of complicated background beliefs about the world, and also about how EA should interface with the wider world, that make me much more confident that guidelines like the one in the OP are good ones.
    
    I actually want to signpost all of that pretty clearly, so people know they can follow up and argue with me about the world and about EA if they have different beliefs/​models about how EA can do the most good.

  • “X will probably improve Y” is a lot weaker than “X is one of the best ways to improve Y”.

  • “Improve the effectiveness of EA” is very vague, and (to my eye) makes it sound like I think these guidelines are useful for things like “making EAs more productive at doing the things they’re already trying to do”.
    
    I do think the guidelines would have that effect, but I also think that they’d help people pick better cause areas and interventions to work on, by making people’s reasoning processes and discussions clearer, more substantive, and more cruxy. You could say that this is also increasing our “effectiveness” (especially in EA settings, where “effective” takes on some vague jargoniness of its own), but connotationally it would still be misleading, especially for EAs who are using “effective” in the normal colloquial sense.

  I think overly-jargony, needlessly complicated text is bad. But if “On my model of EA and of the larger world, trying out stuff like this is one of the best ways for EA to increase the probability it has a positive impact.” crosses your bar for “too jargony” and “too complicated”, I think you’re setting your bar waaaay too low for the EA Forum audience.

• RobBensinger 14 Feb 2023 16:43 UTC

  4 points

  4

  in reply to: Guy Raveh’s comment on: Ele­ments of Ra­tion­al­ist Discourse

  My subjective feeling is that all of the terms on this list make conversations less clear, more exhausting, and broadly unpleasant.

  Why? Picking an example that seems especially innocuous to me: why do you feel like the word “probability” (used to refer to degrees of belief strength) makes conversations “less clear”? What are the specific ways you think it makes for more-exhausting or more-unpleasant conversations?

  You could say that’s unsurprising, coming from a person who deliberately avoids LessWrong.

  I think people who dislike LW should also steal useful terms and habits of thought like these, if any seem useful. In general, a pretty core mental motion in my experience is: if someone you dislike does a thing that works, steal that technique from them and get value from it yourself.

  Don’t handicap yourself by cutting out all useful ways of thinking, ideas, arguments, etc. that come from a source you dislike. Say “fuck the source” and then grab whatever’s useful and ditch the rest.

  If the only problem were “this concept is good but I don’t want to use a word that LessWrong uses”, I’d just suggest coming up with a new label for the same concept and using that. (The labels aren’t the important part.)

• RobBensinger 14 Feb 2023 12:07 UTC

  3 points

  6

  in reply to: RobBensinger’s comment on: Ele­ments of Ra­tion­al­ist Discourse

  On my model of EA and of the larger world, trying out stuff like this is one of the best ways for EA to increase the probability it has a positive impact.

  “Adopt weird jargon”, notably, isn’t one of the items on the list.

  I liked Nate’s argument for weird jargon enough that I included it in footnote 6 (while mainly looking for the explanation of “my model is...”), but IMO you can follow all ten items on the list without using any weird jargon. Though giving probabilities to things while trying to be calibrated does inherently have a lot of the properties of jargon: people who are used to “90% confidence” meaning something a lot fuzzier (that turns out to be wrong quite regularly) may be confused initially when they realize that you literally mean there’s a 9-in-10 chance you’re right.

  The jargon from this post I especially think EAs should use routinely is:

  • symmetric vs. asymmetric weapons

  • crux

  • (rationalist) taboo

  • local validity

  • inferential gap /​ inferential distance

  • Ideological Turing Test (ITT)

  • Typical Mind Fallacy

  • terms related to Bayesian probability: probability, priors, retrodict, etc.

  Where some of these (“crux”, “probability”) are common English words, but I’m proposing using them in a narrower and more precise way.

• RobBensinger 14 Feb 2023 12:05 UTC

  9 points

  8

  in reply to: titotal’s comment on: Ele­ments of Ra­tion­al­ist Discourse

  The main disadvantage of rationalist discourse is that it can do pretty badly when talking to non-rationalists.

  I think posts on LessWrong often do badly on that metric, because they aren’t trying to appeal to (e.g.) the voting public. Which seems to me like the right call on LW’s part.

  I think the norms themselves will actually help you do better at communicating with people outside your bubble, because a lot of them are common-sense (but not commonly applied!) ideas for bridging the gap between yourself and people who don’t have the same background as you.

  • Be internally clear about what your primary conversational goals are, and focus on the things that you expect to help most with those goals, rather than getting side-tracked every time a shiny tangent catches your eye.

  • Treat people you’re talking to with respect, err on the side of assuming good faith, and try to understand their perspective.

  • Try to use simple, concrete language.

  • Make your goals and preferences explicit in relatively chill descriptive language, rather than relying on outrage, shaming, etc. to try to pressure people into agreeing with you.

  • Flag when there are ways to test a claim, and consider testing it if it’s easy. (E.g., “oh wait, let’s google that.”)

  • Etc.

  I.e., these norms play well with multicultural, diverse groups of people.

  I do think they’re better for cooperative dynamics than for heavily adversarial ones: if you’re trying to understand the perspective of someone else, have them better understand your perspective, treat the other person like a peer, respect their autonomy and agency, learn more about the world in collaboration with them, etc., then I think these norms are spot-on.

  If you’re trying to manipulate them or sneak ideas past their defenses, then I don’t think these norms are ideal (though I think that’s generally a bad thing to do, and I think EA will do a lot better and be a healthier environment if it moves heavily away from that approach to discourse).

  If you’re interacting with someone else who’s acting adversarially toward you, then I think these norms aren’t bad but they have their emphasis in the wrong place. Like, “Goodwill” leaves room for noticing bad actors and responding accordingly (see footnote 5), but if I were specifically giving people advice for dealing with bad actors, I don’t think any version of “Goodwill” would go on my top ten list of tips and tricks to employ.

  Instead, these norms are aimed at a target more like “a healthy intellectual community that’s trying to collaboratively figure out what’s true (that can also respond well when bad actors show up in its spaces, but that’s more like a top-ten desideratum rather than being the #1 desideratum)”.

  “Trick an audience of laypeople into believing your views faster than a creationist can trick that audience into believing their views” is definitely not what these discourse norms are optimized for helping with, and I think that’s to their credit. Basically zero EAs should be focusing on a goal like that, IMO, and if it did make sense for a rare EA to skill up in that, they definitely shouldn’t import those norms and habits into discussions with other EAs.

Ele­ments of Ra­tion­al­ist Discourse

• RobBensinger 9 Feb 2023 4:47 UTC

  13 points

  10

  in reply to: RobBensinger’s comment on: EA’s weird­ness makes it un­usu­ally sus­cep­ti­ble to bad behavior

  (I’m interested in the counter-argument, and don’t mean to use platitudes to shout down what sounds like a complicated model I don’t yet understand. But I wanted to at least voice my view.

  Normally I would feel less need to speak up and note disagreement, but right now I think a lot of EAs are feeling a lot of emotional conflict and shame about various EA-related things, so I’m unusually wary of pushes for people to cut ties with tons of their friends or partners or radically restructure their life based on a high-level theory about what’s good for them.

  I think this is a good discussion to have, but I want to encourage EAs to be skeptical of contentful one-size-fits-all arguments about what’s good for them, compared to their own individual-specific sense of what’s helping them flourish in life. I trust individuals to build up self-expertise and steer by their taste more than I trust relationship or sociology experts to give useful advice.)

• RobBensinger 9 Feb 2023 4:43 UTC

  8 points

  6

  in reply to: Gordon Seidoh Worley’s comment on: EA’s weird­ness makes it un­usu­ally sus­cep­ti­ble to bad behavior

  As a poly person whose main relationship over the last seven years has been with a non-rationalist non-EA, I want to say:

  Ingroup and outgroup people are both great. I think it would impoverish the community, and be a tragic loss of a lot of beautiful friendships and relationships, if people tried hard to avoid dating anyone from one group or the other.

• RobBensinger 8 Feb 2023 2:05 UTC

  5 points

  0

  in reply to: Sarah Levin’s comment on: The num­ber of burner ac­counts is too damn high

  Oh, that makes total sense to me and I now feel silly for not believing this before I read your comment.

• RobBensinger 7 Feb 2023 17:07 UTC

  23 points

  4

  in reply to: Nathan Young’s comment on: The num­ber of burner ac­counts is too damn high

  I think as this place becomes a place that gets quoted in external publications more it will be harder to speak freely as a named account without a lot of practice, a very thick skin.

  I mean, perhaps easier said than done, but some of my main advice would be: grow a thicker skin, and spend less time anxiously refreshing social media or focusing your attention on low-quality hit pieces.

  EA is prominent enough now that we will in fact get a lot less good done in the world if we devote a day or two to reading and debating every news article that mentions us. (Bad-but-illustrative example: Imagine trying to solve a physics problem while also feeling a panicky impulse to read every news article in the world that mentions physicists.)

  Writing under stable pseudonyms also seems like it captures a lot of the value of writing stably under your real name.

  I have spent so long on twitter that I red team almost everything I say for “being able to be taken out of context”

  That… sounds incredibly unhealthy to me, and makes me think we need better alternatives to Twitter—some platform where it’s easier to just ignore haters and talk to people who are there to do collaborative truth-seeking and good-faith meeting of minds.

  If EAs feel pressure to do that level of red-teaming whenever they post on Twitter, I’d suggest they strongly consider not using Twitter.

• RobBensinger 7 Feb 2023 16:52 UTC

  14 points

  14

  in reply to: sphor’s comment on: My highly per­sonal skep­ti­cism brain­dump on ex­is­ten­tial risk from ar­tifi­cial in­tel­li­gence.

  What does it mean to align AI with human values?

  This article is… really bad.

  It’s mostly a summary of Yudkowsky/​Bostrom ideas, but with a bunch of the ideas garbled and misunderstood.

  Mitchell says that one of the core assumptions of AI risk arguments is “that any goal could be ‘inserted’ by humans into a superintelligent AI agent”. But that’s not true, and in fact a lot of the risk comes from the fact that we have no idea how to’insert’ a goal into an AGI system.

  The paperclip maximizer hypothetical here is a misunderstanding of the original idea. (Though it’s faithful to the version Bostrom gives in Superintelligence.) And the misunderstanding seems to have caused Mitchell to misunderstood a bunch of other things about the alignment problem. Picking one of many examples of just-plain-false claims:

  “And importantly, in keeping with Bostrom’s orthogonality thesis, the machine has achieved superintelligence without having any of its own goals or values, instead waiting for goals to be inserted by humans.”

  The article also says that “research efforts on alignment are underway at universities around the world and at big AI companies such as Google, Meta and OpenAI”. I assume Google here means DeepMind, but what alignment research at Meta does Mitchell have in mind??

  Also: “Many researchers are actively engaged in alignment-based projects, ranging from attempts at imparting principles of moral philosophy to machines, to training large language models on crowdsourced ethical judgments.”

  … That sure is a bad picture of what looks difficult about alignment.

  The implausibility of intelligence explosion

  This essay is quite bad. A response here: A reply to Francois Chollet on intelligence explosion

  I disagree with Thorstad and Drexler, but those resources seem much better.