MikeJohnson

• MikeJohnson 11 Jun 2015 3:08 UTC

  4 points

  0 ∶ 0

  on: I am Nate Soares, AMA!

  Hi Nate,

  Thanks for the AMA. I’m most curious as to what MIRI’s working definition is for what has intrinsic value. The core worry of MIRI has been that it’s easy to get the AI value problem wrong, to build AIs that don’t value the correct thing. But how do we humans get the value problem right? What should we value?

  Max Tegmark alludes to this in Friendly Artificial Intelligence: the Physics Challenge:

  Quantum effects aside, a truly well-defined goal would specify how all particles in our Universe should be arranged at the end of time. [But] what particle arrangement is preferable, anyway? … What is the ultimate ethical imperative, i.e., how should we strive to shape the future of our Universe? If we fail to answer the last question rigorously, this future is unlikely to contain humans.

  So I have two questions: (1) Do you see this (e.g., what Tegmark is speaking about above) as part of MIRI’s bailiwick? (2) If so, do you have any thoughts or research directions you can share publicly?

• MikeJohnson 30 Sep 2015 8:24 UTC

  3 points

  0 ∶ 0

  on: Sys­tem­at­i­cally un­der ex­plored pro­ject ar­eas?

  I’m not sure I have as good of a handle on the broader EA ecosystem as others, so consider my thoughts provisional, but I’d suggest adding

  • A special subset of low-status blindness: there’s a bias toward more conventional projects that are easy to understand, since it’s easier to get affirmation from others if they understand what you’re working on. (Lifted from Jaan Taallinn’s Singularity Summit 2011 talk)

  • I suspect EAs may prefer going down the nonprofit route, which seems very noble, but more overall long-term utility may often be produced by starting a for-profit business. E.g., Elon Musk is one of the most effective EAs on the planet because he did decide to go the capitalist route.

  I’m not sure whether to add basic research stuff or not- the QALY is a pretty creaky foundation, but I grant there’s a lot of uncertainty as to how to improve it.

• MikeJohnson 9 Jan 2016 20:46 UTC

  0 points

  0 ∶ 0

  in reply to: Diego_Caleiro’s comment on: The EA Newslet­ter & Open Thread—Jan­uary 2016

  All of these seem potentially valuable. I suspect the best choice is the one you’ll be most motivated to pursue. My suggestion is that you should consider who your ‘customers’ are for each project, and figure out which group you’d most like to work with and generate deliverables for.

  Also, some of these may lend themselves better to intermediate/​incremental deliverables, which would be a big plus.

  ---

  All of the above is fully general advice- my low-resolution take on your specific situation is that Convergence Analysis seems by far both the highest leverage (and certainly the largest variance), though the fact that you seem unsure whether to dive down that path may imply there may be some difficult hurdles or complications down that path that you’re dreading?

Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

• MikeJohnson 9 Dec 2016 16:32 UTC

  0 points

  0 ∶ 0

  in reply to: MichaelPlant’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Whoops, fixed!

• MikeJohnson 9 Dec 2016 17:20 UTC

  6 points

  0 ∶ 0

  in reply to: Owen Cotton-Barratt’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Thanks for the comment! I think the time-sensitivity of this research is an important claim, as you say.

  My impression of how MIRI currently views CEV is that it’s ‘a useful intuition pump, but not something we should currently plan to depend on for heavy lifting’. In the last MIRI AMA, Rob noted that

  I discussed CEV some in this answer. I think the status is about the same: sounds like a vaguely plausible informal goal to shoot for in the very long run, but also very difficult to implement. As Eliezer notes in https://​​arbital.com/​​p/​​cev/​​, “CEV is rather complicated and meta and hence not intended as something you’d do with the first AI you ever tried to build.” The first AGI systems people develop should probably have much more limited capabilities and much more modest goals, to reduce the probability of catastrophic accidents.

  As an intuition pump, rough sketch, or placeholder, I really like CEV. What I’m worried about is that discussion of CEV generally happens in “far mode”, and there’s very probably work that could and should be done now in order to evaluate how plausible CEV is, and explore alternatives. Four reasons not to depend too much on CEV alone:

  1. CEV is really hard. This seems consistent with what Rob & Eliezer have said.

  2. CEV may not be plausible. A failure mode acknowledged in the original document is that preferences may never cohere—but I would add that CEV may simply be too underdefined & ambiguous to be useful in many cases. E.g., a “preference” is a rather leaky abstraction sometimes to begin with. A lot of possibilities look reasonable from far away, but not from up close, and CEV might be one of these.

  3. CEV may give bad answers. It seems entirely possible that any specific implementation of CEV would unavoidably include certain undesirable systemic biases. More troublingly, maybe preference utilitarianism is just a bad way to go about ethics (I think this is true, personally).

  4. Research into qualia may help us get CEV right. If we define the landscape of consciousness as the landscape within which morally-significant things happen, then understanding this landscape better should help us see how CEV could- or couldn’t- help us navigate it.

  Aside from these CEV-specific concerns, I think research into consciousness & valence could have larger benefits to AI safety- I wrote up some thoughts on this last year at http://​​opentheory.net/​​2015/​​09/​​fai_and_valence/​​ .

  Rather than time-sensitivity, another way to frame this could be path-dependence based on order of technological development. Do we get better average & median futures if we attempt to build AI without worrying much about qualia, or if we work on both at once?

  (Granted, even if this research is all I say it is, there are potential pitfalls of technological development down this path.)

• MikeJohnson 9 Dec 2016 17:50 UTC

  4 points

  0 ∶ 0

  in reply to: turchin’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Yes, it would be quite civilizationally embarrassing to accidently p-zombie ourselves… more generally, it seems valuable to understand tradeoffs in consciousness. This seems to be an important component in any far-future planning.

  Also- Andres has done some interesting exploratory work re: defining the problem of future drug epidemics, and discussing game-theoretic considerations.

• MikeJohnson 10 Dec 2016 5:03 UTC

  3 points

  0 ∶ 0

  in reply to: Jessica_Taylor’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Hi Jessica,

  Thanks for the thoughtful note. I do want to be very clear that I’m not criticizing MIRI’s work on CEV, which I do like very much! - It seems like the best intuition pump & Schelling Point in its area, and I think it has potential to be more.

  My core offering in this space (where I expect most of the value to be) is Principia Qualia- it’s more up-to-date and comprehensive than the blog post you’re referencing. I pose some hypotheticals in the blog post, but it isn’t intended to stand alone as a substantive work (whereas PQ is).

  But I had some thoughts in response to your response on valence + AI safety:

  ->1. First, I agree that leaving our future moral trajectory in the hands of humans is a great thing. I’m definitely not advocating anything else.

  ->2. But I would push back on whether our current ethical theories are very good- i.e., good enough to see us through any future AGI transition without needlessly risking substantial amounts of value.

  To give one example: currently, some people make the claim that animals such as cows are much more capable of suffering than humans, because they don’t have much intellect to blunt their raw, emotional feeling. Other people make the claim that cows are much less capable of suffering than humans, because they don’t have the ‘bootstrapping strange loop’ mind architecture enabled by language, and necessary for consciousness. Worryingly, both of these arguments seem plausible, with no good way to pick between them.

  Now, I don’t think cows are in a strange quantum superposition of both suffering and not suffering— I think there’s a fact of the matter, though we clearly don’t know it.

  This example may have moral implications, but little relevance to existential risk. However, when we start talking about mind simulations and ‘thought crime’, WBE, selfish replicators, and other sorts of tradeoffs where there might be unknown unknowns with respect to moral value, it seems clear to me that these issues will rapidly become much more pressing. So, I absolutely believe work on these topics is important, and quite possibly a matter of survival. (And I think it’s tractable, based on work already done.)

  Based on my understanding, I don’t think Act-based agents or Task AI would help resolve these questions by default, although as tools they could probably help.

  ->3. I also think theories in IIT’s reference class won’t be correct, but I suspect I define the reference class much differently. :) Based on my categorization, I would object to lumping my theory into IIT’s reference class (we could talk more about this if you’d like).

  ->4. Re: suffering computations- a big, interesting question here is whether moral value should be defined at the physical or computational level. I.e., “is moral value made out of quarks or bits (or something else)?” — this may be the crux of our disagreement, since I’m a physicalist and I gather you’re a computationalist. But PQ’s framework allows for bits to be “where the magic happens”, as long as certain conditions obtain.

  One factor that bears mentioning is whether an AGI’s ontology & theory of ethics might be path-dependent upon its creators’ metaphysics in such a way that it would be difficult for it to update if it’s wrong. If this is a plausible concern, this would imply a time-sensitive factor in resolving the philosophical confusion around consciousness, valence, moral value, etc.

  ->5. I wouldn’t advocate strictly hedonic values (this was ambiguous in the blog post but is clearer in Principia Qualia).

  ->6. However, I do think that “how much horrific suffering is there in possible world X?” is a hands-down, qualitatively better proxy for whether it’s a desirable future than “what is the Dow Jones closing price in possible world X?”

  ->7. Re: neuromorphic AIs: I think an interesting angle here is, “how does boredom stop humans from wireheading on pleasurable stimuli?”—I view boredom as a sophisticated anti-wireheading technology. It seems possible (although I can’t vouch for plausible yet) that if we understand the precise mechanism by which boredom is implemented in human brains, it may help us understand and/​or control neuromorphic AGIs better. But this is very speculative, and undeveloped.

• MikeJohnson 10 Dec 2016 5:07 UTC

  4 points

  0 ∶ 0

  in reply to: lukeprog’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Very cool. Are there any takeaways from your interviews that you personally feel strongly about?

  I always like to ask people interested in this stuff, “if you had to say one true thing about consciousness, what would it be?”

• MikeJohnson 10 Dec 2016 9:01 UTC

  2 points

  0 ∶ 0

  in reply to: Jessica_Taylor’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Awesome, I do like your steelman. More thoughts later, but just wanted to share one notion before sleep:

  With regard to computationalism, I think you’ve nailed it. Downward causation seems pretty obviously wrong (and I don’t know of any computationalists that personally endorse it).

  IMO consciousness has power over physics the same way the Python program has power over physics

  Totally agreed, and I like this example.

  it’s consistent to view the system as a physical system without reference to the Python program

  Right- but I would go even further. Namely, given any non-trivial physical system, there exists multiple equally-valid interpretations of what’s going on at the computational level. The example I give in PQ is: let’s say I shake a bag of popcorn. With the right mapping, we could argue that we could treat that physical system as simulating the brain of a sleepy cat. However, given another mapping, we could treat that physical system as simulating the suffering of five holocausts. Very worryingly, we have no principled way to choose between these interpretive mappings. Am I causing suffering by shaking that bag of popcorn?

  And I think all computation is like this, if we look closely- there exists no frame-invariant way to map between computation and physical systems in a principled way… just useful mappings, and non-useful mappings (and ‘useful’ is very frame-dependent).

  This introduces an inconsistency into computationalism, and has some weird implications: I suspect that, given any computational definition of moral value, there would be a way to prove any arbitrary physical system morally superior to any other arbitrary physical system. I.e., you could prove both that A>B, and B>A.

  … I may be getting something wrong here. But it seems like the lack of a clean quarksbits mapping ultimately turns out to be a big deal, and is a big reason why I advocate not trying to define moral value in terms of Turing machines & bitstrings.

  Instead, I tend to think of ethics as “how should we arrange the [quarks|negentropy] in our light-cone?”—ultimately we live in a world of quarks, so ethics is a question of quarks (or strings, or whatnot).

  However! Perhaps this is just a failure of my imagination. What is ethics if not how to arrange our physical world? Or can you help me steelman computationalism against this inconsistency objection?

  Thanks again for the comments. They’re both great and helpful.

• MikeJohnson 10 Dec 2016 20:51 UTC

  1 point

  0 ∶ 0

  in reply to: Jessica_Taylor’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Physical system P embeds computation C if and only if P has different behavior counterfactually on C taking on different values

  I suspect this still runs into the same problem—in the case of the computational-physical mapping, even if we assert that C has changed, we can merely choose a different interpretation of P which is consistent with the change, without actually changing P.

  If it turns out that no simple rule of this form works, I wouldn’t be too troubled, though; I’d be psychologically prepared to accept that there isn’t a clean quarkscomputations mapping. Similar to how I already accept that human value is complex, I could accept that human judgments of “does this physical system implement this computation” are complex (and thus can’t be captured in a simple rule)

  This is an important question: if there exists no clean quarkscomputations mapping, is it (a) a relatively trivial problem, or (b) a really enormous problem? I’d say the answer to this depends on how we talk about computations. I.e., if we say “the ethically-relevant stuff happens at the computational level”—e.g., we shouldn’t compute certain strings—then I think it grows to be a large problem. This grows particularly large if we’re discussing how to optimize the universe! :)

  I think these are all useful ways of viewing ethics, and I don’t feel the need to pick a single view (although I often find it appealing to look at what some views say about what other views are saying and resolving the contradictions between them). There are all kinds of reasons why it might be psychologically uncomfortable not to have a simple theory of ethics (e.g. it’s harder to know whether you’re being ethical, it’s harder to criticize others for being unethical, it’s harder for groups to coordinate around more complex and ambiguous ethical theories, you’ll never be able to “solve” ethics once and then never have to think about ethics again, it requires holding multiple contradictory views in your head at once, you won’t always have a satisfying verbal justification for why your actions are ethical). But none of this implies that it’s good (in any of the senses above!) to assume there’s a simple ethical theory.

  Let me push back a little here- imagine we live in the early 1800s, and Faraday was attempting to formalize electromagnetism. We had plenty of intuitive rules of thumb for how electromagnetism worked, but no consistent, overarching theory. I’m sure lots of people shook their head and said things like, “these things are just God’s will, there’s no pattern to be found.” However, it turns out that there was something unifying to be found, and tolerance of inconsistencies & nebulosity would have been counter-productive.

  Today, we have intuitive rules of thumb for how we think consciousness & ethics work, but similarly no consistent, overarching theory. Are consciousness & moral value like electromagnetism—things that we can discover knowledge about? Or are they like elan vital—reifications of clusters of phenomena that don’t always cluster cleanly?

  I think the jury’s still out here, but the key with electromagnetism was that Faraday was able to generate novel, falsifiable predictions with his theory. I’m not claiming to be Faraday, but I think if we can generate novel, falsifiable predictions with work on consciousness & valence (I offer some in Section XI, and observations that could be adapted to make falsifiable predictions in Section XII), this should drive updates toward “there’s some undiscovered cache of predictive utility here, similar to what Faraday found with electromagnetism.”

• MikeJohnson 11 Dec 2016 15:43 UTC

  1 point

  0 ∶ 0

  in reply to: Jessica_Taylor’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  It seems like you’re saying here that there won’t be clean rules for determining logical counterfactuals? I agree this might be the case but it doesn’t seem clear to me. Logical counterfactuals seem pretty confusing and there seems to be a lot of room for better theories about them.

  Right, and I would argue that logical counterfactuals (in the way we’ve mentioned them in this thread) will necessarily be intractably confusing, because they’re impossible to do cleanly. I say this because in the “P & C” example above, we need a frame-invariant way to interpret a change in C in terms of P. However, we can only have such a frame-invariant way if there exists a clean mapping (injection, surjection, bijection, etc) between P&C- which I think we can’t have, even theoretically.

  (Unless we define both physics and computation through something like constructor theory… at which point we’re not really talking about Turing machines as we know them—we’d be talking about physics by another name.)

  This is a big part of the reason why I’m a big advocate of trying to define moral value in physical terms: if we start with physics, then we know our conclusions will ‘compile’ to physics. If instead we start with the notion that ‘some computations have more moral value than others’, we’re stuck with the problem—intractable problem, I argue—that we don’t have a frame-invariant way to precisely identify what computations are happening in any physical system (and likewise, which aren’t happening). I.e., statements about computations will never cleanly compile to physical terms. And whenever we have multiple incompatible interpretations, we necessarily get inconsistencies, and we can prove anything is true (i.e., we can prove any arbitrary physical system is superior to any other).

  Does that argument make sense?

  … that said, it would seem very valuable to make a survey of possible levels of abstraction at which one could attempt to define moral value, and their positives & negatives.

  I think we have a lot more theoretical progress to make on understanding consciousness and ethics. On priors I’d expect the theoretical progress to produce more-satisfying things over time without ever producing a complete answer to ethics. Though of course I could be wrong here; it seems like intuitions vary a lot. It seems more likely to me that we find a simple unifying theory for consciousness than ethics.

  Totally agreed!

• MikeJohnson 13 Dec 2016 6:53 UTC

  0 points

  0 ∶ 0

  in reply to: Jessica_Taylor’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  What you’re saying seems very reasonable; I don’t think we differ on any facts, but we do have some divergent intuitions on implications.

  I suspect this question—whether it’s possible to offer a computational description of moral value that could cleanly ‘compile’ to physics—would have non-trivial yet also fairly modest implications for most of MIRI’s current work.

  I would expect the significance of this question to go up over time, both in terms of direct work MIRI expects to do, and in terms of MIRI’s ability to strategically collaborate with other organizations. I.e., when things shift from “let’s build alignable AGI” to “let’s align the AGI”, it would be very good to have some of this metaphysical fog cleared away so that people could get on the same ethical page, and see that they are in fact on the same page.

• MikeJohnson 15 Dec 2016 3:59 UTC

  0 points

  0 ∶ 0

  in reply to: AlexMennen’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  As I understand their position, MIRI tends to not like IIT because it’s insufficiently functionalist—and too physicalist. On the other hand, I don’t think IIT could be correct because it’s too functionalist—and insufficiently physicalist, partially for the reasons I explain in my response to Jessica.

  The core approach I’ve taken is to enumerate the sorts of problems one would need to solve if one was to formalize consciousness. (Whether consciousness is a thing-that-can-be-formalized is another question, of course.) My analysis is that IIT satisfactorily addresses 4 or 5, out of the 8 problems. Moving to a more physical basis would address more of these problems, though not all (a big topic in PQ is how to interpret IIT-like output, which is an independent task of how to generate it).

  Other research along these same lines would be e.g.,

  ->Adam Barrett’s FIIH: https://​​www.ncbi.nlm.nih.gov/​​pmc/​​articles/​​PMC3912322/​​

  ->Max Tegmark’s Perceptronium: https://​​arxiv.org/​​abs/​​1401.1219

• MikeJohnson 15 Dec 2016 4:57 UTC

  1 point

  0 ∶ 0

  in reply to: CarlShulman’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Hi Carl, thanks for your thoughts & time. I appreciate the comments.

  First, to be clear, the hypothesis is that the symmetry of the mathematical object isomorphic to a conscious experience corresponds to valence. This is distinct from (although related to) the symmetry of a stimulus, or even symmetry within brain networks.

  This does not look like a metric that is tightly connected to sensory, cognitive, or behavioral features. In particular, it is not specifically connected to liking, wanting, aversion, and so forth. So, like IIT in the cases discussed by Scott Aaronson, it would seem likely to assign huge values (of valence rather than consciousness, in this case) to systems that lack the corresponding functions, and very low values to systems that possess them.

  I strongly disagree with this in the case of humans, fairly strongly disagree in the more general case of evolved systems, and mildly disagree in the fully general case of arbitrary systems.

  First, it seems extremely like to me that evolved organisms would use symmetry as an organizational principle /​ attractor (Section XII);

  Second, in cases where we do have some relevant data or plausible models (I.e., as noted in Sections IX and XII), the symmetry hypothesis seems plausible. I think the hypothesis does really well when one actually looks at the object-level, particularly e.g., Safron’s model of orgasm & Seth and Friston’s model of interoception;

  Third, with respect to extending Aaronson’s critique, I question whether “this seems to give weird results when put in novel contexts” is a good path to take. As Eric Schwitzgebel notes, “Common sense is incoherent in matters of metaphysics. There’s no way to develop an ambitious, broad-ranging, self- consistent metaphysical system without doing serious violence to common sense somewhere. It’s just impossible. Since common sense is an inconsistent system, you can’t respect it all. Every metaphysician will have to violate it somewhere.” This seems particularly true in the realm of consciousness, and particularly true in contexts where there was no evolutionary benefit in having correct intuitions.

  As such it seems important not to enshrine common sense, with all its inconsistencies, as the gold standard with regard to valence research. In general, I’d say a good sign of a terrible model of consciousness would be that it validates all of our common-sense intuitions about the topic.

  The falsifiable predictions are mostly claims that the computational functions will be (imperfectly) correlated with symmetry, but the treatment of boredom appears to allow that these will be quite imperfect:

  Section XI is intended as the core set of falsifiable predictions—you may be thinking of the ‘implications for neuroscience’ discussion in Section XII, some of which could be extended to become falsifiable predictions.

  Overall, this seems systematically analogous to IIT in its flaws. If one wanted to pursue an analogy to Aaronson’s discussion of trivial expander graphs producing extreme super-consciousness, one could create an RL agent (perhaps in an artificial environment where it has the power to smile, seek out rewards, avoid injuries (which trigger negative reward), favor injured limbs, and consume painkillers (which stop injuries from generating negative reward) whose symmetry could be measured in whatever way the author would like to specify. I think we can say now that we could program the agent in such a way that it sought out things that resulted in either more or less symmetric states, or was neutral to such things. Likewise, switching the signs of rewards would not reliably switch the associated symmetry. And its symmetry could be directly and greatly altered without systematic matching behavioral changes. I would like to know whether the theory in PQ is supposed to predict that such agents couldn’t be built without extraordinary efforts, or that they would have systematic mismatch of their functional beliefs and behavior regarding qualia with actual qualia.

  I’d assert- very strongly- that one could not evolve such a suffering-seeking agent without extraordinary effort, and that if one was to attempt to build one from scratch, it would be orders of magnitude more difficult to do so than making a “normal” agent. (This follows from my reasoning in Section XII.) But let’s keep in mind that whether the agent you’re speaking of is a computational program or a physical system matters a lot—under my model, a RL agent running on a standard Von Neumann physical architecture probably has small & merely fragmentary qualia.

  An analogy here would be the orthogonality thesis: perhaps we can call this “valence orthogonality”: the behavior of a system, and its valence, are usually tightly linked via evolutionary processes and optimization factors, but they are not directly causally coupled, just as intelligence & goals are not causally coupled.

  This hypothesis does also have implications for the qualia of whole-brain emulations, which perhaps is closer to your thought-experiment.

• MikeJohnson 15 Dec 2016 5:25 UTC

  0 points

  0 ∶ 0

  in reply to: Jessica_Taylor’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Yes, the epistemological challenges with distinguishing between ground-truth qualia and qualia reports are worrying. However, I don’t think they’re completely intractable, because there is a causal chain (from Appendix C):

  Our brain’s physical microstates (perfectly correlated with qualia) --> The logical states of our brain’s self-model (systematically correlated with our brain’s physical microstates) --> Our reports about our qualia (systematically correlated with our brain’s model of its internal state)

  .. but there could be substantial blindspots, especially in contexts where there was no adaptive benefit to having accurate systematic correlations.

• MikeJohnson 15 Dec 2016 5:37 UTC

  0 points

  0 ∶ 0

  in reply to: Jessica_Taylor’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  I generally agree with this—getting it right eventually is the most important thing; getting it wrong for 100 years could be horrific, but not an x-risk.

  I do worry some that “trusted reflection process” is a sufficiently high-level abstraction so as to be difficult to critique.

  Interesting piece by Christiano, thanks! I would also point to a remark I made above, that doing this sort of ethical clarification now (if indeed it’s tractable) will pay dividends in aiding coordination between organizations such as MIRI, DeepMind, etc. Or rather, by not clarifying goals, consciousness, moral value, etc, it seems likely to increase risks of racing to be the first to develop AGI, secrecy & distrust between organizations, and such.

  A lot does depend on tractability.

• MikeJohnson 15 Dec 2016 5:52 UTC

  2 points

  0 ∶ 0

  in reply to: WhySpace’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Thank you!

  I’d certainly like to encourage more work in this space, as long as we can approach it with rigor.

  Traditionally, both consciousness research and ethics research have been black holes—smart people go in, but nothing comes out. But I think that’s changing, as a few paths to systematically approach these topics open up. But by default, these paths won’t get taken, because there are no institutions pushing them.

  Re: preference-based ethics vs qualia-based ethics, here’s a piece by Andres Gomez Emilsson that makes the case that although we think we’re optimizing preferences, often we’re actually optimizing qualia: https://​​qualiacomputing.com/​​2016/​​11/​​19/​​the-tyranny-of-the-intentional-object/​​

• MikeJohnson 20 Dec 2016 19:04 UTC

  0 points

  0 ∶ 0

  in reply to: Qiaochu_Yuan’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  Thanks!

  On the value question, what do you think has value? Can we define any necessary or sufficient conditions?

  Perhaps more significantly, at which level(s) of abstraction should value be defined?

• MikeJohnson 9 Jan 2017 22:39 UTC

  0 points

  0 ∶ 0

  in reply to: bcforstadt’s comment on: Prin­cipia Qualia: blueprint for a new cause area, con­scious­ness re­search with an eye to­ward ethics and x-risk

  This seems reasonable; I address this partially in Appendix C, although not comprehensively.

  For me, the fact that ethics seems to exist is an argument for some sort of consciousness&value realism. I fear that Type-A physicalists have no principled basis for saying any use of quarks (say, me having a nice drink of water when I’m thirsty) is better than any other use of quarks (a cat being set on fire). I.e., according to Type-A physicalists this would be a verbal dispute, without an objectively correct answer, so it wouldn’t be ‘wrong’ to take either side. This seems to embody an unnecessarily extreme and unhelpful amount of skepticism to me.

  Do you know of any Type-A physicalist who has tried to objectively ground morality?