Hi Harrison, appreciate the remarks. My response would be more-or-less an open-ended question: do you feel this is a valid scientific mystery? And, what do you feel an answer would/should look like? I.e., correct answers to long-unsolved mysteries might tend to be on the weird side, but there’s “useful generative clever weird” and “bad wrong crazy timecube weird”. How would you tell the difference?
MikeJohnson
Karma: 637
Hi Abby, I‘m happy to entertain well-meaning criticism, but it feels your comment rests fairly heavily on credentialism and does not seem to offer any positive information, nor does it feel like high-level criticism (“their actual theory is also bad”). If your background is as you claim, I’m sure you understand the nuances of “proving” an idea in neuroscience, especially with regard to NCCs (neural correlates of consciousness) — neuroscience is also large enough that “I published a peer reviewed fMRI paper in a mainstream journal” isn’t a particularly ringing endorsement of domain knowledge in affective neuroscience. If you do have domain knowledge sufficient to take a crack at the question of valence I’d be glad to hear your ideas.
For a bit of background to theories of valence in neuroscience I’d recommend my forum post here—it goes significantly deeper into the literature than this primer.
Again, I’m not certain you read my piece closely, but as mentioned in my summary, most of our collaboration with British universities has been with Imperial (Robin Carhart-Harris’s lab, though he recently moved to UCSF) rather than Oxford, although Kringelbach has a great research center there and Atasoy (creator of the CSHW reference implementation, which we independently reimplemented) does her research there, so we’re familiar with the scene.
A Primer on the Symmetry Theory of Valence
I like this theme a lot!
In looking at longest-term scenarios, I suspect there might be useful structure&constraints available if we take seriously the idea that consciousness is a likely optimization target of sufficiently intelligent civilizations. I offered the following on Robin Hanson’s blog:
Premise 1: Eventually, civilizations progress until they can engage in megascale engineering: Dyson spheres, etc.
Premise 2: Consciousness is the home of value: Disneyland with no children is valueless.
Premise 2.1: Over the long term we should expect at least some civilizations to fall into the attractor of treating consciousness as their intrinsic optimization target.Premise 3: There will be convergence that some qualia are intrinsically valuable, and what sorts of qualia are such.
Conjecture: A key piece of evidence for discerning the presence of advanced alien civilizations will be megascale objects which optimize for the production of intrinsically valuable qualia.
--
Essentially: I think formal consciousness research could generate a new heuristic for both how to parse cosmological data for intelligent civilizations, and what longest-term future humanity may choose for itself.
Physicalism seems plausible, and the formulation of physicalism I most believe in (dual-aspect monism) has physics and phenomenology as two sides of the same coin. As Tegmark notes, “humans … aren’t the optimal solution to any well-defined physics problem.” Similarly, humans aren’t the optimal solution to any well-defined phenomenological problem.
I can’t say I know for sure we’ll settle on filling the universe with such an “optimal solution”, nor would I advocate anything at this point, but if we’re looking for starting threads for how to conceptualize the longest-term optimization targets of humanity, a little consciousness research might go a long way.
More:
https://opentheory.net/2019/09/whats-out-there/
https://opentheory.net/2019/06/taking-monism-seriously/
https://opentheory.net/2019/02/simulation-argument/
Hi Daniel,
Thanks for the reply! I am a bit surprised at this:
Getting more clarity on emotional valence does not seem particularly high-leverage to me. What’s the argument that it is?
The quippy version is that, if we’re EAs trying to maximize utility, and we don’t have a good understanding of what utility is, more clarity on such concepts seems obviously insanely high-leverage. I’ve written about specific relevant to FAI here: https://opentheory.net/2015/09/fai_and_valence/ Relevance to building a better QALY here: https://opentheory.net/2015/06/effective-altruism-and-building-a-better-qaly/ And I discuss object-level considerations on how better understanding of emotional valence could lead to novel therapies for well-being here: https://opentheory.net/2018/08/a-future-for-neuroscience/ https://opentheory.net/2019/11/neural-annealing-toward-a-neural-theory-of-everything/ On mobile, pardon the formatting.
Your points about sufficiently advanced AIs obsoleting human philosophers are well-taken, though I would touch back on my concern that we won’t have particular clarity on philosophical path-dependencies in AI development without doing some of the initial work ourselves, and these questions could end up being incredibly significant for our long-term trajectory — I gave a talk about this for MCS that I’ll try to get transcribed (in the meantime I can share my slides if you’re interested). I’d also be curious to flip your criticism and ping your models for a positive model for directing EA donations — is the implication that there are no good places to donate to, or that narrow-sense AI safety is the only useful place for donations? What do you think the highest-leverage questions to work on are? And how big are your ‘metaphysical uncertainty error bars’? What sorts of work would shrink these bars?
Hi Daniel,
Thanks for the remarks! Prioritization reasoning can get complicated, but to your first concern:
Is emotional valence a particularly confused and particularly high-leverage topic, and one that might plausibly be particularly conductive getting clarity on? I think it would be hard to argue in the negative on the first two questions. Resolving the third question might be harder, but I’d point to our outputs and increasing momentum. I.e. one can levy your skepticism on literally any cause, and I think we hold up excellently in a relative sense. We may have to jump to the object-level to say more.
To your second concern, I think a lot about AI and ‘order of operations’. Could we postulate that some future superintelligence might be better equipped to research consciousness than we mere mortals? Certainly. But might there be path-dependencies here such that the best futures happen if we gain more clarity on consciousness, emotional valence, the human nervous system, the nature of human preferences, and so on, before we reach certain critical thresholds in superintelligence development and capacity? Also — certainly.
Widening the lens a bit, qualia research is many things, and one of these things is an investment in the human-improvement ecosystem, which I think is a lot harder to invest effectively in (yet also arguably more default-safe) than the AI improvement ecosystem. Another ‘thing’ qualia research can be thought of as being is an investment in Schelling point exploration, and this is a particularly valuable thing for AI coordination.
I’m confident that, even if we grant that the majority of humanity’s future trajectory will be determined by AGI trajectory — which seems plausible to me — I think it’s also reasonable to argue that qualia research is one of the highest-leverage areas for positively influencing AGI trajectory and/or the overall AGI safety landscape.
Congratulations on the book! I think long works are surprisingly difficult and valuable (both to author and reader) and I’m really happy to see this.
My intuition on why there’s little discussion of core values is a combination of “a certain value system [is] tacitly assumed” and “we avoid discussing it because … discussing values is considered uncooperative.” To wit, most people in this sphere are computationalists, and the people here who have thought the most about this realize that computationalism inherently denies the possibility of any ‘satisfyingly objective’ definition of core values (and suffering). Thus it’s seen as a bit of a faux pas to dig at this—the tacit assumption is, the more digging that is done, the less ground for cooperation there will be. (I believe this stance is unnecessarily cynical about the possibility of a formalism.)
I look forward to digging into the book. From a skim, I would just say I strongly agree about the badness of extreme suffering; when times are good we often forget just how bad things can be. A couple quick questions in the meantime:
If you could change peoples’ minds on one thing, what would it be? I.e. what do you find the most frustrating/pernicious/widespread mistake on this topic?
One intuition pump I like to use is: ‘if you were given 10 billion dollars and 10 years to move your field forward, how precisely would you allocate it, and what do you think you could achieve at the end?’
A core ‘hole’ here is metrics for malevolence (and related traits) visible to present-day or near-future neuroimaging.
Briefly—Qualia Research Institute’s work around connectome-specific harmonic waves (CSHW) suggests a couple angles:
(1) proxying malevolence via the degree to which the consonance/harmony in your brain is correlated with the dissonance in nearby brains;
(2) proxying empathy (lack of psychopathy) by the degree to which your CSHWs show integration/coupling with the CSHWs around you.Both of these analyses could be done today, given sufficient resource investment. We have all the algorithms and in-house expertise.
Background about the paradigm: https://opentheory.net/2018/08/a-future-for-neuroscience/
Very important topic! I touch on McCabe’s work in Against Functionalism (EA forum discussion); I hope this thread gets more airtime in EA, since it seems like a crucial consideration for long-term planning.
Hey Pablo! I think Andres has a few up on Metaculus; I just posted QRI’s latest piece of neuroscience here, which has a bunch of predictions (though I haven’t separated them out from the text):
https://opentheory.net/2019/11/neural-annealing-toward-a-neural-theory-of-everything/
We’ve looked for someone from the community to do a solid ‘adversarial review’ of our work, but we haven’t found anyone that feels qualified to do so and that we trust to do a good job, aside from Scott, and he’s not available at this time. If anyone comes to mind do let me know!
I think this is a great description. “What happens if we seek out symmetry gradients in brain networks, but STV isn’t true?” is something we’ve considered, and determining ground-truth is definitely tricky. I refer to this scenario as the “Symmetry Theory of Homeostatic Regulation”—https://opentheory.net/2017/05/why-we-seek-out-pleasure-the-symmetry-theory-of-homeostatic-regulation/ (mostly worth looking at the title image, no need to read the post)
I’m (hopefully) about a week away from releasing an update to some of the things we discussed in Boston, basically a unification of Friston/Carhart-Harris’s work on FEP/REBUS with Atasoy’s work on CSHW—will be glad to get your thoughts when it’s posted.
I think we actually mostly agree: QRI doesn’t ‘need’ you to believe qualia are real, that symmetry in some formalism of qualia corresponds to pleasure, that there is any formalism about qualia to be found at all. If we find some cool predictions, you can strip out any mention of qualia from them, and use them within the functionalism frame. As you say, the existence of some cool predictions won’t force you to update your metaphysics (your understanding of which things are ontologically ‘first class objects’).
But- you won’t be able to copy our generator by doing that, the thing that created those novel predictions, and I think that’s significant, and gets into questions of elegance metrics and philosophy of science.
I actually think the electromagnetism analogy is a good one: skepticism is always defensible, and in 1600, 1700, 1800, 1862, and 2018, people could be skeptical of whether there’s ‘deep unifying structure’ behind these things we call static, lightning, magnetism, shocks, and so on. But it was much more reasonable to be skeptical in 1600 than in 1862 (the year Maxwell’s Equations were published), and more reasonable in 1862 than it was in 2018 (the era of the iPhone).
Whether there is ‘deep structure’ in qualia is of course an open question in 2019. I might suggest STV is equivalent to a very early draft of Maxwell’s Equations: not a full systematization of qualia, but something that can be tested and built on in order to get there. And one that potentially ties together many disparate observations into a unified frame, and offers novel / falsifiable predictions (which seem incredibly worth trying to falsify!)
I’d definitely push back on the frame of dualism, although this might be a terminology nitpick: my preferred frame here is monism: https://opentheory.net/2019/06/taking-monism-seriously/ - and perhaps this somewhat addresses your objection that ‘QRI posits the existence of too many things’.
Thanks Matthew! I agree issues of epistemology and metaphysics get very sticky very quickly when speaking of consciousness.
My basic approach is ‘never argue metaphysics when you can argue physics’—the core strategy we have for ‘proving’ we can mathematically model qualia is to make better and more elegant predictions using our frameworks, with predicting pain/pleasure from fMRI data as the pilot project.
One way to frame this is that at various points in time, it was completely reasonable to be a skeptic about modeling things like lightning, static, magnetic lodestones, and such, mathematically. This is true to an extent even after Faraday and Maxwell formalized things. But over time, with more and more unusual predictions and fantastic inventions built around electromagnetic theory, it became less reasonable to be skeptical of such.
My metaphysical arguments are in my ‘Against Functionalism’ piece, and to date I don’t believe any commenters have addressed my core claims:
But, I think metaphysical arguments change distressingly few peoples’ minds. Experiments and especially technology changes peoples’ minds. So that’s what our limited optimization energy is pointed at right now.
QRI is tackling a very difficult problem, as is MIRI. It took many, many years for MIRI to gather external markers of legitimacy. My inside view is that QRI is on the path of gaining said markers; for people paying attention to what we’re doing, I think there’s enough of a vector right now to judge us positively. I think these markers will be obvious from the ‘outside view’ within a short number of years.
But even without these markers, I’d poke at your position from a couple angles:
I. Object-level criticism is best
First, I don’t see evidence you’ve engaged with our work beyond very simple pattern-matching. You note that “I also think that I’m somewhat qualified to assess QRI’s work (as someone who’s spent ~100 paid hours thinking about philosophy of mind in the last few years), and when I look at it, I think it looks pretty crankish and wrong.” But *what* looks wrong? Obviously doing something new will pattern-match to crankish, regardless of whether it is crankish, so in terms of your rationale-as-stated, I don’t put too much stock in your pattern detection (and perhaps you shouldn’t either). If we want to avoid accidentally falling into (1) ‘negative-sum status attack’ interactions, and/or (2) hypercriticism of any fundamentally new thing, neither of which is good for QRI, for MIRI, or for community epistemology, object-level criticisms (and having calibrated distaste for low-information criticisms) seem pretty necessary.
Also, we do a lot more things than just philosophy, and we try to keep our assumptions about the Symmetry Theory of Valence separate from our neuroscience—STV can be wrong and our neuroscience can still be correct/useful. That said, empirically the neuroscience often does ‘lead back to’ STV.
Some things I’d offer for critique:
https://opentheory.net/2018/08/a-future-for-neuroscience/#
https://opentheory.net/2018/12/the-neuroscience-of-meditation/
https://www.qualiaresearchinstitute.org/research-lineages
(you can also watch our introductory video for context, and perhaps a ‘marker of legitimacy’, although it makes very few claims https://www.youtube.com/watch?v=HetKzjOJoy8 )
I’d also suggest that the current state of philosophy, and especially philosophy of mind and ethics, is very dismal. I give my causal reasons for this here: https://opentheory.net/2017/10/rescuing-philosophy/ - I’m not sure if you’re anchored to existing theories in philosophy of mind being reasonable or not.
II. What’s the alternative?
If there’s one piece I would suggest engaging with, it’s my post arguing against functionalism. I think your comments presuppose functionalism is reasonable and/or the only possible approach, and the efforts QRI is putting into building an alternative are certainly wasted. I strongly disagree with this; as I noted in my Facebook reply,
>Philosophically speaking, people put forth analytic functionalism as a theory of consciousness (and implicitly a theory of valence?), but I don’t think it works *qua* a theory of consciousness (or ethics or value or valence), as I lay out here: https://forum.effectivealtruism.org/.../why-i-think-the...-- This is more-or-less an answer to some of Brian Tomasik’s (very courageous) work, and to sum up my understanding I don’t think anyone has made or seems likely to make ‘near mode’ progress, e.g. especially of the sort that would be helpful for AI safety, under the assumption of analytic functionalism.
----------
I always find in-person interactions more amicable & high-bandwidth—I’ll be back in the Bay early December, so if you want to give this piece a careful read and sit down to discuss it I’d be glad to join you. I think it could have significant implications for some of MIRI’s work.
Thanks, added.
Buck- for an internal counterpoint you may want to discuss QRI’s research with Vaniver. We had a good chat about what we’re doing at the Boston SSC meetup, and Romeo attended a MIRI retreat earlier in the summer and had some good conversations with him there also.
To put a bit of a point on this, I find the “crank philosophy” frame a bit questionable if you’re using only thin-slice outside view and not following what we’re doing. Probably, one could use similar heuristics to pattern-match MIRI as “crank philosophy” also (probably, many people have already done exactly this to MIRI, unfortunately).
We’re pretty up-front about our empirical predictions; if critics would like to publicly bet against us we’d welcome this, as long as it doesn’t take much time away from our research. If you figure out a bet we’ll decide whether to accept it or reject it, and if we reject it we’ll aim to concisely explain why.
For a fuller context, here is my reply to Buck’s skepticism about the 80% number during our back-and-forth on Facebook—as a specific comment, the number is loosely held, more of a conversation-starter than anything else. As a general comment I’m skeptical of publicly passing judgment on my judgment based on one offhand (and unanswered- it was not engaged with) comment on Facebook. Happy to discuss details in a context we’ll actually talk to each other. :)
--------------my reply from the Facebook thread a few weeks back--------------
I think the probability question is an interesting one—one frame is asking what is the leading alternative to STV?
At its core, STV assumes that if we have a mathematical representation of an experience, the symmetry of this object will correspond to how pleasant the experience is. The latest addition to this (what we’re calling ‘CDNS’) assumes that consonance under Selen Atasoy’s harmonic analysis of brain activity (connectome-specific harmonic waves, CSHW) is a good proxy for this in humans. This makes relatively clear predictions across all human states and could fairly easily be extended to non-human animals, including insects (anything we can infer a connectome for, and the energy distribution for the harmonics of the connectome). So generally speaking we should be able to gather a clear signal as to whether the evidence points this way or not (pending resources to gather this data- we’re on a shoestring budget).
Empirically speaking, the competition doesn’t seem very strong. As I understand it, currently the gold standard for estimating self-reports of emotional valence via fMRI uses regional activity correlations, and explains ~16% of the variance. Based on informal internal estimations looking at coherence within EEG bands during peak states, I’d expect us to do muuuuch better.
Philosophically speaking, people put forth analytic functionalism as a theory of consciousness (and implicitly a theory of valence?), but I don’t think it works *qua* a theory of consciousness (or ethics or value or valence), as I lay out here: https://forum.effectivealtruism.org/.../why-i-think-the...-- This is more-or-less an answer to some of Brian Tomasik’s (very courageous) work, and to sum up my understanding I don’t think anyone has made or seems likely to make ‘near mode’ progress, e.g. especially of the sort that would be helpful for AI safety, under the assumption of analytic functionalism.
So in short, I think STV is perhaps the only option that is well-enough laid out, philosophically and empirically, to even be tested, to even be falsifiable. That doesn’t mean it’s true, but my prior is it’s ridiculously worthwhile to try to falsify, and it seems to me a massive failure of the EA and x-risk scene that resources are not being shifted toward this sort of inquiry. The 80% I gave was perhaps a bit glib, but to dig a little, I’d say I’d give at least an 80% chance of ‘Qualia Formalism’ being true, and given that, a 95% chance of STV being true, and a 70% chance of CDNS+CSHW being a good proxy for the mathematical symmetry of human experiences.
An obvious thing we’re lacking is resources; a non-obvious thing we’re lacking is good critics. If you find me too confident I’d be glad to hear why. :)
Resources:
Principia Qualia: https://opentheory.net/PrincipiaQualia.pdf(exploratory arguments for formalism and STV laid out)
Against Functionalism: https://forum.effectivealtruism.org/.../why-i-think-the...
(an evaluation of what analytic functionalism actually gives us)
Quantifying Bliss: https://qualiacomputing.com/.../quantifying-bliss-talk.../
(Andres Gomez Emilsson’s combination of STV plus Selen Atasoy’s CSHW, which forms the new synthesis we’re working from)
A Future for Neuroscience: https://opentheory.net/2018/08/a-future-for-neuroscience/#
(more on CSHW)Happy to chat more in-depth about details.
Hi Jpmos, really appreciate the comments. To address the question of evidence, this is a fairly difficult epistemological situation but we’re working with high-valence datasets from Daniel Ingram & Harvard, and Imperial College London (jhana data, and MDMA data, respectively) and looking for signatures of high harmony.
Neuroimaging is a pretty messy thing, there are no shortcuts to denoising data, and we are highly funding constrained, so I’m afraid we don’t have any peer-reviewed work published on this yet. I can say that initial results seem fairly promising and we hope to have something under review in 6 months. There is a wide range of tacit evidence that stimulation patterns with higher internal harmony produce higher valence than dissonant patterns (basically: music feels good, nails on a chalkboard feels bad), but this is in a sense ‘obvious’ and only circumstantial evidence for STV.
Happy to ‘talk shop’ if you want to dig into details here.