MikhailSamin

Karma: 497

Are you interested in AI X-risk reduction and strategies? Do you have experience in comms or policy? Let’s chat!

aigsi.org develops educational materials and ads that most efficiently communicate core AI safety ideas to specific demographics, with a focus on producing a correct understanding of why smarter-than-human AI poses a risk of extinction. We plan to increase and leverage understanding of AI and existential risk from AI to impact the chance of institutions addressing x-risk.

Early results include ads that achieve a cost of $0.10 per click (to a website that explains the technical details of why AI experts are worried about extinction risk from AI) and $0.05 per engagement on ads that share simple ideas at the core of the problem.

Personally, I’m good at explaining existential risk from AI to people, including to policymakers. I have experience of changing minds of ³⁄₄ people I talked to at an e/acc event.

Previously, I got 250k people to read HPMOR and sent 1.3k copies to winners of math and computer science competitions (including dozens of IMO and IOI gold medalists); have taken the GWWC pledge; created a small startup that donated >100k$ to effective nonprofits.

I have a background in ML and strong intuitions about the AI alignment problem. I grew up running political campaigns and have a bit of a security mindset.

My website: contact.ms

You’re welcome to schedule a call with me before or after the conference: contact.ms/ea30

MikhailSamin Feb 14, 2024, 12:20 AM
6 points
0 ∶ 0
in reply to: Holly Elmore ⏸️ 🔸’s comment on: EA community needs mechanisms to avoid deceptive messaging and deontologically bad plans
Left a comment there, but to repeat:
because I made an error in my original press release
This was, indeed, an honest mistake that you corrected. It is not at all what I’m talking about in the post. I want the community to pay attention to the final messaging being deceptive about the nature of OpenAI’s relationship with the Pentagon.
I am extremely saddened by the emotional impact this has on you. I did not wish that to happen and was surprised and confused by your reaction. Unfortunately, it seems that you still don’t understand the issue I’m pointing at; it is not the “charter” being in the original announcement, it’s the final wording misleading people.

MikhailSamin Feb 14, 2024, 12:18 AM
7 points
3 ∶ 1
in reply to: Holly Elmore ⏸️ 🔸’s comment on: Holly_Elmore’s Shortform
See my post here.
because I made an error in my original press release
This was an honest mistake that you corrected. It is not at all what I’m talking about in the post. I want the community to pay attention to the final messaging being deceptive about the nature of OpenAI’s relationship with the Pentagon. The TL;DR of my post:
Good-hearted EAs lack the mechanisms to not output information that can mislead people. They organised a protest, with the messaging centred on OpenAI changing their documents to “work with the Pentagon”, while OpenAI only collaborates with DARPA on open-source cybersecurity tools and is in talks with the Pentagon about veteran suicide prevention. Many participants of the protest weren’t aware of this; the protest announcement and the press release did not mention this. People were misled into thinking OpenAI is working on military applications of AI. OpenAI still prohibits the use of their services to “harm people, develop weapons, for communications surveillance, or to injure others or destroy property”. If OpenAI wanted to have a contract with the Pentagon to work on something bad, they wouldn’t have needed to change the usage policies of their publicly available services and could’ve simply provided any services through separate agreements. The community should notice a failure mode and implement something that would prevent unilateral decisions with bad consequences or noticeable violations of deontology.
I am extremely saddened by the emotional impact this has on you. I did not wish that to happen and was surprised and confused by your reaction. Unfortunately, it seems that you still don’t understand the issue I’m pointing at; it is not the “charter” being in the original announcement, it’s the final wording misleading people.

MikhailSamin Feb 8, 2024, 10:01 PM
1 point
0 ∶ 0
in reply to: Charlie Harrison’s comment on: Efficacy of AI Activism: Have We Ever Said No?
I appreciate the reply!

CFCs also saw widespread public mobilisation

Widespread public mobilisation certainly helps to get more policymakers on board with regulation, but the mobilisation has to be caused by a scientific consensus and be pointing at the scientific consensus

If you believe that policymakers regularly sort the “loudest voices” from real scientists

I was talking about the loudest of the activist voices: I’m worried policymakers might hear the public is concerned about hugely beneficial technology instead the public being concerned about the technical reasons for doom and pointing at the scientists to listen to who can explain these reasons

a de-facto moratorium on new approvals

For the context, I want lots of narrow AI to be allowed; I also want things with potential to kill everyone to be prevented from being created, with no chance of someone getting through, anywhere in the planet.

lobbying

$957m in the US alone in 2023 on tech (less on AI): https://www.cnbc.com/2024/02/02/ai-lobbying-spikes-nearly-200percent-as-calls-for-regulation-surge.html

MikhailSamin Jan 4, 2024, 10:24 PM
15 points
1 ∶ 0
in reply to: titotal’s comment on: AI Is Not Software
The bits of code aren’t inscrutable; the matrices the code makes operations on are.

The code for Google Meet represents instructions written by humans; the actual image that you see on your screen and the sound that you hear are a result of something else interacting with these instructions. The words from your speaker or headphones are not intended by the Google Meet designers.

Similarly, the code for GPT-4 represents instructions designed (mostly?) by humans; the actual outputs of GPT-4 are not intended by its designers and depend on the contents of the inscrutable arrays of numbers humans have found.

We understand that we’re multiplying and taking sums of specific matrices in a specific order; but we have no idea how this is able to lead to the results that we see.

The important difference here is that normal software implements algorithms designed by humans, run on hardware designed by humans; AI, in contrast, are algorithms blindly designed by an optimisation process designed by humans, run on software designed by humans, but with no understanding of the algorithms implemented by the numbers our optimisation algorithms find.

It’s like a contrast between CPUs designed by humans and assembly code we don’t understand sent to us by aliens, that we run on CPUs that we do understand

MikhailSamin Dec 27, 2023, 10:29 AM
16 points
7 ∶ 2
on: Only mammals and birds are sentient, according to neuroscientist Nick Humphrey’s theory of consciousness, recently explained in “Sentience: The invention of consciousness”
I’m surprised how confident people are in their theories.

Other mammals and birds exhibiting the same behavior, having much the same neural circuitry, are probably engaging in that behavior for the same reason humans are–because they have internal conscious experiences

Since self-modelling, having feedback mechanisms for reinforcement learning, and having qualia all feel closely related to us, humans, it’s easy to think that “conscious experience” includes all three, and perceive evidence for some of the three as evidence for others. This seems invalid, unless the author does something like making an argument for how exactly having qualia has been evolutionary useful and what properties it’s connected to and then demonstrating related properties in some animals but not others. Does the author actually do that in the book?

MikhailSamin Dec 12, 2023, 10:25 AM
1 point
0 ∶ 0
on: Efficacy of AI Activism: Have We Ever Said No?

Cancellation of prominent SAI geo-engineering experiments (30-60% influence* for SCOPEx, 10-30%* for cancellation of SPICE); De-nuclearization in Kazakhstan in early 1990s (5-15%) and in Sweden in late 1960s (1-15%); Reagan’s move towards a ‘nuclear freeze’ in the 1980s, and subsequent international treaties (20-40%* for former, 5-15%* for latter); Changing UK’s climate policies in 2019 (40-70%* for specific emissions reductions); Germany’s phase-out of nuclear power from 2011 to 2023 (10-30%); Domestic bans on CFCs in late 1970s (5-25%), and influential for stricter international governance from 1990 (1-10%); Europe’s de-facto moratorium on GMOs in late 1990s (30-50%)

I would like to note that none of that had been met with corporations willing to spend potentially dozens of billions of dollars on lobbying, due to the availability of the capital and the enormous profits to be made; and none of these clearly stand out to policymakers as something uniquely important from the competitiveness perspective. AI is more like railroads; it’d be great to make it more like CFCs in the eyes of policymakers, but for that, you need a clear scientific consensus on the existential threat from AI.

It makes sense to study the effectiveness of protests and other activism in general and figure out good strategies to use; but due to the differences, I believe it is too easy to learn wrong lessons from restrictions on GMOs and nuclear and have a substantial net-negative impact.

With AI, we want the policymakers to understand the problem and try to address it; incentivising them to address the public’s concerns won’t lead to the change we need.

The messaging from AI activists should be clear but epistemically honest; we should ask the politicians to listen to the experts, we shouldn’t ask them to listen to the loudest of our voices (as the loudest voices are likely to make claims that the policymakers will know to be incorrect or the representatives of AI companies will be able to explain why they’re wrong, having the aura of science on their side).

Studying protests in general is great; studying whatever led to the Montreal Protocol and the Vienna Convention is excellent; but we need to be clearly different from people who had protested railroads.

—

Also, I’m not sure there’s an actual moratorium on GM crops in Europe- many GMO are officially imported and one GM crop (corn, I think?) is cultivated inside the EU

MikhailSamin Dec 7, 2023, 6:44 PM
3 points
1 ∶ 0
in reply to: Hayven Frienby’s comment on: Some quick thoughts on “AI is easy to control”
pro-AGI advocates such as yourself
I am somewhat confused by your comment. Are you replying to my EA Forum post, the linked article I was replying to, or both? I am certainly not a pro-AGI advocate in the sense you seem to imply: while I think we’re ought to create AGI eventually, after there’s a scientific consensus it’d be safe to, I’m certainly not suggesting to do this now. I’m the author of moratorium.ai, a resource advocating for an AI moratorium.
I am not making an assumption that we’d have any substantial insight into the workings of a superintelligence. The way we develop AI now, we don’t know or understand what the billions-trillions of numbers that make them up represent, and don’t have a way to extract and understand the cognitive architecture that runs on these numbers.
In my model, the default way we develop ASI is via deep learning, and I expect us to not understand that ASI and die shortly afterwards (~80%, and the 20% comes primarily from international governance delaying ASI until we solve all the related problems).
I’d certainly hope we’d instead develop the architecture of ASI manually, thoroughly designing and understanding all its inner workings, not doing any gradient descent; unfortunately, at the moment, this doesn’t seem realistic (although I believe possible in principle).
I have to say that I can imagine how it’d be possible, in principle, to make a safe AGI with deep learning. It’d require a research direction like Infra-Bayesianism to produce results and insights that could be used as constrains for a training run; and it’d require many other research results to avoid inner misalignment; but I think it’s not literally impossible to be relatively confident in safety of an AGI trained with deep learning. This is not something I expect to happen, at all, but it seems theoretically possible.
You raise an important problem of stability under reflection. I don’t think it makes much sense to talk about a subhuman AGI performing CEV (it is a procedure that requires being more capable than humans). But designing a coherent AI system in a way that safely maximises humanity’s CEV, without drifting away from it even as it increases its capabilities, is indeed a complicated problem. I expect, again, that it is solvable in principle, even though I wouldn’t be surprised at all if it takes generations of thousands to millions of scientists to solve.
Even if its neural circuits are rewired, it can store multiple copies of its plan in various places to avoid such a goal interruption, and if it is more intelligent than us, it’s safe to assume it has already done this
I am not sure what you mean by that, as we don’t program these systems and they themselves can’t really rewrite their source code, and storing something internally requires something like gradient hacking; in any case, while generally I expect alignment to not be preserved as a system’s capabilities increase, there are systems that we could, theoretically, get to (although I don’t expect us to), that would care about CEV in a way that doesn’t change as they increase their capabilities.
Humans just aren’t intuitively good at eliminating specific thoughts in the same way we could find, for example, a bad line of code in a traditional program. Therefore, I kind of have to conclude that the only thing that could reliably align an AGI is an equally intelligent AGI—and if this intelligence level is superhuman, the actions of both systems (the misaligned and the aligner) would be an absolute black box to us.
Humanity is capable of producing complicated systems. We usually need to understand laws that govern them first, but we didn’t need rockets capable of getting to the Moon to design the first rocket capable of getting to the Moon. I think the problem of understanding some parts of the space of possible minds is solvable in principle; and it is possible to understand the laws that govern those parts of the space enough to come up with a target—with a mind design that would be safe—and then actually design and launch a corresponding mind, even if it is smarter than humans. Not that I expect any of that to happen within the time constraints.
the burden of proof rests with the person making a positive claim, with skepticism being the default position. It seems more than rational to assume the existence of a superintelligent artificial system anywhere in our lightcone poses some x-risk to us, which means the assumption of x-risk with ASI is the default position
I disagree with that. No previously existing technology wiped out humanity; for anthropic reasons, it’s not obvious how much of evidence this is, but most new technologies certainly haven’t wiped us out, and it seems like the default outside view to take about a new technology. The claim that ASI is likely to kill everyone is extraordinary and requires good reasons.
I think we have really good reasons to think that, and I summarised some of them on moratorium.ai, in this post, and in my other posts. I think it is actively harmful to do advocacy that focuses on “ASI = x-risk by default, burden of proof is on those who build it”, without technical arguments. Policymakers are going to ask Meta about x-risk, and if the policymakers are not already familiar with our technical arguments, they’re going to believe whatever Meta is saying, as they seem to know what they’re talking about and don’t say anything flawed enough for a policymaker not familiar with our arguments to see. To avoid that, we need to explain the technical reasons why ASI is likely to literally kill everyone, in ways that’d, e.g., allow policymakers to call bullshit on what Meta representatives might be saying.
pro-AGI advocates such as yourself to demonstrate, with hard data, that alignment is not only possible but the most likely outcome
Sorry, but I’m confused about who it is addressed to, as I think alignment is extremely unlikely in the current situation. I think the probability of doom conditional on AGI before 2040 is >98%. I think I have good arguments for why it’s the case. I think there’s ~80% everyone will die, and I really hope I’m wrong. The burden on pro-AGI-ASAP advocates is to not just demonstrate that alignment is likely to someone new, but to refute my arguments and arguments of those I agree with, and establish a scientific consensus that’d agree with them.
(Also,
I don’t refer to any potentially conscious system as “subhuman,” all sentient beings (human / animal, or artificial) are intrinsically valuable
By default, ASI is not going to be sentient (in the sense of having qualia), and I wouldn’t consider it to be intrinsically valuable. I also hope we won’t make sentient AIs until we’re fully ready to, see this post’s first footnote for some links.
I’m talking about intelligence as I define it here. The quality of being generally subhuman or generally superhuman on this axis seems important.)

MikhailSamin Nov 14, 2023, 12:00 AM
−10 points
0 ∶ 3
in reply to: Max_Carpendale’s comment on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
(Won’t reply in enough detail, sorry.)
RL, avoiding negative rewards, and seeking positive rewards is probably much simpler than qualia, and we certainly understand this concept better. We can make RL agents and understand what it is that they’re doing. We don’t understand qualia on a similar level.
But I think I failed at communicating the argument because this comment completely missed it.
The point is that your circuits having an explanation ready doesn’t allow you to just always use it no matter what; you need actual reasons to expect this behaviour is better explained by the existence of qualia than simply by RL; you need actual evidence to be able to match something external to a specific thing happening inside you.
There are things functionally outside qualia circuits. If your hand is on fire, the hand doesn’t experience that, something in the brain does. Our pain and decision-making don’t function entirely within our qualia. The updates away from pain and towards pleasure, all the reinforcement learning mechanisms provided by evolution, certainly exist outside the qualia circuits. There’s a lot happening in our brains outside of qualia; I cited studies showing we make decisions seconds before we feel that we’re making decisions.
(If I understood what you meant by solving problems correctly- maybe you meant not solving problems as behaviour, but solving the problem of explaining, though?) If there are problems we solve through qualia, and animals also solve these problems, this would be valid evidence! Please explain what are the problems we solve using qualia, and which animals also solve these problems.

MikhailSamin Nov 13, 2023, 11:44 PM
−2 points
0 ∶ 2
in reply to: Max_Carpendale’s comment on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
The existence of a verbal report of qualia is strong evidence that the reporter has subjective experience (or someone they’ve learned to report this way from having subjective experience). I’m not talking about specific emotional states being reported

MikhailSamin Nov 13, 2023, 11:34 PM
−5 points
0 ∶ 1
in reply to: Alexander Herwix 🔸’s comment on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
Not sure I understand why is this a good reductio ad absurdum.
You know qualia exist somewhere in the causal structure. Sure, maybe the universe is a two-player game, and everyone else is played by a single party; or maybe we’re pranking ourselves, wrote scripts for everyone and erased our own memory or something. All of that is just far less likely than a normal universe.
The argument is that hearing others talk about qualia is extremely strong evidence for living in worlds where there’s qualia causing these talks; it makes more absurd worlds (like one with p-zombies who accidentally- like monkeys writing random letters- talk about something that describes your experience) unlikely, it doesn’t promote worlds that were strongly less likely than others

MikhailSamin Nov 13, 2023, 11:22 PM
2 points
1 ∶ 3
in reply to: Constance Li’s comment on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
I want to be precise, so I’ll point at what can be parsed from your message differently from what I think.
if I understand your argument correctly
This is not a summary of the argument. My argument is about the specifics of how people make invalid inferences. Most of what you understand was intended to be supplementary and not the core of the argument.
Nonetheless, will clarify on the points (note these are not central and I thought about it less than about the actual argument):
You believe the current focus on invertebrate (including shrimp) welfare is based on a flawed inference of sentience, specifically on shallow behavioral observations, presence of pain receptors, and natural human tendencies towards anthropomorphizing everything.
The “specifically” part is not precise, as it’s not just the presence of pain receptors but also behaviour to seek, avoid, make trade-offs, etc., and many other things. There’s a specific way I consider the inference people are making to be invalid.
You would like these criteria to be more considered:
I would like them to be what people consciously understand to be the reason of certain facts being evidence one way or another. Those are not specific factors, it was an attempt to describe possible indirect evidence.
You think that being able to communicate details about one’s qualia is the ultimate standard for inclusion in the group of qualia possessing species.
I think if something talks about qualia without ever hearing about it from humans, you should strongly expect it to have qualia. I wouldn’t generalise this to the automatic inclusion of the whole species, as it would be a weaker statement and I can imagine edge cases.
You wouldn’t eat anything that passes the mirror test
Yep, as it is strong indirect evidence.
Based on your perception that there is a lack of evidence for shrimp possessing qualia, you are recommending to readers that it is “OK to eat shrimp.”
It is not just about a lack of evidence, it is about a fundamentally invalid way of thinking shrimp have subjective experience in the first place, and I don’t think there’s enough valid evidence for subjective experience in shrimp. The evidence people tend to cite is not valid.
And it was not what I was trying to say, but it might still be valuable to reply to your comment.
There are many other markers of sentience/pain/qualia
The first time I wanted to write this post was a couple of years ago when I saw Rethink Priorities research using many markers that have approximately nothing to do with meaningful evidence for the existence of experience of pain.
features which, according to expert agreement, seem to be necessary –although not sufficient– for consciousness
Remarks: I do mention “something that we think could be a part of how qualia works exists in that species” as a valid way to infer evidence. The absence of certain features might be extremely strong evidence for not having subjective experience, but the presence of many of these features might only only extremely weak evidence. (if you don’t have changing parts, like a rock, you don’t have qualia; if you’re a planet with moving parts, you probably still don’t have qualia and it’s ok to eat you even if you have hundreds of features like moving/not moving; also note features are not always independent). (Consciousness is an awful word because people mean totally different things by it.)
Neuroanatomical structures
It’s maybe okay to defer to them and feel free to eat biological organisms from Earth without those, although I’m not a biologist to verify.
Note that the presence of these things doesn’t say much unless you have reasons to believe their evolutionary role is tied to the role of qualia. It is Bayesian evidence if you didn’t know anything about a thing and now know it has these properties, but most of it is probably due to (8 billion humans + many mammals and maybe birds) : all things with it compared to all things without it, including rocks or something.
Behavioral responses that are potential indicators of pain experience, such as defensive behavior or fighting back, and moving away from noxious stimuli. These reactions seem to take into account a noxious stimulus’ intensity and direction. Other observed behaviors include pain relief learning, and long-term behavior alteration to avoid a noxious stimulus.”
Long-term behaviour alterations to avoid what got you an immediate big negative reward is a really helpful adaptation, but how is also having qualia more helpful? Taking the presence of things like that as meaningful evidence for subjective experience is exactly what shows confusion about ways to make valid inferences and surprised me about Rethink’s research a couple of years ago. These things are helpful for a reinforcement learning agent to learn; you need to explain how having qualia is additionally helpful/makes it easier to implement those/is a side effect of implementing those adaptations. Until you have not, this does not provide additional evidence after you know you’re talking about an RL adaptation, if you screen off the increased probability of talking about humans or mammals/birds/things we have other evidence about. (And I think some bacteria might have defensive behaviour and fighting back and moving away from certain things, though I’m not a biologist/might be wrong/didn’t google sources for that background sort-of-maybe-knowledge.)
The mirror test is classically designed for capturing human-like behaviors. In a new format that was designed for natural behaviors of roosters, they actually did pass the mirror test.
I don’t eat chickens, because I spent maybe an hour on this question and was uncertain enough for it to make sense to be safe.
I don’t know of any scientific research that states that the presence of pain receptors is sufficient for possession of qualia. Generally, the more sentience indicators found, the higher the assigned probability of sentience.
Indicators are correlated, and a lot of them are not valid evidence if you’ve already conditioned on states of valid evidence.
If we were in the age where we didn’t have tools for cross language comprehension, then this reasoning would support inferring that Japanese-only speaking people don’t understand the subject mater of a test written in english if they are unable to give satisfactory answers in english.
I feel like this is a digression and won’t comment.
people have historically done a poor job
Yep. I want people to make valid experiments instead.
There is a precedent set to avoid assuming individuals can’t experience pain just because they cannot communicate it the high standards we set. into the 1980s, many surgeons believed babies could not feel pain and so they rarely used anesthetics in surgery
I don’t have reasons to believe newborn babies experience pain, but it is probably a good idea to use anaesthesia, as the stress (without any experience of pain) might have a negative impact on the future development of the baby.
animal communication
Wanna bet fish don’t talk about having subjective experiences?
Recently, the evidence was even sufficient for invertebrate sentience to be recognized by law
I think for most of the UK history, the existence of god is also recognised by law (at least implicitly? and maybe it is still?). How is that evidence?
Also, I don’t eat octopuses.
It seems like you have judged the entire base of evidence on conversations with EAs that are not formally working on sentience research
Nope, I have read a bunch of stuff written by Rethink and I think they should rethink their approach.
I think that the title and conclusion of your post (aka “It’s OK to eat shrimp) is based mostly on a straw man fallacy because it argues against the weakest arguments for invertebrate sentience. If you make any updates after exploring the evidence base further, please consider changing this wording to prevent potential harms from people looking for moral license to continue eating shrimp.
I don’t feel like you understood or addressed my actual arguments, which are about invalid ways EAs make inferences about qualia in certain things. If you explain my argument to me and then explain how exactly the inferences e.g. Rethink make are actually more valid than what I described, and that these valid ways show there’s a meaningful chance shrimp have qualia, I’ll be happy to retract all of that and change the post title and add a disclaimer. So far, I think my argument isn’t even just strawmanned in your comment: it is not considered at all.
The post is mainly addressed to people who already don’t eat shrimp, as I hope they’ll reconsider/will make thought-through decisions on their own (I don’t think many people are likely to read the conclusion and stop being vegan because a random person on the internet says they can).

MikhailSamin Nov 13, 2023, 9:43 PM
−4 points
1 ∶ 5
in reply to: Ben Stevenson’s comment on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
Thanks for the comment!
As I mentioned in the post,
If an LLM talks about qualia, either it has qualia or qualia somewhere else caused some texts to exist, and the LLM read those.
If the LLM describes “its experience” to you, and the experience matches your own subjective experience, you can be pretty sure there’s subjective experience somewhere in the causal structure behind the LLM’s outputs. If the LLM doesn’t have subjective experience but talks about it, that means someone had subjective experience, which made them write a text about it, which the LLM then read. You shouldn’t expect an LLM to talk about subjective experience if it was never trained by anything caused by subjective experience and doesn’t have subjective experience itself.
This means that the ability to talk about qualia is extremely strong evidence for having qualia or having learned about qualia as a result of something that has qualia talking.
I don’t think fish simulate qualia; I think they’re just automation, simply with nothing like experience and nothing resembling experience. They perform adaptations that include efficient reinforcement learning but don’t include experience of processed information.
How do you know whether you scream because of the subjective experience of pain or because of the mechanisms for the instinctive ways to avoid death- how do you know that the scream is caused by the outputs of the neural circuits running qualia and not just by the same stuff that causes the inputs to the circuits that you experience as extremely unpleasant?
It’s not about whether they can talk; parrots and LLMs can be trained to say words in reaction to stuff. If you can talk about having subjective experience, it is valid to assume there’s subjective experience somewhere down the line. If you can’t talk about subjective experience, other, indirect evidence is needed. Assuming you have subjective experience because you react to something external similarly to those with subjective experience is pattern-matching that works on humans for the above reasons, but invalid on everything else without valid evidence for qualia. Neural networks trained with RL would react to pain and whatever is the evolutionary reason for screaming on pain, if you provide similar incentives, RL agents would scream on pain; that doesn’t provide evidence for whether there’s also experience of anything in them.
I’m certain enough fish don’t have qualia to be ok with eating fish; if we solve more critical short-term problems, then, in the future, hopefully, we’ll figure out how subjective experience actually works and will know for sure.

MikhailSamin Nov 13, 2023, 8:58 PM
1 point
0 ∶ 2
in reply to: Jackson Wagner’s comment on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
I appreciate this comment.
Qualia (IMO) certainly is “information processing”: there are inputs and outputs. And it is a part of a larger information-processing thing, the brain. What I’m saying is that there’s information processing happening outside of the qualia circuits, and some of the results of the information processing outside of the qualia circuits are inputs to our qualia.
I think it’s likely that even simple “RL algorithms” might have a very limited, very shallow, non-self-aware kinds of experience: an image-classifier is doing visual-information-processing, so it probably also produces isolated “experiences of vision”
Well, how do you know that visual information processing produces qualia? You can match when algorithms implemented by other humans’ brains to algorithms implemented by your brain, because all of you talk about subjective experience; how do you, inside your neural circuitry, make an inference that a similar thing happens in neurons that just process visual information?
You know you have subjective experience, self-evidently. You can match the computation run by the neural circuitry of your brain to the computation run by the neural circuitry of other humans: because since they talk about subjective experience, you can expect this to be caused by similar computation. This is valid. Thinking that visual information processing is part of what makes qualia (i.e., there’s no way to replace a bunch of your neurons with something that outputs the same stuff without first seeing and processing something, such that you’ll experience seeing as before) is something you can make theories about but is not a valid inference, you don’t have a way of matching the computation of qualia to the whole of your brain.
And, how can you match it to matrix multiplications that don’t talk about qualia, did not have evolutionary reasons for experience, etc.? Do you think an untrained or a small convolutional neural network experiences images to some extent, or only large and trained? Where does that expectation come from?
I’m not saying that qualia is solved. We don’t yet know how to build it, and we can’t yet scan brains and say which circuits implement it. But some people seem more confused than warranted, and they spend resources less effectively than they could’ve.
And I’m not equating qualia to self-model. Qualia is just the experience of information. It doesn’t required a self-model, also on Earth, so far, I expect these things to have been correlated.
If there’s suffering and experience of extreme pain, in my opinion, it matters even if there isn’t reflectivity.
What links here?
- It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood by Mikhail Samin (LessWrong; Nov 13, 2023, 4:51 PM; 0 points)
- It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood by MikhailSamin (Nov 13, 2023, 4:51 PM; -4 points)

MikhailSamin Nov 13, 2023, 7:06 PM
3 points
1 ∶ 1
in reply to: Jackson Wagner’s comment on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
Both (modeling stuff about others by reusing circuits for modeling stuff about yourself without having experience; and having experience without modelling others similarly to yourself) are possible, and the reason why I think the suggested experiment would provide indirect evidence is related to the evolutionary role I consider qualia to possibly play. It wouldn’t be extremely strong evidence and certainly wouldn’t be proof, but it’d be enough evidence for me to stop eating fish that has these things.
The studies about optimistic/pessimistic behaviour tell us nothing about whether these things experience optimism/pessimism, as they are an adaptation an RL algorithm would implement without the need to implement circuits that would also experience these things, unless you can provide a story for why circuitry for experience is beneficial or a natural side effect of something beneficial.
One of the points of the post is that any evidence we can have except for what we have about humans would be inderect, and people call things evidence for confused reasons. Pain-related behaviour is something you’d see in neural networks trained with RL, because it’s good to avoid pain and you need a good explanation for how exactly it can be evidence for qualia.

MikhailSamin Nov 13, 2023, 6:47 PM
1 point
0 ∶ 0
in reply to: Angelina Li’s comment on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
Yep!
I believe that a lot of that is not valid evidence for whether there’s the experience of pain etc. or not, and RL+qualia doesn’t seem to be in any way a better explanation than just reinforcement learning

MikhailSamin Nov 13, 2023, 6:37 PM
−8 points
0 ∶ 2
in reply to: MichaelDickens’s comment on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
Nope, this is not my argument at all. Perhaps this comment can clarify the argument a bit.
I’m definitely not saying that a being that can’t describe qualia does not have qualia. I’m saying that if a being can describe qualia, we know this is caused by qualia somewhere in the causal structure. So we can be pretty sure that humans have qualia. But when our brains identify emotions in things, they can think that humans have qualia, and also that geometric shapes have qualia. When we look at humans and feel like they feel something, this feeling is probably correct, because humans would talk about qualia. When we look at other things, the feeling that other things feel something is no longer linked to a valid way of inferring that other things have qualia, and we need other evidence.
I’ll add a TL;DR

MikhailSamin Nov 13, 2023, 6:30 PM
2 points
0 ∶ 0
on: It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood
Oops. I think I forgot to add a couple of lines, which might’ve made the argument harder to understand. I slightly updated the post, most of the added stuff is bold.
We are a collection of atoms interacting in ways that make us feel and make inferences. The level of neurons is likely the relevant level of abstraction: if the structure of neurons is approximately identical, but the atoms are different, we expect that inputs and outputs will probably be similar, which means that whatever determines the outputs runs on the level of neurons.

...
When we interpret other humans as feeling something when we see their reactions or events happening to them, imagine what it must be like to be like them, feel something we think they must be feeling, and infer there’s something they’re feeling in this moment, our neural circuits make an implicit assumption that other people have qualia. This assumption is, coincidentally, correct: we can infer in a valid way that neural circuits of other humans run subjective experiences because they output words about qualia, which we wouldn’t expect to happen randomly, in the absence of qualia.
But when we see animals that don’t talk about qualia, … [our] neural circuits still recognise emotion in animals like they do in humans, but it is no longer tied to a valid way of inferring that there must be an experience of this emotion.
What links here?
- MikhailSamin's comment on It’s OK to eat shrimp: EAs Make Invalid Inferences About Fish Qualia and Moral Patienthood by MikhailSamin (Nov 13, 2023, 6:37 PM; -8 points)

MikhailSamin Oct 16, 2023, 12:46 AM
8 points
0 ∶ 0
on: Sharing Information About Nonlinear
Have there been any updates from Nonlinear? Have they written a response?

MikhailSamin Sep 11, 2023, 12:06 PM
1 point
0 ∶ 0
on: AI pause/governance advocacy might be net-negative, especially without focus on explaining the x-risk
Back in January, Michael Cohen talked at the House of Commons about the possibility of AI killing everyone. At this point, when policymakers want to understand the problem and turn to you, downplaying x-risk doesn’t make them listen to you more; it makes them less worried and more dismissive. I think a lot of AI governance people/think tanks haven’t updated on this.

MikhailSamin Aug 30, 2023, 1:10 PM
1 point
0 ∶ 1
on: Eliezer Yudkowsky Is Frequently, Confidently, Egregiously Wrong
I don’t know much about philosophy to participate in the zombies & animal consciousness debate meaningfully. (It takes me hours to get people who think there’s a 30% chance microns have qualia to start to understand the reason why they’re likely not. And the word “consciousness” is a not a good one, as people mean totally different things when they use it. And Yudkowsky eats fish but not octopi and some other seafood, because he think there’s a high enough chance octopi have consciousness. But, this is not my area, this is just something that’s fun to talk about.)

But the critique of FDT doesn’t seem valid at all.

If a simulated copy of you gives in to a threat, it makes sense to identically blackmail the real you. If you don’t, it doesn’t make sense to spend resources on reducing your utility.

If you’re the kind of agent who gives in to blackmail, everyone across the multiverse will extract everything you have from you, and you’ll get quite a negative utility for all the threats you didn’t have resources left to give in to. If you don’t give in to threats, you’ll get much less threats and won’t lose as much.

If you’re an AI trained with machine learning that does what a logical decision theory says you should do, you get a lower loss then an AI that does something else, and you get selected.