MikhailSamin

• MikhailSaminFeb 14, 2024, 9:06 AM

  3 points

  0 ∶ 0

  on: EA com­mu­nity needs mechanisms to avoid de­cep­tive mes­sag­ing and de­on­tolog­i­cally bad plans

  It seems that one of the protest organisers now says that what happened was what I described in the post as “Even the best-intending people are not perfect, often have some inertia, and aren’t able to make sure the messaging they put out isn’t misleading and fully propagate updates”.

  I suggest focusing on what mechanisms can be designed to prevent misleading messaging from emerging in the future.

• MikhailSaminFeb 14, 2024, 8:56 AM

  3 points

  0 ∶ 0

  in reply to: Holly Elmore ⏸️ 🔸’s comment on: EA com­mu­nity needs mechanisms to avoid de­cep­tive mes­sag­ing and de­on­tolog­i­cally bad plans

  I wouldn’t care if people knew some number to some approximation and not fully. This is quite different from saying something that’s technically not false but creates a misleading impression you thought was more likely to get people to support your message.

  I don’t want to be spending time this way and would be happy if you found someone we’d both be happy with them reading our message exchange and figuring out how deceptive or not was the protest messaging.

• MikhailSaminFeb 14, 2024, 8:43 AM

  0 points

  0 ∶ 1

  in reply to: Holly Elmore ⏸️ 🔸’s comment on: EA com­mu­nity needs mechanisms to avoid de­cep­tive mes­sag­ing and de­on­tolog­i­cally bad plans

  I thought he was saying mostly that it wasn’t bad to work with the military bc it was cybersecurity, where to me that wasn’t the crux

  ? It is clearly not what I was telling you. At the end of January, I told you pretty directly that for people who are not aware of the context what you wrote might be misleading, because you omitted crucial details. It’s not about how good or bad what OpenAI are doing is. It’s about people not having important details of the story to judge for themselves.

  Mikhail may have felt I was being soldier-y bc I wouldn’t spend more time trying to figure out what he was talking about, but that had more to do with me thinking I had basically understood his point (which I took to be basically that I wasn’t including enough details in the promotional materials so people wouldn’t have a picture he considered accurate enough) and just disagreed with it.

  I’m not sure where it’s coming from. I suggest you look at the messages we exchanged around January 31 and double-check you’re not misleading people here.

• MikhailSaminFeb 14, 2024, 8:01 AM

  3 points

  0 ∶ 0

  in reply to: Holly Elmore ⏸️ 🔸’s comment on: EA com­mu­nity needs mechanisms to avoid de­cep­tive mes­sag­ing and de­on­tolog­i­cally bad plans

  (Edit- I changed the title to better reflect the post contents and what I currently think happened; this comment doesn’t represent my views on the current post title)

  I’m honest in that I consider the community as a whole to have been putting out deceptive messages, which caused the post title; and, separately, I’m honestly saying that while it’s not my main guess for what happened, I can’t rule out the possibility that you understood the messaging could’ve been misleading and still went on with it, accepting that some people can be misled and thinking it’s ok. Things I have heard from you but wouldn’t share publicly don’t inspire confidence.

• MikhailSaminFeb 14, 2024, 7:51 AM

  0 points

  0 ∶ 1

  in reply to: Holly Elmore ⏸️ 🔸’s comment on: EA com­mu­nity needs mechanisms to avoid de­cep­tive mes­sag­ing and de­on­tolog­i­cally bad plans

  (Edit- I no longer believe the above is a true recount of what happened and retract the below.)

  Good to see this. I’m sad that you didn’t think that after reading “Even the best-intending people are not perfect, often have some inertia, and aren’t able to make sure the messaging they put out isn’t misleading and fully propagate updates” in the post. As I mentioned before, it could’ve been simply a postmortem.

FTX ex­pects to re­turn all cus­tomer money; claw­backs may go away

• MikhailSaminFeb 14, 2024, 3:28 AM

  −11 points

  1 ∶ 3

  in reply to: Jason’s comment on: EA com­mu­nity needs mechanisms to avoid de­cep­tive mes­sag­ing and de­on­tolog­i­cally bad plans

  (Edit- I changed the title to better reflect the post contents and what I currently think happened; this comment doesn’t represent my views on the current post title or my current estimates of what happened)

  I believe there’s a chance that protest organisers understood their phrasing could potentially cause people to have an impression (OpenAI is working with the military) not informed by details (on cybersecurity and veteran suicide prevention) but kept the phrasing because they thought it suited their goals better.

  I don’t expect them to have an active malicious intent (I don’t think they wanted to do something bad). But I think it’s likely enough they were acting deceptively. I consider the chance of them thinking it’s not “misleading” in a bad way and it’s alright to do if it suits the goals to be much higher than what I would be comfortable with. (And people who attended the protest told me that yeah, this is deceptive, after familiarising themselves with the details.)

  ---

  The title of the post is chosen because I think that regardless of the above, the community (as a whole/​as an entity/​agent) locally expected to benefit from people having this impression, while knowing it to be a misled impression, and went on with the messaging. Even if individual actors did not take decisions they knew would create impressions they knew were different from what reality is, I think community as a whole can still be deceptive in the sense of having the ability to realise all that and prevent misleading messaging, but not having done so. I think the community should work on strengthening and maybe institutionalising this ability, with the goal of being able to prevent even actively deceptive messaging, where the authors might hope to achieve something good but violate deontology.

• MikhailSaminFeb 14, 2024, 2:46 AM

  −3 points

  3 ∶ 3

  in reply to: Holly Elmore ⏸️ 🔸’s comment on: EA com­mu­nity needs mechanisms to avoid de­cep­tive mes­sag­ing and de­on­tolog­i­cally bad plans

  (Edit- I updated towards them knowing the message is misleading, so this comment no longer correctly represents the level of my uncertainty.)

  To be clear, in the post, I’m not saying that you, personally, tried to deceive people. I don’t mention your name or any of the orgs. I’m saying the community as a whole had knowledge of truth but communicated something misleading. My goal with the post is to ask the community to come up with ways of avoiding misleading people. I did not wish for you out your reputation to be involved in any way you wouldn’t want.

  But it’s not clear to me what happened, exactly, that led to the misleading messaging being preserved. At the end of January, I shared this with you: “I mean, possibly I don’t know English well and parse this incorrectly, but when I read “OpenAI would be taking the Pentagon as a client” in your tweet, it parses in that context for me as “OpenAI is taking Pentagon as a client to help with offensives”, which is not quite different from what we know”, “ Like, it’s feels different from saying “OpenAI says they’re collaborating with DARPA on cybersecurity. We don’t know whether they’re working with Pentagon on weapons now, but the governance structure wouldn’t stop them from it””, “Like, it doesn’t matter for people aware of the context, but for people without the context, this might be misleading”.

  I shared these concerns, thinking that it might be helpful to point out how a message you wrote might be misleading, assuming you were unaware, and not worrying about spending too much time on it.

  I’m also confused about the “small ask”- the protest announcement was pretty focused on Pentagon.

  (To be clear, I don’t want to be checking anyone’s future comms when this is not my job. I want the community to have mechanisms that don’t involve me. It seemed cheap to make something slightly better a couple of times, and so I talked to a founder of an org organising protests and they fixed some pretty bad claims, sent you these messages, etc., but I dislike the situation where the community doesn’t have mechanisms to systematically avoid mistakes like that.)

• MikhailSaminFeb 14, 2024, 12:20 AM

  6 points

  0 ∶ 0

  in reply to: Holly Elmore ⏸️ 🔸’s comment on: EA com­mu­nity needs mechanisms to avoid de­cep­tive mes­sag­ing and de­on­tolog­i­cally bad plans

  Left a comment there, but to repeat:

  because I made an error in my original press release

  This was, indeed, an honest mistake that you corrected. It is not at all what I’m talking about in the post. I want the community to pay attention to the final messaging being deceptive about the nature of OpenAI’s relationship with the Pentagon.

  I am extremely saddened by the emotional impact this has on you. I did not wish that to happen and was surprised and confused by your reaction. Unfortunately, it seems that you still don’t understand the issue I’m pointing at; it is not the “charter” being in the original announcement, it’s the final wording misleading people.

• MikhailSaminFeb 14, 2024, 12:18 AM

  7 points

  3 ∶ 1

  in reply to: Holly Elmore ⏸️ 🔸’s comment on: Holly_El­more’s Shortform

  See my post here.

  because I made an error in my original press release

  This was an honest mistake that you corrected. It is not at all what I’m talking about in the post. I want the community to pay attention to the final messaging being deceptive about the nature of OpenAI’s relationship with the Pentagon. The TL;DR of my post:

  Good-hearted EAs lack the mechanisms to not output information that can mislead people. They organised a protest, with the messaging centred on OpenAI changing their documents to “work with the Pentagon”, while OpenAI only collaborates with DARPA on open-source cybersecurity tools and is in talks with the Pentagon about veteran suicide prevention. Many participants of the protest weren’t aware of this; the protest announcement and the press release did not mention this. People were misled into thinking OpenAI is working on military applications of AI. OpenAI still prohibits the use of their services to “harm people, develop weapons, for communications surveillance, or to injure others or destroy property”. If OpenAI wanted to have a contract with the Pentagon to work on something bad, they wouldn’t have needed to change the usage policies of their publicly available services and could’ve simply provided any services through separate agreements. The community should notice a failure mode and implement something that would prevent unilateral decisions with bad consequences or noticeable violations of deontology.

  I am extremely saddened by the emotional impact this has on you. I did not wish that to happen and was surprised and confused by your reaction. Unfortunately, it seems that you still don’t understand the issue I’m pointing at; it is not the “charter” being in the original announcement, it’s the final wording misleading people.

An EA used de­cep­tive mes­sag­ing to ad­vance her pro­ject; we need mechanisms to avoid de­on­tolog­i­cally du­bi­ous plans

• MikhailSaminFeb 8, 2024, 10:01 PM

  1 point

  0 ∶ 0

  in reply to: Charlie Harrison’s comment on: Effi­cacy of AI Ac­tivism: Have We Ever Said No?

  I appreciate the reply!

  CFCs also saw widespread public mobilisation

  Widespread public mobilisation certainly helps to get more policymakers on board with regulation, but the mobilisation has to be caused by a scientific consensus and be pointing at the scientific consensus

  If you believe that policymakers regularly sort the “loudest voices” from real scientists

  I was talking about the loudest of the activist voices: I’m worried policymakers might hear the public is concerned about hugely beneficial technology instead the public being concerned about the technical reasons for doom and pointing at the scientists to listen to who can explain these reasons

  a de-facto moratorium on new approvals

  For the context, I want lots of narrow AI to be allowed; I also want things with potential to kill everyone to be prevented from being created, with no chance of someone getting through, anywhere in the planet.

  lobbying

  $957m in the US alone in 2023 on tech (less on AI): https://​​www.cnbc.com/​​2024/​​02/​​02/​​ai-lobbying-spikes-nearly-200percent-as-calls-for-regulation-surge.html

• MikhailSaminJan 4, 2024, 10:24 PM

  15 points

  1 ∶ 0

  in reply to: titotal’s comment on: AI Is Not Software

  The bits of code aren’t inscrutable; the matrices the code makes operations on are.

  The code for Google Meet represents instructions written by humans; the actual image that you see on your screen and the sound that you hear are a result of something else interacting with these instructions. The words from your speaker or headphones are not intended by the Google Meet designers.

  Similarly, the code for GPT-4 represents instructions designed (mostly?) by humans; the actual outputs of GPT-4 are not intended by its designers and depend on the contents of the inscrutable arrays of numbers humans have found.

  We understand that we’re multiplying and taking sums of specific matrices in a specific order; but we have no idea how this is able to lead to the results that we see.

  The important difference here is that normal software implements algorithms designed by humans, run on hardware designed by humans; AI, in contrast, are algorithms blindly designed by an optimisation process designed by humans, run on software designed by humans, but with no understanding of the algorithms implemented by the numbers our optimisation algorithms find.

  It’s like a contrast between CPUs designed by humans and assembly code we don’t understand sent to us by aliens, that we run on CPUs that we do understand

NYT is su­ing OpenAI&Microsoft for alleged copy­right in­fringe­ment; some quick thoughts

• MikhailSaminDec 27, 2023, 10:29 AM

  16 points

  7 ∶ 2

  on: Only mam­mals and birds are sen­tient, ac­cord­ing to neu­ro­scien­tist Nick Humphrey’s the­ory of con­scious­ness, re­cently ex­plained in “Sen­tience: The in­ven­tion of con­scious­ness”

  I’m surprised how confident people are in their theories.

  Other mammals and birds exhibiting the same behavior, having much the same neural circuitry, are probably engaging in that behavior for the same reason humans are–because they have internal conscious experiences

  Since self-modelling, having feedback mechanisms for reinforcement learning, and having qualia all feel closely related to us, humans, it’s easy to think that “conscious experience” includes all three, and perceive evidence for some of the three as evidence for others. This seems invalid, unless the author does something like making an argument for how exactly having qualia has been evolutionary useful and what properties it’s connected to and then demonstrating related properties in some animals but not others. Does the author actually do that in the book?

• MikhailSaminDec 12, 2023, 10:25 AM

  1 point

  0 ∶ 0

  on: Effi­cacy of AI Ac­tivism: Have We Ever Said No?

  Cancellation of prominent SAI geo-engineering experiments (30-60% influence* for SCOPEx, 10-30%* for cancellation of SPICE); De-nuclearization in Kazakhstan in early 1990s (5-15%) and in Sweden in late 1960s (1-15%); Reagan’s move towards a ‘nuclear freeze’ in the 1980s, and subsequent international treaties (20-40%* for former, 5-15%* for latter); Changing UK’s climate policies in 2019 (40-70%* for specific emissions reductions); Germany’s phase-out of nuclear power from 2011 to 2023 (10-30%); Domestic bans on CFCs in late 1970s (5-25%), and influential for stricter international governance from 1990 (1-10%); Europe’s de-facto moratorium on GMOs in late 1990s (30-50%)

  I would like to note that none of that had been met with corporations willing to spend potentially dozens of billions of dollars on lobbying, due to the availability of the capital and the enormous profits to be made; and none of these clearly stand out to policymakers as something uniquely important from the competitiveness perspective. AI is more like railroads; it’d be great to make it more like CFCs in the eyes of policymakers, but for that, you need a clear scientific consensus on the existential threat from AI.

  It makes sense to study the effectiveness of protests and other activism in general and figure out good strategies to use; but due to the differences, I believe it is too easy to learn wrong lessons from restrictions on GMOs and nuclear and have a substantial net-negative impact.

  With AI, we want the policymakers to understand the problem and try to address it; incentivising them to address the public’s concerns won’t lead to the change we need.

  The messaging from AI activists should be clear but epistemically honest; we should ask the politicians to listen to the experts, we shouldn’t ask them to listen to the loudest of our voices (as the loudest voices are likely to make claims that the policymakers will know to be incorrect or the representatives of AI companies will be able to explain why they’re wrong, having the aura of science on their side).

  Studying protests in general is great; studying whatever led to the Montreal Protocol and the Vienna Convention is excellent; but we need to be clearly different from people who had protested railroads.

  —

  Also, I’m not sure there’s an actual moratorium on GM crops in Europe- many GMO are officially imported and one GM crop (corn, I think?) is cultivated inside the EU

• MikhailSaminDec 7, 2023, 6:44 PM

  3 points

  1 ∶ 0

  in reply to: Hayven Frienby’s comment on: Some quick thoughts on “AI is easy to con­trol”

  pro-AGI advocates such as yourself

  I am somewhat confused by your comment. Are you replying to my EA Forum post, the linked article I was replying to, or both? I am certainly not a pro-AGI advocate in the sense you seem to imply: while I think we’re ought to create AGI eventually, after there’s a scientific consensus it’d be safe to, I’m certainly not suggesting to do this now. I’m the author of moratorium.ai, a resource advocating for an AI moratorium.

  I am not making an assumption that we’d have any substantial insight into the workings of a superintelligence. The way we develop AI now, we don’t know or understand what the billions-trillions of numbers that make them up represent, and don’t have a way to extract and understand the cognitive architecture that runs on these numbers.

  In my model, the default way we develop ASI is via deep learning, and I expect us to not understand that ASI and die shortly afterwards (~80%, and the 20% comes primarily from international governance delaying ASI until we solve all the related problems).

  I’d certainly hope we’d instead develop the architecture of ASI manually, thoroughly designing and understanding all its inner workings, not doing any gradient descent; unfortunately, at the moment, this doesn’t seem realistic (although I believe possible in principle).

  I have to say that I can imagine how it’d be possible, in principle, to make a safe AGI with deep learning. It’d require a research direction like Infra-Bayesianism to produce results and insights that could be used as constrains for a training run; and it’d require many other research results to avoid inner misalignment; but I think it’s not literally impossible to be relatively confident in safety of an AGI trained with deep learning. This is not something I expect to happen, at all, but it seems theoretically possible.

  You raise an important problem of stability under reflection. I don’t think it makes much sense to talk about a subhuman AGI performing CEV (it is a procedure that requires being more capable than humans). But designing a coherent AI system in a way that safely maximises humanity’s CEV, without drifting away from it even as it increases its capabilities, is indeed a complicated problem. I expect, again, that it is solvable in principle, even though I wouldn’t be surprised at all if it takes generations of thousands to millions of scientists to solve.

  Even if its neural circuits are rewired, it can store multiple copies of its plan in various places to avoid such a goal interruption, and if it is more intelligent than us, it’s safe to assume it has already done this

  I am not sure what you mean by that, as we don’t program these systems and they themselves can’t really rewrite their source code, and storing something internally requires something like gradient hacking; in any case, while generally I expect alignment to not be preserved as a system’s capabilities increase, there are systems that we could, theoretically, get to (although I don’t expect us to), that would care about CEV in a way that doesn’t change as they increase their capabilities.

  Humans just aren’t intuitively good at eliminating specific thoughts in the same way we could find, for example, a bad line of code in a traditional program. Therefore, I kind of have to conclude that the only thing that could reliably align an AGI is an equally intelligent AGI—and if this intelligence level is superhuman, the actions of both systems (the misaligned and the aligner) would be an absolute black box to us.

  Humanity is capable of producing complicated systems. We usually need to understand laws that govern them first, but we didn’t need rockets capable of getting to the Moon to design the first rocket capable of getting to the Moon. I think the problem of understanding some parts of the space of possible minds is solvable in principle; and it is possible to understand the laws that govern those parts of the space enough to come up with a target—with a mind design that would be safe—and then actually design and launch a corresponding mind, even if it is smarter than humans. Not that I expect any of that to happen within the time constraints.

  the burden of proof rests with the person making a positive claim, with skepticism being the default position. It seems more than rational to assume the existence of a superintelligent artificial system anywhere in our lightcone poses some x-risk to us, which means the assumption of x-risk with ASI is the default position

  I disagree with that. No previously existing technology wiped out humanity; for anthropic reasons, it’s not obvious how much of evidence this is, but most new technologies certainly haven’t wiped us out, and it seems like the default outside view to take about a new technology. The claim that ASI is likely to kill everyone is extraordinary and requires good reasons.

  I think we have really good reasons to think that, and I summarised some of them on moratorium.ai, in this post, and in my other posts. I think it is actively harmful to do advocacy that focuses on “ASI = x-risk by default, burden of proof is on those who build it”, without technical arguments. Policymakers are going to ask Meta about x-risk, and if the policymakers are not already familiar with our technical arguments, they’re going to believe whatever Meta is saying, as they seem to know what they’re talking about and don’t say anything flawed enough for a policymaker not familiar with our arguments to see. To avoid that, we need to explain the technical reasons why ASI is likely to literally kill everyone, in ways that’d, e.g., allow policymakers to call bullshit on what Meta representatives might be saying.

  pro-AGI advocates such as yourself to demonstrate, with hard data, that alignment is not only possible but the most likely outcome

  Sorry, but I’m confused about who it is addressed to, as I think alignment is extremely unlikely in the current situation. I think the probability of doom conditional on AGI before 2040 is >98%. I think I have good arguments for why it’s the case. I think there’s ~80% everyone will die, and I really hope I’m wrong. The burden on pro-AGI-ASAP advocates is to not just demonstrate that alignment is likely to someone new, but to refute my arguments and arguments of those I agree with, and establish a scientific consensus that’d agree with them.

  (Also,

  I don’t refer to any potentially conscious system as “subhuman,” all sentient beings (human /​ animal, or artificial) are intrinsically valuable

  By default, ASI is not going to be sentient (in the sense of having qualia), and I wouldn’t consider it to be intrinsically valuable. I also hope we won’t make sentient AIs until we’re fully ready to, see this post’s first footnote for some links.

  I’m talking about intelligence as I define it here. The quality of being generally subhuman or generally superhuman on this axis seems important.)

Some quick thoughts on “AI is easy to con­trol”

• MikhailSaminNov 14, 2023, 12:00 AM

  −10 points

  0 ∶ 3

  in reply to: Max_Carpendale’s comment on: It’s OK to eat shrimp: EAs Make In­valid In­fer­ences About Fish Qualia and Mo­ral Patienthood

  (Won’t reply in enough detail, sorry.)

  RL, avoiding negative rewards, and seeking positive rewards is probably much simpler than qualia, and we certainly understand this concept better. We can make RL agents and understand what it is that they’re doing. We don’t understand qualia on a similar level.

  But I think I failed at communicating the argument because this comment completely missed it.

  The point is that your circuits having an explanation ready doesn’t allow you to just always use it no matter what; you need actual reasons to expect this behaviour is better explained by the existence of qualia than simply by RL; you need actual evidence to be able to match something external to a specific thing happening inside you.

  There are things functionally outside qualia circuits. If your hand is on fire, the hand doesn’t experience that, something in the brain does. Our pain and decision-making don’t function entirely within our qualia. The updates away from pain and towards pleasure, all the reinforcement learning mechanisms provided by evolution, certainly exist outside the qualia circuits. There’s a lot happening in our brains outside of qualia; I cited studies showing we make decisions seconds before we feel that we’re making decisions.

  (If I understood what you meant by solving problems correctly- maybe you meant not solving problems as behaviour, but solving the problem of explaining, though?) If there are problems we solve through qualia, and animals also solve these problems, this would be valid evidence! Please explain what are the problems we solve using qualia, and which animals also solve these problems.

• MikhailSaminNov 13, 2023, 11:44 PM

  −2 points

  0 ∶ 2

  in reply to: Max_Carpendale’s comment on: It’s OK to eat shrimp: EAs Make In­valid In­fer­ences About Fish Qualia and Mo­ral Patienthood

  The existence of a verbal report of qualia is strong evidence that the reporter has subjective experience (or someone they’ve learned to report this way from having subjective experience). I’m not talking about specific emotional states being reported