Michael_Cohen

• Michael_Cohen 7 Apr 2023 18:31 UTC

  1 point

  0 ∶ 0

  in reply to: Michael_Cohen’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  Do you have a minute to react to this? Are you satisfied with my response?

• Michael_Cohen 15 Mar 2023 21:50 UTC

  2 points

  0 ∶ 1

  on: Suc­cess with­out dig­nity: a nearcast­ing story of avoid­ing catas­tro­phe by luck

  How “natural” are intended generalizations (like “Do what the supervisor is hoping I’ll do, in the sense that most humans would mean this phrase rather than in a precise but malign sense”) vs. unintended ones (like “Do whatever maximizes reward”)?

  I think this is an important point. I consider the question in this paper, published last year at AI Magazine. See the “Competing Models of the Goal” section, and in particular the “Arbitrary Reward Protocols” subsection. (2500 words)

  I think there’s something missing from the discussion here, which the key point of that section.First, I claim that sufficiently advanced agents will likely need to engage in hypothesis testing between multiple plausible models of what worldly events lead to reinforcement, or else they would fail at certain tasks. So even if the “intended generalization” is a quite bit more plausible to the agent than the unintended one, as long as it is cheap to test them, and as long as it has a long horizon, it would likely deem wireheading to be worth trying out, just in case. That said, in some situations (I mention a chess game in the paper) I expect the intended generalization to be so much simpler that it isn’t even worth trying out.

  Just a warning before you read it, I use the word “reward” a bit differently than you appear to. In my terminology, I would phrase is this as “Do what the supervisor is hoping” vs. “Do whatever maximizes the relevant physical signal”, and the agent would essentially wonder which of the two constitutes “reward”, rather than being a priori sure that its past rewards “are” those physical signals.

• Michael_Cohen 11 Mar 2023 9:10 UTC

  7 points

  0 ∶ 0

  in reply to: Aaron Gertler’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  This is very high-quality. No disputes just clarifications.

  I don’t just mean meta-orgs.

  I think working for a well-financed grantmaking organization is not outrageously unconventional, although I suspect most lean on part-time work from well-respected academics more than OpenPhil does.

  And I think 80k may just be an exception (a minor one, to some extent), borne out of an unusually clear gap in the market. I think some of their work should be done in academia instead (basically whatever work it’s possible to do), but some of the very specific stuff like the jobs board wouldn’t fit there.

  Also, if we imagine an Area Dad from an Onion Local News article, I don’t think he’s skepticism would be quite as pronounced for 80k as for other orgs like, e.g., an AI Safety camp.

• Michael_Cohen 10 Mar 2023 11:00 UTC

  3 points

  1 ∶ 0

  in reply to: Rubi J. Hudson’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  Thank you for the edit, and thank you again for your interest. I’m still not sure what you mean by a person “having access to the ground truth of the universe”. There’s just no sense I can think of where it is true that this a requirement for the mentor.

  “The system is only safe if the mentor knows what is safe.” It’s true that if the mentor kills everyone, then the combined mentor-agent system would kill everyone, but surely that fact doesn’t weight against this proposal at all. In any case, more importantly a) the agent will not aim to kill everyone regardless of whether the mentor would (Corollary 14), which I think refutes your comment. And b) for no theorem in the paper does the mentor need to know what is safe; for Theorem 11 to be interesting, he just needs to act safely (an important difference for a concept so tricky to articulate!). But I decided these details were beside the point for this post, which is why I only cited Corollary 14 in the OP, not Theorem 11.

• Michael_Cohen 9 Mar 2023 23:37 UTC

  1 point

  0 ∶ 0

  in reply to: Douglas Knight’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  Robin Hanson didn’t occur to me when I wrote it or any of the times I read it! I was just trying to channel what I thought conventional advice would be.

• Michael_Cohen 9 Mar 2023 23:22 UTC

  1 point

  0 ∶ 0

  in reply to: Michael_Cohen’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  So basically, just philosophy, math, and some very simple applied math (like, say, the exponential growth of an epidemic), but already that last example is quite shaky.

• Michael_Cohen 9 Mar 2023 23:19 UTC

  2 points

  0 ∶ 0

  in reply to: Ben Millwood’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  In fields where it’s possible to make progress with first-principles arguments/​armchair reasoning, I think smart non-experts stand a chance of outperforming. I don’t want to make strong claims about the likelihood of success here; I just want to say that it’s a live possibility. I am much more comfortable saying that outperforming conventional wisdom is extremely unlikely on topics where first-principles arguments/​armchair reasoning are insufficient.

  (As it happens, EAs aren’t really disputing the experts in philosophy, but that’s beside the point...)

• Michael_Cohen 9 Mar 2023 23:10 UTC

  1 point

  0 ∶ 0

  in reply to: Ben Millwood’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  academics themselves have criticized the peer review system a great deal for various reasons, including predatory journals, incentive problems, publication bias, Why Most Published Research Findings Are False, etc

  I think we could quibble on the scale and importance of all of these points, but I’m not prepared to confidently deny any of them. The important I want to make is: compared to what alternatives? The problem is hard, and even the best solution can be expected to have many visible imperfections. How persuaded you be by a revoluationary enumerating the many weaknesses of democratic government without comparing them to a proposed (often imagined) alternative?

  people outside academia, e.g. practitioners in industry, are often pretty sharply skeptical of the usefulness of academic work, or unaware of it entirely,

  Again, compared to what alternative? I’m guessing they would say that at a company, the “idea people” get their ideas tested by the free market, and this grounds them, and makes their work practical. I am willing to believe that free-market-testing is a more reliable institution than peer review for evaluating certain propositions, and I am willing to buy that this is conventionally well-understood. But for many propositions, I would think there is no way to “productize” it such that profitability of the product is logically equivalent to the truth of the proposition. (This may not be the alternative you had in mind at all, if you had an alternative in mind).

  the peer review system didn’t obviously arise from a really intentional and thoughtful design effort

  This is definitely not a precondition for a successful social institution.

  at the time the peer review system was developed, an enormous amount of our modern tools for communication and information search, processing, dissemination etc. didn’t exist, so it really arose in an environment quite different from the one it’s now in.

  But conventional wisdom persists in endorsing peer-review anyway!

• Michael_Cohen 9 Mar 2023 17:11 UTC

  6 points

  2 ∶ 2

  in reply to: lilly’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  What “major life goals should include (emphasis added)” is not a sociological question. It is not a topic that a sociology department would study. See my comment that I agree “conventional wisdom is wrong” in dismissing the philosophy of effective altruism (including the work of Peter Singer). And my remark immediately thereafter: “Yes, these are philosophical positions, not sociological ones, so it is not so outrageous to have a group of philosophers and philosophically-minded college students outperform conventional wisdom by doing first-principles reasoning”.

  I am not citing Amazon as an example of an actor using evidence and reason to do as much good as possible. I am citing it as an example of an organization that is effective at what it aims to do.

• Michael_Cohen 9 Mar 2023 12:08 UTC

  6 points

  1 ∶ 0

  in reply to: Aaron Gertler’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  Upvoted this.

  You generally shouldn’t take Forum posts as seriously as peer-reviewed papers in top journals

  I suspect I would advise taking them less seriously than you would advise, but I’m not sure.

  It could also imply that EA should have fewer and larger orgs, but that’s a question too complicated for this comment to cover

  I think there might be a weak conventional consensus in that direction, yes. By looking at the conventional wisdom on this point, we don’t have deal with the complicatedness of the question—that’s kind of my whole point. But even more importantly, perhaps fewer EA orgs that are not any larger; perhaps only two EA orgs (I’m thinking of 80k and OpenPhil; I’m not counting CHAI as an EA org). There is not some fixed quantity of people that need to be employed in EA orgs! Conventional wisdom would suggest, I think, that EAs should mostly be working at normal, high-quality organizations/​universities, getting experience under the mentorship of highly qualified (probably non-EA) people.

• Michael_Cohen 9 Mar 2023 11:47 UTC

  9 points

  2 ∶ 0

  in reply to: Rubi J. Hudson’s comment on: Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

  I’m someone who has read your work (this paper and FGOIL, the latter of which I have included in a syllabus), and who would like to see more work in similar vein, as well as more formalism in AI safety. I say this to establish my bona fides, the way you established your AI safety bona fides.

  Thanks! I should have clarified it has received some interest from some people.

  you don’t show that “when a certain parameter of a certain agent is set sufficiently high, the agent will not aim to kill everyone”, you show something more like “when you can design and implement an agent that acts and updates its beliefs in a certain way and can restrict the initial beliefs to a set containing the desired ones and incorporate a human into the process who has access to the ground truth of the universe, then you can set a parameter high enough that the agent will not aim to kill everyone”

  “When you can design and implement an agent that acts and updates its beliefs in a certain way and can restrict the initial beliefs to a set containing the desired ones”. That is the “certain agent” I am talking about. “Restrict” is an odd word choice, since the set can be as large as you like as long as it contains the truth. “and incorporate a human into the process who has access to the ground truth of the universe.” This is incorrect; can I ask you to edit your comment? Absolutely nothing is assumed about the human mentor, certainly no access to the ground truth of the universe; it could be a two-year-old or a corpse! That would just make the Mentor-Level Performance Corollary less impressive.

  I don’t deny that certain choices about the agent’s design make it intractable. This is why my main criticism was “People don’t seem to bother investigating or discussing whether their concerns with the proposal are surmountable.” Algorithm design for improved tractability is the bread and butter of computer science.

Against EA-Com­mu­nity-Re­ceived-Wis­dom on Prac­ti­cal So­ciolog­i­cal Questions

• Michael_Cohen 24 Feb 2023 1:06 UTC

  17 points

  1 ∶ 0

  on: Tak­ing a leave of ab­sence from Open Philan­thropy to work on AI safety

  I’d love to chat with you about directions here, if you’re interested. I don’t know anyone with a bigger value of p(survival | West Wing levels of competence in major governments) - p(survival | leave it to OpenAI and DeepMind leadership). I’ve published technical AI existential safety research at top ML conferences/​journals, and I’ve gotten two MPs in the UK onside this week. You can see my work at michael-k-cohen.com, and you can reach me at michael.cohen@eng.ox.ac.uk.

• Michael_Cohen 22 Nov 2022 12:28 UTC

  13 points

  2 ∶ 1

  on: Pre-An­nounc­ing the 2023 Open Philan­thropy AI Wor­ld­views Contest

  Glad to hear about this!

  I have a recommendation for the structure of it. I’d recommend that anonymous reviewers review submissions and share their reviews with the authors (perhaps privately) before a rebuttal phase (also perhaps private). And then reviewers can revise their reviews, and then chairs can make judgments about which submissions to publish.

• Michael_Cohen 2 Aug 2022 16:42 UTC

  9 points

  1 ∶ 0

  on: On how var­i­ous plans miss the hard bits of the al­ign­ment challenge

  I constructed an agent where you can literally prove that if you set a parameter high enough, it won’t try to kill everyone, while still eventually at least matching human-level intelligence. Sure it uses a realizability assumption, sure it’s intractable in its current form, sure it might require an enormously long training period, but these are computer science problems, not philosophy problems, and they clearly suggest paths forward. The underlying concept is sound. It struck me as undignified to say this in the past, but maybe dignity rightly construed should compel me to: it absolutely boggles me that ~no one in the EA community talks about this. It’s not in this blog post; it’s not in Richard’s curriculum; it wasn’t in Evan’s list of promising AGI safety ideas.

  I agree with your perspective on all of these approaches, except my initial reaction is to be more pessimistic about natural abstractions. It seems to me that a good understanding of natural abstractions is not good enough for putting a handle on a part of an agent’s mind. We’d also need to understand “natural types”, the type signatures that agents’ brains use to represent those abstractions. And I think that there is a long, long list of types, in which each is as natural as the rest.

  There’s an interpretability benchmark that occurred to me recently, which I may as well mention here, because I agree approximately none of the interpretability research I see strikes me as progress toward strategically relevant interpretation of AGI. Try to understand what corvids are saying to each other.

• Michael_Cohen 31 May 2022 12:23 UTC

  9 points

  0 ∶ 0

  on: What are the coolest top­ics in AI safety, to a hope­lessly pure math­e­mat­i­cian?

  Here are some of mine to add to Vanessa’s list.

  One on imitation learning. [Currently an “accept with minor revisions” at JMLR]

  One on conservatism in RL. A special case of Vanessa’s infra-Bayesianism. [COLT 2020]

  One on containment and myopia. [IEEE]

• Michael_Cohen 15 Jul 2020 19:22 UTC

  6 points

  0 ∶ 0

  on: Ob­jec­tions to Value-Align­ment be­tween Effec­tive Altruists

  Among many things I agree with, the part I agree the most with:

  EAs give high credence to non-expert investigations written by their peers, they rarely publish in peer-review journals and become increasingly dismissive of academia

  I think a fair amount of the discussion of intelligence loses its bite if “intelligence” is replaced with what I take to be its definition: “the ability to succeed a randomly sampled task” (for some reasonable distribution over tasks). But maybe you’d say that perceptions of intelligence in the EA community are only loosely correlated with intelligence in this sense?

  As for cached beliefs that people accept on faith from the writings of perceived-intelligent central figures, I can’t identify any beliefs that I have that I couldn’t defend myself (with the exception that I think many mainstream cultural norms are hard to improve on, so for a particular one, like “universities are the best institutions for producing new ideas”, I can’t necessarily defend this on the object level). But I’m pretty sure there aren’t any beliefs I hold just because a high-status EA holds them. Of course, some high-status EAs have convinced me of some positions, most notably Peter Singer. But that mechanism for belief transmission within EA, i.e. object-level persuasion, doesn’t run afoul of your concerns about echochamberism, I don’t think.

  But maybe you’ve had a number of conversations with people who appeal to “authority” in defending certain positions, which I agree would be a little dicey.

• Michael_Cohen 16 Oct 2019 13:27 UTC

  1 point

  0 ∶ 0

  in reply to: Michael_Wiebe’s comment on: X-risk dol­lars → An­drew Yang?

  it doesn’t follow that it’s a good investment overall

  Yes, it doesn’t by itself—my point was only meant as a counterargument to your claim that the efficient market hypothesis precluded the possibility of political donations being a good investment.

• Michael_Cohen 12 Oct 2019 16:01 UTC

  5 points

  0 ∶ 0

  in reply to: kbog’s comment on: X-risk dol­lars → An­drew Yang?

  Well, there are >100 million people who have to join some constituency (i.e. pick a candidate), whereas potential EA recruits aren’t otherwise picking between a small set of cults philosophical movements. Also, AI PhD-ready people are in much shorter supply than, e.g. Iowans, and they’d be giving up much much much more than someone just casting a vote for Andrew Yang.

• Michael_Cohen 12 Oct 2019 5:50 UTC

  5 points

  0 ∶ 0

  in reply to: kbog’s comment on: X-risk dol­lars → An­drew Yang?

  we’ve had two presidents now who actively tried to counteract mainstream views on climate-change, and they haven’t budged climate scientists at all.

  I have updated in your direction.

  Of course, AI alignment is substantially more scientifically accepted and defensible than climate skepticism.

  Yep.

  You only mean this as a possibility in the future, if there is any point where AGI is believed to be imminent, right?

  No I meant starting today. My impression is that coalition-building in Washington is tedious work. Scientists agreed to avoid gene editing in humans well before it was possible (I think). In part, that might have made it easier since the distantness of it meant fewer people were researching it to begin with. If AGI is a larger part of an established field, it seems much harder to build a consensus to stop doing it.