oh54321

• oh54321 15 Mar 2023 22:04 UTC

  10 points

  1 ∶ 0

  in reply to: oh54321’s comment on: What hap­pened to the OpenPhil OpenAI board seat?

  Source is that I remember Ajeya mentioning at one point that it led to positive changes and she doesn’t think it was a bad decision in retrospect, but cannot get into said changes for NDA reasons.

• oh54321 15 Mar 2023 22:03 UTC

  6 points

  0 ∶ 0

  on: What hap­pened to the OpenPhil OpenAI board seat?

  AFAIK this is not something that can be shared publicly.

• oh54321 28 Jan 2023 16:37 UTC

  1 point

  0 ∶ 1

  in reply to: Zach Stein-Perlman’s comment on: Will Bostrom still be head of FHI in the near fu­ture?

  Ah, sure. Justification is something like there are many people who would be great at running FHI and would want to, I’m guessing there is someone who is much better for optics and would make people more comfortable than Bostrom. Replacing him with one of these just-as-capable people seems to have few downsides and several upsides.

• oh54321 28 Jan 2023 3:55 UTC

  3 points

  0 ∶ 0

  on: Will Bostrom still be head of FHI in the near fu­ture?

  Hm, fairly confused by the downvotes. I’m guessing a) people disagree with this being a good decision or b) there’s a really obvious answer? If it’s b), can you please tell me?

• oh54321 12 Jan 2023 17:40 UTC

  2 points

  1 ∶ 1

  in reply to: Lukas_Gloor’s comment on: [Linkpost] Nick Bostrom’s “Apol­ogy for an Old Email”

  This makes more sense. I still feel a bit irked by the downvotes though—I would like people to be aware of the email, and feel much more strongly about this than about not wanting people to see some of pseudonym’s takes about the apology.

• oh54321 12 Jan 2023 16:27 UTC

  3 points

  2 ∶ 2

  on: [Linkpost] Nick Bostrom’s “Apol­ogy for an Old Email”

  While I agree that these kinds of “bad EA optics” posts are generally unproductive and it makes sense for them to get downvoted, I’m surprised that this specific one isn’t getting more upvoted? Unlike most links to hit pieces and criticisms of EA, this post actually contains new information that has changed my perception of EA and EA leadership.

• oh54321 15 Dec 2022 7:23 UTC

  24 points

  7 ∶ 0

  in reply to: worldoptimization was based’s comment on: Wall Street Jour­nal Re­porter Look­ing to Speak to EAs

  with less intensity, we should discourage the framing of ‘auditing’ very established journalists for red flags

  Why? If I was making a decision to be interviewed by Rachel or not, probably the top thing I’d be worried about is whether they’ve previously written not-very-journalistic hit pieces on tech-y people (which is not all critical pieces in general! some are pretty good and well researched). I agree that there’s such thing as going too far, but I don’t think my comment was doing that.

  I think “there are situations this is valid (but not for the WSJ!)” is wrong? There have been tons of examples of kind of crap articles in usually highly credible newspapers. For example, this article in the NYT seemed to be pretty wrong and not that good.

• oh54321 15 Dec 2022 0:53 UTC

  9 points

  4 ∶ 0

  Error

  The value
    NIL
  is not of type
    SIMPLE-STRING
  when binding #:USER-ID162

• oh54321 7 Dec 2022 3:07 UTC

  4 points

  4 ∶ 1

  on: Some­thing to make my­self fas­ci­nated with com­put­ing sci­ence and AI.

  Not an answer, but why are you trying to do this? If you’re excited about Biology, there seem to be plenty of ways to do impactful biological work.

  Even if you’re purely trying to maximize your impact, for areas like AI Alignment, climate change, or bioweapons, the relevant question is something like: what is the probability that me working on this area prevents a catastrophic event? According to utilitarianism, your # of lives saved is basically this time the total number of people that will ever live or something like this.

  So if there’s a 10% chance of AI killing anyone, and you working on this brings it down to 9.999999%, this is less impactful than if there’s a 0.5 % chance of climate change killing everyone, and you working on this brings it down to 0.499 %. Since it’s much more likely to do impactful work in an area that excites you, seems like bio is solid, since it’s relevant to bioweapons and climate change?

• oh54321 9 Nov 2022 16:37 UTC

  1 point

  0 ∶ 0

  in reply to: Greg_Colbourn ⏸️ ’s comment on: All AGI Safety ques­tions wel­come (es­pe­cially ba­sic ones) [~monthly thread]

  Ok, cool, that’s helpful to know. Is your intuition that these examples will definitely occur and we just haven’t seen them yet (due to model size or something like this)? If so, why?

• oh54321 9 Nov 2022 16:24 UTC

  3 points

  0 ∶ 0

  in reply to: Greg_Colbourn ⏸️ ’s comment on: All AGI Safety ques­tions wel­come (es­pe­cially ba­sic ones) [~monthly thread]

  Right, so I’m pretty on board with optimal policies (i.E., “global maximum” policies) usually involve seeking power. However, gradient descent only finds local maximums, not global maximums. It’s unclear to me whether these global maximums would involve something like power-seeking. My intuition for why this might not be the case is that “small tweaks” in the direction of power-seeking would probably not reap immediate benefits, so gradient descent wouldn’t go down this path.

  This is where my question kind of arose from. If you have empirical examples of power-seeking coming up in tasks where it’s nontrivial that it would come up, I’d find that particularly helpful.

  Does the paper you sent address this? If so, I’ll spend more time reading it.

• oh54321 6 Nov 2022 21:53 UTC

  1 point

  0 ∶ 0

  in reply to: Ben Millwood🔸’s comment on: All AGI Safety ques­tions wel­come (es­pe­cially ba­sic ones) [~monthly thread]

  Thanks! I think most of this made sense to me. I’m a bit fuzzy on the fourth bullet. Also, I’m still confused why a model would even develop an alternative goal to maximizing its reward function, even if it’s theoretically able to pursue one.

• oh54321 4 Nov 2022 18:36 UTC

  3 points

  1 ∶ 0

  on: All AGI Safety ques­tions wel­come (es­pe­cially ba­sic ones) [~monthly thread]

  Should we expect power-seeking to often be naturally found by gradient descent? Or should we primarily expect it to come up when people are deliberately trying to make power-seeking AI, and train the model as such?

• oh54321 2 Nov 2022 22:17 UTC

  1 point

  0 ∶ 0

  on: All AGI Safety ques­tions wel­come (es­pe­cially ba­sic ones) [~monthly thread]

  “RL agents with coherent preference functions will tend to be deceptively aligned by default.”—Why?

• oh54321 2 Nov 2022 12:34 UTC

  4 points

  1 ∶ 0

  on: All AGI Safety ques­tions wel­come (es­pe­cially ba­sic ones) [~monthly thread]

  I’m relatively unconvinced by most arguments I’ve read that claim deceptive alignment will be a thing (which I understand to be a model that intentionally behaves differently on its training data and test data to avoid changing its parameters in training).

  Most toy examples I’ve seen, or thought experiments, don’t really seem to actually be examples of deceptive alignment since the model is actually trained on the “test” data in these examples. For example, while humans can decieve their teachers in etiquette school then use poor manners outside the school, you modify neurons in your brain on both using manners inside etiquette school and using them outside of etiquette school, so it makes sense that you would distinguish them.

  I certainly think it is possible for a model to be deceptive, but this seems to be much, much more complicated and harder-to-find in the gradient descent landscape, and it seems to me that there’s pretty much no reason for a model to learn to be deceptive in the first place. This makes it seem like it won’t come up at all in practice, or at the least will be very easy to avoid.

  Why do people buy deceptive alignment? Could you give some concrete examples where it could come up?

[Question] What are the best ways to en­courage de-es­ca­la­tion in re­gards to Ukraine?

• oh54321 1 Oct 2022 17:04 UTC

  1 point

  0 ∶ 0

  in reply to: DC’s comment on: How scared should we be about nu­clear war right now?

  Thanks! Why did you not put this on EA forum?

• oh54321 7 Sep 2022 6:46 UTC

  2 points

  0 ∶ 0

  in reply to: Michael Townsend🔸’s comment on: What highly-im­pact­ful work is the most similar to solv­ing fun math puz­zles (if any)?

  This is extremely helpful, thank you.

[Question] What highly-im­pact­ful work is the most similar to solv­ing fun math puz­zles (if any)?

• oh54321 2 Sep 2022 21:28 UTC

  2 points

  0 ∶ 0

  on: Oxford /​ Cam­bridge Union Societies

  From what I’ve heard, Cambridge EAs have already been elected to positions in the Cambridge Union. I think several EAs at Oxford have also been in Oxford Union-adjacent circles, although it seems like most haven’t engaged with the Union due to lack of interest.

  FWIW, I’m guessing most people focused on a social science at Oxbridge have already made up their mind about whether they’d like to network at the Unions or try their hand at getting elected for something. That being said, it seems like relatively few people have considered the potential impact of voting in said elections, as well as other things that would not require serious engagement.