Ofer

Karma: 2,144

Last nontrivial update: 2024-02-01.

Send me anonymous feedback: https://docs.google.com/forms/d/1qDWHI0ARJAJMGqhxc9FHgzHyEFp-1xneyl9hxSMzJP0/viewform

Any type of feedback is welcome, including arguments that a post/comment I wrote is net negative.

I’m interested in ways to increase the EV of the EA community by mitigating downside risks from EA related activities. Without claiming originality, I think that:

Complex cluelessness is a common phenomenon in the domains of anthropogenic x-risks and meta-EA (due to an abundance of crucial considerations). It is often very hard to judge whether a given intervention is net-positive or net-negative.
The EA community is made out of humans. Humans’ judgement tends to be influenced by biases and self-deception. That is a serious source of risk, considering the previous point.
- Some potential mitigations involve improving some aspects of how EA funding works, e.g. with respect to conflicts of interest. Please don’t interpret my interest in such mitigations as accusations of corruption etc.

Feel free to reach out by sending me a PM. I’ve turned off email notifications for private messages, so if you send me a time sensitive PM consider also pinging me about it via the anonymous feedback link above.

Ofer Mar 4, 2023, 10:46 PM
19 points
6 ∶ 6
in reply to: TW123’s comment on: Nick Bostrom should step down as Director of FHI
Judging by Bostrom’s body of work so far, if him stepping down from being the sole director of FHI is very high EV, it’s probably because it will free more of his time for thinking and writing (rather than due to someone else making FHI have higher EV.) So the more relevant question here, I think, is whether being the sole director of FHI is really the best use of Bostrom’s time. It ~~could~~ may be, especially if it will cause him to have more influence in the future. Though when considering that question, he should be mindful about the strong bias humans have towards decisions that cause the decision maker to have higher status (like a decision to be the leader of a prestigious organization).

(Whatever the answer to that question is, it’s probably also the right answer in a world where the 26-year-old-email trainwreck did not occur.)

Ofer Mar 4, 2023, 7:06 PM
111 points
73 ∶ 14
on: Nick Bostrom should step down as Director of FHI

But the Apology […] reused the original racial slur[…]

Where? You mean in the 26-year-old email that he quoted in the apology? If so, the above claim seems unfair and deceptive.

Ofer Mar 3, 2023, 8:36 PM
15 points
0 ∶ 0
in reply to: Alex Catalán Flores’s comment on: Make Conflict of Interest Policies Public
Hi there!

Has Charity Entrepreneurship published its CoI policy? (I didn’t find it on CE’s website.)

Ofer Mar 2, 2023, 9:45 PM
6 points
0 ∶ 0
on: Counterproductive Altruism: The Other Heavy Tail
Is there a version of the paper that is freely available?

Ofer Mar 2, 2023, 9:06 PM
2 points
1 ∶ 1
on: Why I think it’s important to work on AI forecasting
(I skimmed this so sorry if I just missed it, but…) I think you should also discuss the potential downside risks of AI forecasting (e.g. risks related to drawing attention to some ML approaches, that currently get very little attention, when listing the approaches that are most likely to enable a fast takeoff).

Ofer Mar 2, 2023, 8:30 PM
47 points
17 ∶ 1
on: Call to demand answers from Anthropic about joining the AI race
I think for starters Anthropic should publish the list of its board members, its bylaws and cap table. I’m not aware of any of that currently being public info. (Since Anthropic is a public-benefit corporation, presumably the CEO can choose to publish that info even if doing so is not aligned with the financial interests of the shareholders.)

Ofer Feb 24, 2023, 8:46 AM
5 points
1 ∶ 0
on: Manifund Impact Market / Mini-Grants Round On Forecasting
Does this initiative includes attempts to mitigate risks regarding incentivizing risky, net-negative projects that can end up being extremely beneficial?

For example, should an impact market fund forecasting about which particular machine learning techniques will end up being most transformative? How about a gain-of-function research effort that was completed successfully and caused the development of an extremely important vaccine? (I’m not saying the efforts in these examples are net-negative, just that they can be.)

Ofer Feb 24, 2023, 8:06 AM
105 points
33 ∶ 6
on: Taking a leave of absence from Open Philanthropy to work on AI safety
In your recent Cold Takes post you disclosed that your wife owns equity in both OpenAI and Anthropic. (She was appointed to a VP position at OpenAI, as was her sibling, after you joined OpenAI’s board of directors^[1]). In 2017, under your leadership, OpenPhil decided to generally stop publishing “relationship disclosures”. How do you intend to handle conflicts of interest, and transparency about them, going forward?

You wrote here that the first intervention that you’ll explore is AI safety standards that will be “enforced via self-regulation at first, and potentially government regulation later”. AI companies can easily end up with “self-regulation” that is mostly optimized to appear helpful, in order to avoid regulation by governments. Conflicts of interest can easily influence decisions w.r.t. regulating AI companies (mostly via biases and self-deception, rather than via conscious reasoning).
1. ↩︎
  EDIT: you joined OpenAI’s board of directors as part of a deal between OpenPhil and OpenAI that involved recommending a $30M grant to OpenAI.
What links here?
- Lauren Maria's comment on Consent Isn’t Always Enough by Jeff Kaufman 🔸 (Feb 24, 2023, 9:03 PM; 2 points)

Ofer Feb 21, 2023, 1:46 PM
4 points
2 ∶ 3
in reply to: Nathan Young’s comment on: A statement and an apology

And if sockpuppets could be used on polls they could be used everywhere. Why not complain that the whole voting system is wrong?

I agree that such concerns are not limited to the polls on this page. (I’ve also wrote a comment related to such concerns here.)

Ofer Feb 21, 2023, 11:20 AM
10 points
11 ∶ 1
in reply to: Nathan Young’s comment on: A statement and an apology
Even if sock puppet accounts do not currently influence votes, the more the EA community will use such polls to decide on things the more likely such influence will become.

Ofer Feb 11, 2023, 10:42 AM
3 points
1 ∶ 4
on: In (mild) defence of the social/professional overlap in EA
If the head of the NIH had had a romantic relationship with the head of the Wuhan Institute of Virology, would that have been reasonable?

Ofer Feb 10, 2023, 5:55 PM
4 points
0 ∶ 0
in reply to: aog’s comment on: Does Open Philanthropy have a public document regarding their conflict of interest policy?

OpenPhil has been fairly transparent about this, stating it in their grant report on OpenAI (and I believe I’ve seen it elsewhere).

Back in March 2017, in a writeup about the $30M grant recommendation to OpenAI, OpenPhil were transparent about HK (then-CEO of OpenPhil) being engaged to DA’s sister, while DA was a researcher at OpenAI and also a technical advisor to OpenPhil and living in the same house as HK. (This was before the two siblings were appointed to VP positions at OpenAI, which I’m not aware was ever publicly reported by OpenPhil).

As you mentioned in your comment, OpenPhil changed their policy about publicly disclosing relationships. If today OpenPhil faces CoI situations that are similar to the ones they faced when recommending that $30M grant to OpenAI, they may not mention those CoIs publicly at all. It is also possible that the relationship disclosures about the $30M grant are publicly available on OpenPhil’s website today only because they were publicly discussed prior to the change in OpenPhil’s policy (I don’t know whether they were). Quoting from OpenPhil’s Relationship Disclosure Policy:

Previously, we also included relationship disclosures in our public grant writeups. As of August 2017, we generally no longer do so, and we have removed the bulk of historical disclosures from our website, though we have left a few in place where they had already been discussed in other public fora or were important to understanding the basic case for a grant. This decision was a result of our evolving thinking on what information is important to share publicly and our view that some disclosures seemed to unnecessarily infringe on the privacy of our staff and grantees. Given these considerations, while we no longer publish relationship disclosures by default, we may do so when we think it would help others learn from our work and maximize the impact of their own giving.
What links here?
- Ofer's comment on Taking a leave of absence from Open Philanthropy to work on AI safety by Holden Karnofsky (Feb 24, 2023, 8:06 AM; 105 points)

Ofer Feb 3, 2023, 1:30 PM
8 points
1 ∶ 0
in reply to: Aleksi Maunu’s comment on: Questions about OP grant to Helena
I think the stated reasoning there by OP is that it’s important to influence OpenAI’s leadership’s stance and OpenAI’s work on AI existential safety. Do you think this is unreasonable?

I do not think that reasoning was unreasonable, but I also think that deciding to give $30M to OpenAI in 2017 was not obviously net-positive, and it might have been one of the most influential decisions in human history (e.g. due to potentially influencing timelines, takeoff speed and the research trajectory of AGI, due to potentially^[1] inspiring many talented people to pursue/invest in AI, and due to potentially^[1:1] increasing the number of actors who competitively pursue the development of AGI).

Therefore, the appointments of the fiancée and her sibling to VP positions, after OpenPhil’s decision to recommend that $30M grant, seems very problematic. I’m confident that HK consciously judged that $30M grant to be net-positive. But conflicts of interest can easily influence people’s decisions by biasing their judgment and via self-deception, especially with respect to decisions that are very non-obvious and deciding either way can be reasonable.

Furthermore, being appointed to VP at OpenAI seems very financially beneficial (in expectation) not just because of the salary from OpenAI. The appointments of the fiancée and her sibling to VP positions probably helped them to successfully approach investors as part of their effort to found Anthropic, which ended up raising at least $704M. HK said in an interview:

Anthropic is a new AI lab, and I am excited about it, but I have to temper that or not mislead people because Daniela, my wife, is the president of Anthropic. And that means that we have equity, and so [...] I’m as conflict-of-interest-y as I can be with this organization.

You wrote:

I would be highly suspicious of the grant if I didn’t happen to place a lot of trust in Holden Karnofsky and OP.

I don’t think there is a single human on earth whose judgement can be trusted to be resilient to severe conflicts of interest while making such influential non-obvious decisions (because such decisions can be easily influenced by biases and self-deception).
1. ↩︎↩︎
  EDIT: added “potentially”; the $30M grant may not have caused those things if OpenAI would have sufficiently succeeded without it.

Ofer Feb 3, 2023, 7:15 AM
29 points
8 ∶ 3
in reply to: Lukas_Gloor’s comment on: Questions about OP grant to Helena
The OP (original poster) should not have invoked “nepotism” here at all. The alleged nepotism here is seemingly not worse than a situation in which a person uses their own wealth to fund a Helena-like organization that they lead.

It’s important that such misplaced invocations of nepotism w.r.t. OpenPhil will not distract from concerns that are actually valid. In particular: OpenPhil recommended a $30M grant to OpenAI in a deal that involved HK^[1] (then-CEO of OpenPhil) becoming a board member of OpenAI. This occurred no later than March 2017. Later, OpenAI appointed both HK’s then-fiancée and the fiancée’s sibling to VP positions^[2].
1. ↩︎
  This comment originally referred to “OP” instead of “HK” by mistake, due to me copying from another comment, sorry about that.
2. ↩︎
  See these two LinkedIn profiles and the “Relationship disclosures” section in this OpenPhil writeup.
What links here?
- Ofer's comment on Taking a leave of absence from Open Philanthropy to work on AI safety by Holden Karnofsky (Feb 24, 2023, 8:06 AM; 105 points)

Ofer Jan 16, 2023, 10:52 AM
0 points
2 ∶ 12
on: Does EA understand how to apologize for things?

Imagine if I called someone’s mother overweight in a vulgar manner. When they get upset, I compose a long apology email where I apologize for the language, but then note that I believe it is factually true their mother has a BMI substantially above average, as does their sister, father, and wife.

Strong downvote. Readers who have not read Bostrom’s apology can get the false impression that the apology contains a note that is analogous to the above. It does not.

Ofer Jan 12, 2023, 10:30 PM
69 points
38 ∶ 14
on: CEA statement on Nick Bostrom’s email
Bostrom’s email was horrible, but I think it’s unreasonable on CEA’s part to make this short statement without mentioning that the email was written 26 years ago, as part of a discussion about offending people.

Ofer Jan 12, 2023, 5:15 PM
3 points
4 ∶ 5
in reply to: Jeroen Willems🔸’s comment on: [Linkpost] Nick Bostrom’s “Apology for an Old Email”

You’re still saying it’s likely almost everyone has done things like that email or worse, which seems unlikely to me.

Consider a random human who spent about 278,860 hours as an adult person on earth (as Bostrom has, according to Wikipedia). Let’s label that random person as “extremely morally robust” if during those 278,860 hours they have not once done something at least as horrible as writing that email (as a philosophy student, in a discussion about offending people, in 1995).

Suppose someone is robust to the point that the chance of them doing something at least as horrible in a random hour of their adult life is like the chance to flip a coin and get heads 15 times in a row. Even that hypothetical human is very unlikely (~0.02% chance) to get the “extremely morally robust” label as defined above.

Ofer Jan 12, 2023, 1:42 PM
2 points
0 ∶ 0
in reply to: Lukas_Gloor’s comment on: [Linkpost] Nick Bostrom’s “Apology for an Old Email”

and I agree with Amber Dawn that the original email is somewhat worse than something I’d have expected most people to have in their past.

When I wrote “If you look at the most horrible thing that every person has done in their entire life[…]” I meant: “If for every person you look at the most horrible thing that that person has done[…]” (I.e. it can be a completely different thing for each person.) I’ve edited my comment to make that clear.

Ofer Jan 12, 2023, 1:41 PM
9 points
3 ∶ 3
in reply to: Amber Dawn’s comment on: [Linkpost] Nick Bostrom’s “Apology for an Old Email”
(Not sure whether this clarification is needed, but just in case...)

When I wrote “If you look at the most horrible thing that every person has done in their entire life[…]” I meant: “If for every person you look at the most horrible thing that that person has done[…]” (I.e. it can be a completely different thing for each person.) I’ve edited my comment to make that clear.

Ofer Jan 12, 2023, 11:10 AM
75 points
34 ∶ 31
on: [Linkpost] Nick Bostrom’s “Apology for an Old Email”
The original email (written 26 years ago) was horrible, but Bostrom’s apology seems reasonable.

If you look at the most horrible thing that ~~every~~ each person has done in their entire life, it’s likely that almost everyone (in that age) has done things that are at least as horrible as writing that email.

The OP reads as if it was optimized to cause as much drama as possible, rather than for pro-social goals.
What links here?
- David_Althaus's comment on [Linkpost] Nick Bostrom’s “Apology for an Old Email” by pseudonym (Jan 13, 2023, 10:31 AM; 17 points)