Ofer

Karma: 2,139

Last nontrivial update: 2024-02-01.

Send me anonymous feedback: https://docs.google.com/forms/d/1qDWHI0ARJAJMGqhxc9FHgzHyEFp-1xneyl9hxSMzJP0/viewform

Any type of feedback is welcome, including arguments that a post/comment I wrote is net negative.

I’m interested in ways to increase the EV of the EA community by mitigating downside risks from EA related activities. Without claiming originality, I think that:

Complex cluelessness is a common phenomenon in the domains of anthropogenic x-risks and meta-EA (due to an abundance of crucial considerations). It is often very hard to judge whether a given intervention is net-positive or net-negative.
The EA community is made out of humans. Humans’ judgement tends to be influenced by biases and self-deception. That is a serious source of risk, considering the previous point.
- Some potential mitigations involve improving some aspects of how EA funding works, e.g. with respect to conflicts of interest. Please don’t interpret my interest in such mitigations as accusations of corruption etc.

Feel free to reach out by sending me a PM. I’ve turned off email notifications for private messages, so if you send me a time sensitive PM consider also pinging me about it via the anonymous feedback link above.

Ofer Nov 14, 2023, 6:30 PM
−2 points
0 ∶ 6
on: Naive application of the ITN framework on a situation like the one in Gaza might lead us wrong
This post seems to have gotten several downvotes. I hope they are from relatively-impartial readers.

Note to Israelis who may be reading this: I did not upvote/downvote this post and I do not intent to vote on such posts going forward. I think you should do the same.

Ofer Nov 14, 2023, 5:56 PM
43 points
6 ∶ 1
on: Naive application of the ITN framework on a situation like the one in Gaza might lead us wrong
Disclosure: I have served in Israel Defense Forces, I live in Israel, I feel horrible about what Israel has done in the past 75 years to millions of Palestinians and I do not want Israel to end up as a horrible stain on human history. I am probably unusually biased when dealing with this topic. I am not making here a claim that people in EA should or should not get involved and in what way.

I want to add some context that I think many people are missing. (I think that context is necessary for applying the ITN framework to the topic).

TL;DR: Israel had a very problematic incentive to empower Hamas and to cause it to gain and maintain control over the Gaza strip, while weakening the much more peaceful Palestinian National Authority. The way that humanity will handle the current conflict and its aftermath may influence whether states can generally expect to benefit from acting on such incentives while violating basic, ~universal norms related to justice and human decency. A failure to uphold such norms can undermine the ability of humanity to coordinate and to achieve a peaceful future^[1].

Quoting from Wikipedia (extended-protected entry):

During the 1947–49 Palestine war, an estimated 700,000 Palestinians fled or were expelled, comprising around 80% of the Palestinian Arab inhabitants of what became Israel.

I believe it is widely accepted that if that had not happened, it would have been impossible to establish a Jewish, democratic state in the ~~current borders of Israel~~ borders that Israel had after that war. The UN General Assembly Resolution 194 from 1948 resolves that:

refugees wishing to return to their homes and live at peace with their neighbours should be permitted to do so at the earliest practicable date, and that compensation should be paid for the property of those choosing not to return and for loss of or damage to property which, under principles of international law or equity, should be made good by the Governments or authorities responsible.

In 1949, in the debates about Israel’s admittance to the UN, Israel’s UN representative promised to “co-operate with the organs of the United Nations with all the means at our disposal in the fulfillment of [Resolution 194]”. It has been 74 years and Israel did not fulfill that promise so far.

The international community has put some amount of pressure on Israel over the years with respect to this issue and the related issue of allowing the establishment of a Palestinian state. ~~But the intensity of that pressure was negatively correlated with the amount of violence that Israel was facing from Hamas et al.~~ But the international community was less inclined to put pressure on Israel while it was facing violence from Hamas et al. This gave Israel a problematic incentive: to make Hamas stronger and the Palestinian National Authority weaker. Israel had various ~~ways~~ possible ways to act on that incentive, including generally promoting violence (e.g. adapting policies that cause incidents in which settlers attack/harass Palestinian people) and allowing the establishment of new settlements that constitute “a flagrant violation under international law” according to the UN Security Council. It is often difficult to know whether a state did X due to an incentive Y. I will now provide some evidence that the incentive above was influential.

Here is a quote from a NY Times article; it refers to Benjamin Netanyahu who has been the prime minster of Israel from 1996 to 1999 and most of the time from 2009 until today (2023).

Over time, however, [Netanyahu] came to see Hamas as a way to balance power against the Palestinian Authority, which has administrative control over the West Bank and has long sought a peace agreement in Israel in exchange for a Palestinian state. Mr. Netanyahu told aides over the years that a feeble Palestinian Authority lowered the pressure on him to make concessions to Palestinians in negotiations, according to several former Israeli officials and people close to Mr. Netanyahu.

Also, here is a quote from a CNN article that is based on WikiLeaks:

The leaked cables also indicate the Israelis appear to have seen an advantage in Hamas consolidating its control in Gaza. A cable from June 13, 2007, quotes the head of Israeli military intelligence, Maj. Gen. Amos Yadlin, as saying that “Israel would be ‘happy’ if Hamas took over Gaza because the (Israel Defense Forces) could then deal with Gaza as a hostile state.”
1. ↩︎
  Though we also have an intrinsic desire to enforce (our subjective sense of) justice, which can bias our judgment and make us overweight this utilitarian reasoning.
What links here?
- Ofer's comment on EA thoughts from Israel-Hamas war by ezrah (Dec 6, 2023, 5:33 PM; 115 points)

Ofer Nov 13, 2023, 12:44 PM
6 points
1 ∶ 1
on: Survey on the acceleration risks of our new RFPs to study LLM capabilities
I think that the survey should have been designed and carried out by an independent party. The author is a senior program officer at OPP who is known to have a lot of influence over funding in the AI x-risks space. It seems they themselves have selected “47 relatively senior people who have been full-time working on AI x-risk reduction for multiple years”. The author sent those people a request to fill a survey that starts with the words: “I want to launch […]”. The survey was not anonymous, a fact that was addressed on the survey form:

(Seeing names will be useful to me for doing “robustness sanity checks” on how different the outcome would have been if I had surveyed a moderately different population, and will thus inform my qualitative analysis.)

(I am not taking here a stance for or against the specific initiative that the survey was asking about.)

Ofer Nov 10, 2023, 3:29 PM
4 points
0 ∶ 0
in reply to: johnburidan’s comment on: Will Maskill Gave Up on Utilitarianism (and the rest of us are soon to follow)
The words “I’m a utilitarian” in the OP link to a different interview.

Is this the interview you are talking about here? I failed to find that quote in the transcript of that interview either. (Both in the transcript that YouTube provides automatically and in the transcript that the description links to.)

Ofer Nov 5, 2023, 2:32 PM
10 points
4 ∶ 0
in reply to: Erich_Grunewald 🔸’s comment on: Which aid charity should I give to for people in Gaza?
It seems that currently the bottleneck is not funding for orgs that provide humanitarian aid, but rather Israel’s willingness to allow that humanitarian aid to get to the civilians that need it.

The Israeli government and individuals within it seem to have some myopic incentives to cause harm to civilians, including: (1) as a lever over Hamas; (2) for the purpose of increasing its deterrence against ~~Lebanon~~ Hezbollah and in general; and (3) perhaps for the purpose of eventually causing civilians to flee to Egypt.

Unrelated note: This is an extremely contentious political issue around the world and especially in the US. We should consider that the involvement of EA in such issues can be harmful. It seems relevant to note here this AMA post about this war, from a seemingly strongly pro-Israel author who so far never wrote anything on this forum (from that account) outside that AMA. That post currently has 62 karma points.

Disclosure: I have served in Israel Defense Forces, live in Israel, feel horrible about what Israel has done so far to Palestinians and about the current war, and I do not want Israel to end up as a horrible stain on human history. I am probably unusually biased when dealing with this topic.

Ofer May 5, 2023, 8:40 AM
0 points
1 ∶ 0
in reply to: Dawn Drescher’s comment on: Why aren’t more of us working to prevent AI hell?

but what work on s-risks really looks like in the end is writing open source game theory simulations and writing papers

Research that involves game theory simulations can be net-positive, but it also seems very dangerous, and should not be done unilaterally. Especially when it involves publishing papers and source code.

Ofer May 4, 2023, 8:38 AM
4 points
0 ∶ 0
on: [AISN #4]: AI and Cybersecurity, Persuasive AIs, Weaponization, and Geoffrey Hinton talks AI risks
I couldn’t find on the website of the Center for AI Safety any information about who is running it, or who is on the board. Is this information publicly available anywhere?

Ofer May 2, 2023, 7:04 AM
4 points
1 ∶ 0
in reply to: MichaelStJules’s comment on: Discussion about AI Safety funding (FB transcript)
The local incentives people face often discourage publicly giving negative feedback that may cause an applicant to not get funding. (“I would gain nothing from giving negative feedback, and that person might hate me.”)

Ofer Apr 29, 2023, 12:09 PM
11 points
0 ∶ 0
on: New open letter on AI — “Include Consciousness Research”
Notably, it seems that Yoshua Bengio is one of the signatories (he is an extremely prominent AI researcher; one of the three researchers who won a Turing Award for their work in deep learning).

Ofer Apr 19, 2023, 3:44 PM
23 points
4 ∶ 0
in reply to: MichaelPlant’s comment on: Updates to the Effective Ventures US board
Also: When CEA (now Effective Ventures) appointed an Executive Director in 2019, they wrote in their blog that the CEO of OpenPhil (at the time) “provided extensive advice” to the search committee and “contributed to the final recommendation to the board of trustees”.

Ofer Apr 8, 2023, 10:48 PM
0 points
0 ∶ 1
on: A Fresh FAQ on Impact Markets

These risks mostly seem like “black swan” risks to us – deleterious but highly unlikely risks.

This effort can end up popularizing a mechanism that incentivizes and funds risky, net-negative projects—in anthropogenic x-risk domains, using EA funding.

Conditional on this effort ending up being extremely impactful, do you really believe the downside risks are “highly unlikely”? (And do you think most of the EA community would agree)?

Ofer Mar 30, 2023, 2:44 PM
−2 points
0 ∶ 0
on: Want to win the AGI race? Solve alignment.

Solving (scalable) alignment might be worth lots of $$$ and key to beating China.

.

I really don’t want Xi Jinping Thought to rule the world

.

If you want to win the AGI race, if you want to beat China, [...]

.

Let’s not lose to China [...]

The China-is-an-opponent-that-we-must-beat-in-the-AI-race is a classic talking point of AI companies in the US, that is used as an argument against regulation. Are you by any chance affiliated with an AI company, or an organization that is funded by one?

Ofer Mar 29, 2023, 9:37 PM
2 points
0 ∶ 0
in reply to: EdoArad’s comment on: Casting the Decisive Vote
I’m not sure. Very few people would use the term “correlation” here; but perhaps quite a few people sometimes reason along the lines of: “Should I (not) do X? What happens if many people (not) do it?”

Ofer Mar 28, 2023, 6:33 PM
3 points
0 ∶ 0
on: Casting the Decisive Vote
Relatedly: deciding to vote can also be important due to one’s decisions being correlated with the decisions of other potential voters. A more general version of this consideration is discussed in Multiverse-wide Cooperation via Correlated Decision Making by Caspar Oesterheld.

Ofer Mar 17, 2023, 12:51 AM
3 points
0 ∶ 0
in reply to: Dustin Moskovitz’s comment on: What happened to the OpenPhil OpenAI board seat?

How did you come to the conclusion that funding ML research is “pretty messy and unpredictable”? I’ve seen many ML companies funded over the years as straightforwardly as other tech startups, […]

I think it’s important to distinguish here between companies that intend to use existing state-of-the-art ML approaches (where the innovation is in the product side of things) and companies that intend to advance the state-of-the-art in ML. I’m only claiming that research that aims to advance the state-of-the-art in ML is messy and unpredictable.

To illustrate my point: If we use an extreme version of the messy-and-unpredictable view, we can imagine that OpenAI’s research was like repeatedly drawing balls from an urn, where drawing each ball costs $1M and there is a 1% chance (or whatever) to draw a Winning Ball (that is analogous to getting a super impressive ML model). The more funding OpenAI has the more balls they can draw, and thus the more likely they are to draw a Winning Ball. Giving OpenAI $30M increases their chance to draw a Winning Ball; though that increase must be small if they have access to much more funding than $30M (without a super impressive ML model).

Ofer Mar 16, 2023, 9:16 PM
3 points
1 ∶ 0
in reply to: Dustin Moskovitz’s comment on: What happened to the OpenPhil OpenAI board seat?
I think this type of ML research (i.e. trying to train groundbreaking neural networks) is pretty messy and unpredictable; and money and luck are fungible to some extent. It’s not like back in 2017 OpenAI’s researchers could perfectly predict which ML experiments would succeed, and how to turn $X of GPU hours into an impressive model that would allow them to raise >$X in the next round, with probability 1.

For example, suppose OpenAI’s researchers ran some expensive experiment in 2017, and did not get impressive results. They then need to decide whether to give up on that particular approach, or just tweak some hyperparameters and run another such experiment. The amount of remaining funding they have at that point may determine their decision.

Ofer Mar 16, 2023, 2:03 PM
9 points
3 ∶ 0
in reply to: Dustin Moskovitz’s comment on: What happened to the OpenPhil OpenAI board seat?

I give effectively 0% of the probability mass to OpenAI not starting up.

I think an important question here is whether OpenAI would have reached a critical level of success that is necessary for, say, convincing Microsoft to throw $1B at them—before exhausting their runway—if OpenPhil had not recommended the $30M grant in 2017.

[...] I think the most likely counterfactual is that the time to the next fundraising (2019) and creation of the for-profit entity would have been shortened (due to less financial runway).

This seems reasonable. On the other hand, if OpenAI had not reached a critical level of success as a non-profit org, I think it is not obvious that their for-profit spinoff would have succeeded in raising investments that are sufficient to get them to a critical level of success. They would have probably needed to compete with many other for-profit AI startups for investments.

Ofer Mar 15, 2023, 1:14 PM
11 points
6 ∶ 0
on: Shutting Down the Lightcone Offices

E.g. recently I’ve had multiple pretty influential people ping me to delete or threaten moderation action against the authors of posts and comments talking about: How OpenAI doesn’t seem to take AI Alignment very seriously, [...]

This is surprising to me. Can you provide a link to the relevant post/comment?

Ofer Mar 8, 2023, 7:08 PM
4 points
1 ∶ 0
on: 3 Basic Steps to Reduce Personal Liability as an Org Leader

I know most are hesitant to start an actual non-profit since that is more expensive and time-consuming, but at the least, you can form an LLC.

It may be better to seek fiscal sponsorship from a relevant existing non-profit org.

Ofer Mar 6, 2023, 2:41 PM
19 points
18 ∶ 4
in reply to: HaydnBelfield’s comment on: Nick Bostrom should step down as Director of FHI
That’s a very unusual use of the word “reused” that would predictably mislead readers of the OP who have not read Bostrom’s apology. Bostrom quoted the entire 26-year-old email as is (excluding the first and last lines with the name of the person he was replying to and his own name and email address).