Ofer

Karma: 2,159

Last nontrivial update: 2024-02-01.

Send me anonymous feedback: https://docs.google.com/forms/d/1qDWHI0ARJAJMGqhxc9FHgzHyEFp-1xneyl9hxSMzJP0/viewform

Any type of feedback is welcome, including arguments that a post/comment I wrote is net negative.

I’m interested in ways to increase the EV of the EA community by mitigating downside risks from EA related activities. Without claiming originality, I think that:

Complex cluelessness is a common phenomenon in the domains of anthropogenic x-risks and meta-EA (due to an abundance of crucial considerations). It is often very hard to judge whether a given intervention is net-positive or net-negative.
The EA community is made out of humans. Humans’ judgement tends to be influenced by biases and self-deception. That is a serious source of risk, considering the previous point.
- Some potential mitigations involve improving some aspects of how EA funding works, e.g. with respect to conflicts of interest. Please don’t interpret my interest in such mitigations as accusations of corruption etc.

Feel free to reach out by sending me a PM. I’ve turned off email notifications for private messages, so if you send me a time sensitive PM consider also pinging me about it via the anonymous feedback link above.

Ofer 23 Jun 2018 15:29 UTC
7 points
0 ∶ 0
on: Informational hazards and the cost-effectiveness of open discussion of catastrophic risks
In this FLI podcast episode, Andrew Critch suggested handling a potentially dangerous idea like a software update rollout procedure, in which the update is distributed gradually rather than to all customers at once:

… I would tell you the same thing I would tell anyone who discovers a potentially dangerous idea, which is not to write a blog post about it right away.

I would say, find three close, trusted individuals that you think reason well about human extinction risk, and ask them to think about the consequences and who to tell next. Make sure you’re fair-minded about it. Make sure that you don’t underestimate the intelligence of other people and assume that they’ll never make this prediction

...

Then do a rollout procedure. In software engineering, you developed a new feature for your software, but it could crash the whole network. It could wreck a bunch of user experiences, so you just give it to a few users and see what they think, and you slowly roll it out. I think a slow rollout procedure is the same thing you should do with any dangerous idea, any potentially dangerous idea. You might not even know the idea is dangerous. You may have developed something that only seems plausibly likely to be a civilizational scale threat, but if you zoom out and look at the world, and you imagine all the humans coming up with ideas that could be civilizational scale threats.

...

If you just think you’ve got a small chance of causing human extinction, go ahead, be a little bit worried. Tell your friends to be a little bit worried with you for like a day or three. Then expand your circle a little bit. See if they can see problems with the idea, see dangers with the idea, and slowly expand, roll out the idea into an expanding circle of responsible people until such time as it becomes clear that the idea is not dangerous, or you manage to figure out in what way it’s dangerous and what to do about it, because it’s quite hard to figure out something as complicated as how to manage a human extinction risk all by yourself or even by a team of three or maybe even ten people. You have to expand your circle of trust, but, at the same time, you can do it methodically like a software rollout, until you come up with a good plan for managing it. As for what the plan will be, I don’t know. That’s why I need you guys to do your slow rollout and figure it out.
What links here?
- Gregory Lewis's comment on Informational hazards and the cost-effectiveness of open discussion of catastrophic risks by turchin (3 Jul 2018 22:47 UTC; 7 points)

Ofer 24 Jan 2019 11:21 UTC
4 points
0 ∶ 0
on: If slow-takeoff AGI is somewhat likely, don’t give now
When planning how to donate, it seems very important to consider the impact of market returns increasing due to progress in AI. But I think more considerations should be taken into account before drawing the conclusion in the OP.
For each specific cause, we should estimate the curve over time of EV-per-additional-dollar-invested-in-2019-and-used-now (given an estimate of market returns over time). As Richard pointed out, for reducing AI x-risk, it is not obvious we will have time to effectively use the money we invest today if we wait for too long (so “the curve” for AI safety might be sharply decreasing).
Here is another consideration I find relevant for AI x-risk: in slow takeoff worlds more people are likely to become worried about x-risk from AI (e.g. after they see that the economy has doubled in the past 4 years and that lots of weird things are happening). In such worlds, it might be the case that a very small fraction of the money that will be allocated for reducing AI x-risk would be donated by people who are currently worried about AI x-risk. This consideration might make us increase the weight of fast takeoff worlds.
On the other hand, maybe in slow takeoff worlds there is generally a lot more that could be done for reducing x-risk from AI (especially if slow takeoff correlates with longer timelines), which suggests we increase the weight of slow takeoff worlds.
If you think a fast takeoff is more likely, it probably makes more sense to either invest your current capital in tooling up as an AI alignment researcher, or to donate now to your favorite AI alignment organization (Larks’ 2018 review (a) is a good starting point here).
I just wanted to note that some of the research directions for reducing AI x-risk, including ones that seem relevant in fast takeoff worlds, are outside of the technical AI alignment field (for example, governance/policy/strategy research).

Ofer 14 Feb 2019 6:15 UTC
41 points
0 ∶ 0
on: Three Biases That Made Me Believe in AI Risk
If people here would appreciate it, I would be happy to write one or more posts on object-level arguments as to why I am now sceptical of AI risk. Let me know in the comments.
I would like to read about these arguments.

Ofer 16 Feb 2019 10:38 UTC
2 points
0 ∶ 0
on: Evidence on good forecasting practices from the Good Judgment Project: an accompanying blog post
Thank you for writing this.
Is the one-hour training module publicly available?
One might worry that training improves accuracy by motivating the trainees to take their jobs more seriously. Indeed it seems that the trained forecasters made more predictions per question than the control group, though they didn’t make more predictions overall. Nevertheless it seems that the training also had a direct effect on accuracy as well as this indirect effect.34
I could not find results like the ones in Table 4 in which the Brier scores are based only on the first answer that forecasters provide. Allowing forecasters to update their forecasts as frequently as they want (while reporting average daily Brier scores) plausibly gives an advantage to the forecasters who are willing to invest more time in their task.
The paper from which Table 4 is from stated that “Training was a significant predictor of average number of forecasts per question for year 1 and the number of forecasts per question was also significant predictor of accuracy (measured as mean standardized Brier score)”. Consider Table 10 in the paper that shows “Forecasts per question per user by year”. Notice that in year 3 the forecasters that got training made 4.27 forecasts per question, while forecasters that did not get training made only 1.90 forecasts per question. The paper includes additional statistical analyses related to this issue (unfortunately I don’t have the combination of time and background in statistics to understand them all).

Ofer 16 Feb 2019 15:26 UTC
1 point
0 ∶ 0
in reply to: kokotajlod’s comment on: Evidence on good forecasting practices from the Good Judgment Project: an accompanying blog post
For sure, forecasters who devoted more effort to it tended to make more accurate predictions. It would be surprising if that wasn’t true!
I agree. But I am not referring to an extra effort that makes a person provide a better forecast (e.g. by spending more time looking for arguments), but rather an extra effort that allows one to improve their average daily Brier scores by simply using new public information that was not available when the question was first presented (e.g. new poll results).

Ofer 29 Mar 2019 10:54 UTC
1 point
0 ∶ 0
on: $100 Prize to Best Argument Against Donating to the EA Hotel
To verify I’m a real person that will in fact award $100, find me on FB here.
The link appears to be broken.
(my interest here is in finding/popularizing ways for users of this forum to easily prove their identity to other users in case they wish to).

Ofer 30 Mar 2019 14:51 UTC
3 points
0 ∶ 0
on: Severe Depression and Effective Altruism
Just an additional point to consider:
If you (and therefore other people similar to you) decide to act in a way that causes a lot of harm/suffering to yourself or your family, and you wouldn’t have acted in that way had you never heard about EA, then that would create a causal link between “Alice learns about EA” and “Alice or her family suffer”. From a utilitarian perspective, such a causal link seems extremely harmful (e.g. making it less likely that a random talented/rich person would end up being involved in EA related efforts).
So this is an argument in favor of NOT making such decisions.

Ofer 1 Apr 2019 5:09 UTC
16 points
1 ∶ 0
on: The Case for the EA Hotel
There’s an additional argument in favor of the EA Hotel idea which I find very compelling (I’ve read it on this forum in a comment that I can’t find; EDIT: it was this comment by the user Agrippa—the following is not at all a precise description of the original comment and contains extra things that Agrippa might not agree with):
A lot of people are optimizing to get money as an instrumental goal and funders don’t always have a great way to evaluate how much a person that is asking for money is “EA-aligned” (for any reasonable definition of that term).
The willingness to travel and live for a while in a building with people that are excited about EA probably correlates with “being EA-aligned”.
So supporting people via funding their residency in a place like the EA Hotel seems to allow an implicit weak vetting mechanism that doesn’t exist when funding people directly.

Ofer 10 Apr 2019 10:31 UTC
1 point
0 ∶ 0
in reply to: Greg_Colbourn’s comment on: The Case for the EA Hotel
Yes, thanks.

Ofer 4 Sep 2019 15:36 UTC
10 points
0 ∶ 0
on: Are we living at the most influential time in history?
Interesting post!
But even if we restricted ourselves to a uniform prior over the first 10% of civilisation’s history, the prior would still be as low as 1 in 100,000.
Why should we use a uniform distribution as a prior? If I had to bet on which century would be the most influential for a random alien civilization, my prior distribution for “most influential century” would be a monotonically decreasing function.

Ofer 18 Sep 2019 16:02 UTC
2 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Ask Me Anything!
(ii) trying to map the Yudkowsky/Bostrom arguments, which were made before the deep learning paradigm, onto actual progress in machine learning, and finding them hard to fit well. Going into this properly would require a lot more discussion though!)
I’d be happy to read more about this point.
If we end up with powerful deep learning models that optimize a given objective extremely well, the main arguments in Superintelligence seem to go through.
(If we end up with powerful deep learning models that do NOT optimize a given objective, it seems to me plausible that x-risks from AI are more severe, rather than less.)
[EDIT: replaced “a specified objective function” with “a given objective”]

Ofer 19 Sep 2019 15:33 UTC
2 points
0 ∶ 0
in reply to: SoerenMind’s comment on: Are we living at the most influential time in history?
So your argument doesn’t seems to save existential risk work. The only way to get a non-trivial P(high influence | long future) with your prior seems to be by conditioning on an additional observation “we’re extremely early”. As I argued here, that’s somewhat sketchy to do.
As you wrote, the future being short “doesn’t necessarily imply that xrisk work doesn’t have much impact because the future might just be short in terms of people in our anthropic reference class”.
Another thought that comes to mind is that there may exist many evolved civilizations that their behavior is correlated with our behavior. If so, us deciding to work hard on reducing x-risks means it’s more likely that those other civilizations would also decide—during early centuries—to work hard on reducing x-risks.

Ofer 9 Oct 2019 13:45 UTC
25 points
0 ∶ 0
on: Long-Term Future Fund: August 2019 grant recommendations
It might be useful to get some opinions/intuitions from fund managers on the following question:
How promising is the most promising application that you ended up not recommending a grant for? How would a counterfactually valid grant for that application compare to the $439,197 that was distributed in this round, in terms of EV per dollar?

Ofer 10 Oct 2019 8:33 UTC
16 points
0 ∶ 0
in reply to: Habryka’s comment on: Long-Term Future Fund: August 2019 grant recommendations
Thank you!
This suggests that at an additional counterfactually valid donation of $10,000 to the fund, donated prior to this grant round, would have had (if not saved for future rounds) about 60% of the cost-effectiveness of the $439,197 that was distributed.
It might be useful to understand how much more money the fund could have distributed before reaching a very low marginal cost-effectiveness. For example, if the fund had to distribute in this grant round a counterfactually valid donation of $5MM, how would the cost-effectiveness of that donation compare to that of the $439,197 that was distributed?

Ofer 14 Oct 2019 5:59 UTC
10 points
0 ∶ 0
on: The Future of Earning to Give
Interesting post!
Today, there’s almost enough money going into far future causes, so that vetting and talent constraints have become at least as important as funding.
This seems to rely on the assumption that existing prestigious orgs are asking for all the funding they can effectively use. My best guess is that these orgs tend to not ask for a lot more funding than what they predict they can get. One potential reason for this is that orgs/grant-seekers regard such requests as a reputational risk.
Here’s some supporting evidence for this, from this Open Phil blog post by Michael Levine (August 2019):
After conversations with many funders and many nonprofits, some of whom are our grantees and some of whom are not, our best model is that many grantees are constantly trying to guess what they can get funded, won’t ask for as much money as they should ask for, and, in some cases, will not even consider what they would do with some large amount because they haven’t seriously considered the possibility that they might be able to raise it.

Ofer 22 Oct 2019 6:41 UTC
1 point
0 ∶ 0
on: Conditional interests, asymmetries and EA priorities
My very tentative view is that we’re sufficiently clueless about the probability distribution of possible outcomes from “Risks posed by artificial intelligence” and other x-risks, that the ratio between [the value one places on creating a happy person] and [the value one places on helping a person who is created without intervention] should have little influence on the prioritization of avoiding existential catastrophes.

Ofer 24 Oct 2019 10:34 UTC
1 point
0 ∶ 0
in reply to: Michelle_Hutchinson’s comment on: A single person decides about funding for community builders world-wide
(unrelated to the OP)
You might well think that eg MIRI’s agenda should be more widely worked on, or that it would be better if MIRI had more sources of funding. But it doesn’t seem worrying that that isn’t case.
This consideration seems important and I couldn’t understand it (I’m talking about the general consideration, not its specific application to MIRI’s agenda). I’d be happy to read more about it.

Ofer 24 Oct 2019 18:47 UTC
1 point
0 ∶ 0
in reply to: JP Addison’s comment on: A single person decides about funding for community builders world-wide
The latter (not MIRI in particular).

Ofer 25 Oct 2019 17:05 UTC
2 points
0 ∶ 0
in reply to: Michelle_Hutchinson’s comment on: A single person decides about funding for community builders world-wide
Thanks for this helpful explanation!

Ofer 30 Oct 2019 20:36 UTC
1 point
0 ∶ 0
on: Reflections on EA Global London 2019 (Mrinank Sharma)
What’s Stopping Advanced Applications of AI?
In many cases, there are cultural issues (within an industry) about the application of algorithms to make crucial decisions. Whilst interpretability of systems would increase the buy in, there are also key issues with the quality of data, and the infrastructure to collect high quality data.
It is worth nothing that the barriers here seem to not be technical, so it is unclear how much of an impact technical research would have here.
Perhaps this model was proposed for certain domains? Maybe ones in which laws restrict applications, like driverless cars?
It doesn’t seem to me plausible for all domains (for example, it doesn’t seem to me plausible for language models and quantitative trading).