Mau

Karma: 2,783

Mau 31 Jan 2023 23:20 UTC
6 points
1 ∶ 0
in reply to: RobBensinger’s comment on: My highly personal skepticism braindump on existential risk from artificial intelligence.
Fair! Sorry for the slow reply, I missed the comment notification earlier.

I could have been clearer in what I was trying to point at with my comment. I didn’t mean to fault you for not meeting an (unmade) challenge to list all your assumptions—I agree that would be unreasonable.

Instead, I meant to suggest an object-level point: that the argument you mentioned seems pretty reliant on a controversial discontinuity assumption—enough that the argument alone (along with other, largely uncontroversial assumptions) doesn’t make it “quite easy to reach extremely dire forecasts about AGI.” (Though I was thinking more about 90%+ forecasts.)

(That assumption—i.e. the main claims in the 3rd paragraph of your response—seems much more controversial/non-obvious among people in AI safety than the other assumptions you mention, as evidenced by researchers criticizing it and researchers doing prosaic AI safety work.)

Mau 30 Jan 2023 3:45 UTC
7 points
2 ∶ 0
on: Protect Our Future’s Crypto Politics
Thanks for doing this! I think the most striking part of what you found is the donations to representatives who sit on the subcommittee that oversees the CFTC (i.e. the House Agriculture Subcommittee on Commodity Exchanges, Energy, and Credit), so I wanted to look into this more. From a bit of Googling:
- It looks like you’re right that Rep. Delgado sits on (and is even the Chair of) this subcommittee.
- On the other hand, it looks like Rep. Spanberger doesn’t actually sit on this subcommittee, and hasn’t done so since 2021. In other words, she hasn’t been on this subcommittee since the Protect our Future PAC was founded (which was early 2022).
  - Spanberger’s Wikipedia page does say she sits on this subcommittee, but Spanberger’s own website (both now and before the most recent elections) and the Wikipedia page on the subcommittee don’t list her as having served on this subcommittee in the 2021-23 or current session of Congress.
    The latter source also says she served on the subcommittee before 2021, so my guess is that Spanberger’s Wikipedia page just has outdated info.
  - (I don’t think this settles doubts about the PAC.)
I didn’t spend much time on this, so I very possibly missed or misinterpreted things.

Mau 27 Jan 2023 1:56 UTC
2 points
2 ∶ 0
in reply to: RobBensinger’s comment on: My highly personal skepticism braindump on existential risk from artificial intelligence.
Nitpick: doesn’t the argument you made also assume that there’ll be a big discontinuity right before AGI? That seems necessary for the premise about “extremely novel software” (rather than “incrementally novel software”) to hold.

Mau 18 Jan 2023 1:39 UTC
53 points
25 ∶ 7
in reply to: Gideon Futerman’s comment on: Doing EA Better

why they would want to suggest to these bunch of concerned EAs how to go about trying to push for the ideas that Buck disagrees with better

My guess was that Buck was hopeful that, if the post authors focus their criticisms on the cruxes of disagreement, that would help reveal flaws in his and others’ thinking (“inasmuch as I’m wrong it would be great if you proved me wrong”). In other words, I’d guess he was like, “I think you’re probably mistaken, but in case you’re right, it’d be in both of our interests for you to convince me of that, and you’ll only be able to do that if you take a different approach.”

[Edit: This is less clear to me now—see Gideon’s reply pointing out a more recent comment.]

Mau 18 Jan 2023 1:16 UTC
50 points
28 ∶ 15
in reply to: Gideon Futerman’s comment on: Doing EA Better
I interpreted Buck’s comment differently. His comment reads to me, not so much like “playing the man,” and more like “telling the man that he might be better off playing a different game.” If someone doesn’t have the time to write out an in-depth response to a post that takes 84 minutes to read, but they take the time to (I’d guess largely correctly) suggest to the authors how they might better succeed at accomplishing their own goals, that seems to me like a helpful form of engagement.

Mau 15 Dec 2022 0:01 UTC
8 points
3 ∶ 0
in reply to: Chris Leong’s comment on: What Rethink Priorities General Longtermism Team Did in 2022, and Updates in Light of the Current Situation
This seems helpful, though I’d guess another team that’s in more frequent contact with AI safety orgs could do this for significantly lower cost, since they’ll be starting off with more of the needed info and contacts.

Mau 4 Dec 2022 2:55 UTC
12 points
9 ∶ 1
on: Byrne Hobart & Dwarkesh Patel on hardcore believers, monasteries, and effective altruism
Thanks for sharing! The speakers on the podcast might not have had the time to make detailed arguments, but I find their arguments here pretty uncompelling. For example:
- They claim that “many belief systems they have a way of segregating and limiting the impact of the most hardcore believers.” But (at least from skimming) their evidence for this seems to be just the example of monastic traditions.
- A speaker claims that “the leaders who take ideas seriously don’t necessarily have a great track record.” But they just provide a few cherry-picked (and dubious) examples, which is a pretty unreliable way of assessing a track record.
  - Counting Putin a “man of ideas” because he made a speech with lots of historical references—while ignoring the many better leaders who’ve also made history-laden speeches—looks like especially egregious cherry-picking.
So I think, although their conclusions are plausible, these arguments don’t pass enough of an initial sanity check to be worth lots of our attention.

Mau 16 Nov 2022 3:33 UTC
38 points
11 ∶ 1
on: I think EA will make it through stronger
Thanks for writing this! I want to push back a bit. There’s a big middle ground between (i) naive, unconstrained welfare maximization and (ii) putting little to no emphasis on how much good one does. I think “do good, using reasoning” is somewhat too quick to jump to (ii) while passing over intermediate options, like:
- “Do lots of good, using reasoning” (roughly as in this post)
- “be a good citizen, while ambitiously working towards a better world” (as in this post)
- “maximize good under constraints or with constraints incorporated into the notion of goodness”
There are lots of people out there (e.g. many researchers, policy professionals, entrepreneurs) who do good using reasoning; this community’s concern for scope seems rare, important, and totally compatible with integrity. Given the large amounts of good you’ve done, I’d guess you’re sympathetic to considering scope. Still, it seems important enough to include in the tagline.

Also, a nitpick:

now it’s obvious that the idea of maximizing goodness doesn’t work in practice—we have a really clear example of where trying to do that fails (SBF if you attribute pure motives to him); as well as a lot of recent quotes from EA luminaries saying that you shouldn’t do that

This feels a bit fast; the fact that this example had to include a (dubious) “if” clause means it’s not a really clear example, and maximizing goodness is compatible with constraints if we incorporate constraints into our notion of goodness (just by the fact that any behavior can be thought of as maximizing some notion of goodness).

(Made minor edits.)

Mau 9 Oct 2022 9:25 UTC
5 points
0 ∶ 0
on: Deliberate practice for research?
Chris Olah’s research taste exercises might be examples of this.

Mau 6 Oct 2022 22:57 UTC
7 points
0 ∶ 0
on: Warning Shots Probably Wouldn’t Change The Picture Much
Readers might be interested in the comments over here, especially Daniel K.’s comment:

The only viable counterargument I’ve heard to this is that the government can be competent at X while being incompetent at Y, even if X is objectively harder than Y. The government is weird like that. It’s big and diverse and crazy. Thus, the conclusion goes, we should still have some hope (10%?) that we can get the government to behave sanely on the topic of AGI risk, especially with warning shots, despite the evidence of it behaving incompetently on the topic of bio risk despite warning shots.

Or, to put it more succinctly: The COVID situation is just one example; it’s not overwhelmingly strong evidence.

Mau 30 Sep 2022 20:07 UTC
3 points
0 ∶ 0
in reply to: Stephen McAleese’s comment on: Estimating the Current and Future Number of AI Safety Researchers
[Edit: I think the following no longer makes sense because the comment it’s responding to was edited to add explanations, or maybe I had just missed those explanations in my first reading. See my other response instead.]

Thanks for this. I don’t see how the new estimates incorporate the above information. (The medians for CSER, Leverhulme, and FLI seem to still be at 5 each.)

(Sorry for being a stickler here—I think it’s important that readers get accurate info on how many people are working on these problems.)

Mau 30 Sep 2022 2:35 UTC
3 points
0 ∶ 0
in reply to: Stephen McAleese’s comment on: Estimating the Current and Future Number of AI Safety Researchers
Thanks for the updates!

I have it on good word that CSET has well under 10 safety-focused researchers, but fair enough if you don’t want to take an internet stranger’s word for things.

I’d encourage you to also re-estimate the counts for CSER, Leverhulme, and the Future of Life Institute.
- CSER’s list of team members related to AI lists many affiliates, advisors, and co-founders but only ~3 research staff.
- The Future of Life Institute seems more focused on policy and field-building than on research; they don’t even have a research section on their website. Their team page lists ~2 people as researchers.
- Of the 5 people listed in Leverhulme’s relevant page, one of them was already counted for CSER, and another one doesn’t seem safety-focused.
I also think the number of “Other” is more like 4.

Mau 29 Sep 2022 4:22 UTC
9 points
0 ∶ 0
on: Estimating the Current and Future Number of AI Safety Researchers
Thanks for posting, seems good to know these things! I think some of the numbers for non-technical research should be substantially lower—enough that an estimate of ~55 non-technical safety researchers seems more accurate:
- CSET isn’t focused on AI safety; maybe you could count a few of their researchers (rather than 10).
- I think SERI and BERI have 0 full-time non-technical research staff (rather than 10 and 5).
- As far as I’m aware, the Leverhulme Centre for the Future of Intelligence + CSER only have at most a few non-technical researchers in total focused on AI safety (rather than 10 & 5). Same for FLI (rather than 5).
- I hear Epoch has ~3 FTEs (rather than 10).
- GoodAI’s research roadmap makes no mention of public/corporate policy or governance, so I’d guess they have at most a few non-technical safety-focused researchers (rather than 10).
If I didn’t mess up my math, all that should shift our estimate from 93 to ~42. Adding in 8 from Rethink (going by Peter’s comment) and 5 (?) from OpenPhil, we get ~55.

Mau 28 Sep 2022 19:34 UTC
11 points
1 ∶ 0
on: Optimism, AI risk, and EA blind spots
Thanks for posting! I’m sympathetic to the broad intuition that any one person being at the sweet spot where they make a decisive impact seems unlikely , but I’m not sold on most of the specific arguments given here.

Recall that there are decent reasons to think goal alignment is impossible—in other words, it’s not a priori obvious that there’s any way to declare a goal and have some other agent pursue that goal exactly as you mean it.

I don’t see why this is the relevant standard. “Just” avoiding egregiously unintended behavior seems sufficient for avoiding the worst accidents (and is clearly possible, since humans do it often).

Also, I don’t think I’ve heard these decent reasons—what are they?

Recall that engineering ideas very, very rarely work on the first try, and that if we only have one chance at anything, failure is very likely.

It’s also unclear that we only have one chance at this. Optimistically (but not that optimistically?), incremental progress and failsafes can allow for effectively multiple chances. (The main argument against seems to involve assumptions of very discontinuous or abrupt AI progress, but I haven’t seen very strong arguments for expecting that.)

Recall that getting “humanity” to agree on a good spec for ethical behavior is extremely difficult: some places are against gene drives to reduce mosquito populations, for example, despite this saving many lives in expectation.

Agree, but also unclear why this is the relevant standard. A smaller set of actors agreeing on a more limited goal might be enough to help.

Recall that there is a gigantic economic incentive to keep pushing AI capabilities up, and referenda to reduce animal suffering in exchange for more expensive meat tend to fail.

Yup, though we should make sure not to double-count this, since this point was also included earlier (which isn’t to say you’re necessarily double-counting).

Recall that we have to implement any solution in a way that appeals to the cultural sensibilities of all major and technically savvy governments on the planet, plus major tech companies, plus, under certain circumstances, idiosyncratic ultra-talented individual hackers.

This also seems like an unnecessarily high standard, since regulations have been passed and enforced before without unanimous support from affected companies.

Also, getting acceptance from all major governments does seem very hard but not quite as hard as the above quotes makes it sound. After all, many major governments (developed Western ones) have relatively similar cultural sensibilities, and ambitious efforts to prevent unilateral actions have previously gotten very broad acceptance (e.g. many actors could have made and launched nukes, done large-scale human germline editing, or maybe done large-scale climate engineering, but to my knowledge none of those have happened).

The we-only-get-one-shot idea applies on this stage too.

Yup, though this is also potential double-counting.

Mau 23 Sep 2022 1:48 UTC
10 points
4 ∶ 0
in reply to: Linch’s comment on: AGI Battle Royale: Why “slow takeover” scenarios devolve into a chaotic multi-AGI fight to the death
+1 on this being a relevant intuition. I’m not sure how limited these scenarios are—aren’t information asymmetries and commitment problems really common?

Mau 21 Sep 2022 3:57 UTC
3 points
0 ∶ 0
in reply to: Alex P’s comment on: Why AGIs utility can’t outweigh humans’ utility?
Ah sorry, I had totally misunderstood your previous comment. (I had interpreted “multiply” very differently.) With that context, I retract my last response.

By “satisfaction” I meant high performance on its mesa-objective (insofar as it has one), though I suspect our different intuitions come from elsewhere.

it should robustly include “building copy of itself”

I think I’m still skeptical on two points:
- Whether this is significantly easier than other complex goals
  - (The “robustly” part seems hard.)
- Whether this actually leads to a near-best outcome according to total preference utilitarianism
  - If satisfying some goals is cheaper than satisfying others to the same extent, then the details of the goal matter a lot
    As a kind of silly example, “maximize silicon & build copies of self” might be much easier to satisfy than “maximize paperclips & build copies of self.” If so, a (total) preference utilitarian would consider it very important that agents have the former goal rather than the latter.

Mau 21 Sep 2022 1:21 UTC
1 point
0 ∶ 0
in reply to: Alex P’s comment on: Why AGIs utility can’t outweigh humans’ utility?

getting the “multiply” part right is sufficient, AI will take care of the “satisfaction” part on its own

I’m struggling to articulate how confused this seems in the context of machine learning. (I think my first objection is something like: the way in which “multiply” could be specified and the way in which an AI system pursues satisfaction are very different; one could be an aspect of the AI’s training process, while another is an aspect of the AI’s behavior. So even if these two concepts each describe aspects of the AI system’s objectives/behavior, that doesn’t mean its goal is to “multiply satisfaction.” That’s sort of like arguing that a sink gets built to be sturdy, and it gives people water, therefore it gives people sturdy water—we can’t just mash together related concepts and assume our claims about them will be right.)

(If you’re not yet familiar with the basics of machine learning and this distinction, I think that could be helpful context.)

Mau 20 Sep 2022 22:51 UTC
2 points
0 ∶ 0
in reply to: Alex P’s comment on: Why AGIs utility can’t outweigh humans’ utility?
I can’t, but I’m not sure I see your point?

Mau 20 Sep 2022 19:12 UTC
2 points
0 ∶ 0
in reply to: Alex P’s comment on: Why AGIs utility can’t outweigh humans’ utility?
Maybe, but is “multiply” enough to capture the goal we’re talking about? “Maximize total satisfaction” seems much harder to specify (and to be robustly learned) - at least I don’t know what function would map states of the world to total satisfaction.

Mau 20 Sep 2022 18:38 UTC
2 points
0 ∶ 0
in reply to: HjalmarWijk’s comment on: Why AGIs utility can’t outweigh humans’ utility?
I think this gets a lot right, though

As I am not a preference utilitarian I strongly reject this identification.

While this does seem to be part of the confusion of the original question, I’m not sure (total) preference vs. hedonic utilitarianism is actually a crux here. An AI system pursuing a simple objective wouldn’t want to maximize the number of satisfied AI systems; it would just pursue its objective (which might involve relatively few copies of itself with satisfied goals). So highly capable AI systems pursuing very simple or random goals aren’t only bad by hedonic utilitarian lights; they’re also bad by (total) preference utilitarian lights (not to mention “common sense ethics”).