Isaac Dunn

Karma: 657

Isaac Dunn May 12, 2025, 8:21 PM
4 points
2 ∶ 0
in reply to: titotal’s comment on: Joseph Lemien’s Shortform
Actually, computer science conferences are peer reviewed. They play a similar role as journals in other fields. I think it’s just a historical curiosity that it’s conferences rather than journals that are the prestigious places to publish in CS!
Of course, this doesn’t change the overall picture of some AI work and much AI safety work not being peer reviewed.

Isaac Dunn Mar 29, 2025, 8:37 AM
1 point
0 ∶ 0
in reply to: Avik Garg’s comment on: Extinction is probably only 10^10 times worse than one random death
Thanks, this back and forth is very helpful. I think I’ve got a clearer idea about what you’re saying.
I think I disagree that it’s reasonable to assume that there will be a fixed N = 10^35 future lives, regardless of whether it ends up Malthusian. If it ends up not Malthusian, I think I’d expect the number of people in the future to be far less than whatever the max imposed by resource constraints is, ie much less than 10^35.
So I think that changes the calculation of E[saving one life], without much changing E[preventing extinction], because you need to split out the cases where Malthusianism is true vs false.
E[saving one life] is 1 if Malthusianism is true, or something fraction of the future if Malthusianism is false, but if it’s false, then we should expect the future to be much smaller than 10^35. So the EV will be much less than 10^35.
E[preventing extinction] is 10^35 if Malthusianism is true, and much less if it’s false. But you don’t need that high a credence to get an EV around 10^35.
So I guess all that to say that I think your argument is right and also action relevant, except I think the future is much smaller in non-Malthusian worlds, so there’s a somewhat bigger gap than “just” 10^10. I’m not sure how much bigger.
What do you think about that?

Isaac Dunn Mar 28, 2025, 7:27 PM
1 point
0 ∶ 0
in reply to: Avik Garg’s comment on: Extinction is probably only 10^10 times worse than one random death
Ah nice, thanks for explaining! I’m not following all the calculations still, but that’s on me, and I think they’re probably right.
But I don’t think your argument is actually that relevant to what we should do, even if it’s right. That’s because we don’t care about how good our actions are as a fraction/multiple of what our other options are. Instead, we just want to do whatever leads to the best expected outcomes.
Suppose there was a hypothetical world where there was a one in ten chance the total figure population was a billion, and 90% chance the population was two. And suppose we have two options: save one person, or save half the people.
In that case, the expected value of saving half the people would be 0.9*1 + 0.1*500,000,000 = about 50,000,001. That’s compared to the expected value of 1 of saving one person. Imo, this is a strong reason for picking the “save half the people option”.
But the expected fraction of people saved by the options is quite different. The “save half” option always results in half being saved. And the expected value of the “save one” option is also very close to half: 0.9*0.5 + 0.1*1/1,000,000,000. Even though the two interventions look very similar from this perspective, I think it’s basically irrelevant—expected value is the relevant thing.
What do you think? I might well have made a mistake, or misunderstood still.

Isaac Dunn Mar 28, 2025, 6:18 PM
3 points
0 ∶ 0
on: Extinction is probably only 10^10 times worse than one random death
I think your calculations must be wrong somewhere, although I can’t quite follow them well enough to see exactly where.
If you have a 10% credence in Malthusianism, then the expected badness of extinction is 0.1*10^35, or whatever value you think a big future is. That’s still a lot closer to 10^35 times the badness of one death than 10^10 times.
Does that seem right?

Isaac Dunn Feb 25, 2025, 12:01 PM
1 point
0 ∶ 0
in reply to: Davidmanheim’s comment on: How confident are you that it’s preferable for America to develop AGI before China does?
Agree coin flip is unacceptable! Or even much less than coin flip is still unacceptable.

Isaac Dunn Feb 24, 2025, 8:52 PM
1 point
0 ∶ 0
in reply to: Davidmanheim’s comment on: How confident are you that it’s preferable for America to develop AGI before China does?
I agree with this comment, but I interpreted your original comment as implying a much greater degree of certainty of extinction assuming ASI is developed than you might have intended. My disagree vote was meant to disagree with the implication that it’s near certain. If you think it’s not near certain it’d cause extinction or equivalent, then it does seem worth considering who might end up controlling ASI!

Isaac Dunn Feb 23, 2025, 4:37 PM
2 points
1 ∶ 1
in reply to: Davidmanheim’s comment on: How confident are you that it’s preferable for America to develop AGI before China does?
You’re stating it as a fact that “it is” a game of chicken, i.e. that it’s certain or very likely that developing ASI will cause a global catastrophe because of misaligned takeover. It’s an outcome I’m worried about, but it’s far from certain, as I see it. And if it’s not certain, then it is worth considering what people would do with aligned AI.

Isaac Dunn Feb 21, 2025, 6:44 AM
6 points
0 ∶ 0
in reply to: Ivan Burduk’s comment on: Ivan Burduk’s Quick takes
I heard reports of it getting out of sync or being out of date in some way. For example, a room change on Swapcard not being reflected in the Google calendar. I haven’t tried it myself, and I haven’t heard anything less vague, sorry.

Isaac Dunn Feb 20, 2025, 6:41 AM
4 points
1 ∶ 0
in reply to: Ivan Burduk’s comment on: Ivan Burduk’s Quick takes
I think that the Google calendar syncing is at least a bit buggy for now, FYI. Agree good news though!

Isaac Dunn Feb 15, 2025, 5:32 AM
16 points
1 ∶ 0
on: Quick nudge to apply to the LTFF grant round (closing on Saturday)
When will the next round likely be?

Isaac Dunn Dec 23, 2024, 6:20 PM
3 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: GiveWell may have made 1 billion dollars of harmful grants, and Ambitious Impact incubated 8 harmful organisations via increasing factory-farming?
Thanks Vasco! :)

I agree that thinking about other moral theories is useful for working out what utilitarianism would actually recommend.

That’s an interesting point re increasing the total amount of killing, I hadn’t considered that! But I was actually picking up on your comment which seemed to say something more general—that you wouldn’t intrinsically take into account whether an option involved (you) killing people, you’d just look at the consequences (and killing can lead to worse consequences, including in indirect ways, of course). But it sounds like maybe your response to that is you’re not worried about moral uncertainty / you’re sure about utilitarianism / you don’t have any reason to avoid killing people, other than the (normally very significant) utilitarian reasons not to kill?

Isaac Dunn Dec 23, 2024, 12:43 AM
3 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: GiveWell may have made 1 billion dollars of harmful grants, and Ambitious Impact incubated 8 harmful organisations via increasing factory-farming?
Do you not worry about moral uncertainty? Unless you’re certain about consequentialism, surely you should put some weight on avoiding killing even if it maximises impartial welfare?

Isaac Dunn Nov 28, 2024, 5:14 PM
3 points
0 ∶ 0
in reply to: Jim Buhler’s comment on: The ‘Dog vs Cat’ cluelessness dilemma (and whether it makes sense)
You’re welcome! N=1 though, so might be worth seeing what other people think too.

Isaac Dunn Nov 28, 2024, 2:04 PM
4 points
2 ∶ 0
on: The ‘Dog vs Cat’ cluelessness dilemma (and whether it makes sense)
For what it’s worth, although I do think we are clueless about the long-run (and so overall) consequences of our actions, the example you’ve given isn’t intuitively compelling to me. My intuition wants to say that it’s quite possible that the cat vs dog decision ends up being irrelevant for the far future / ends up being washed out.
Sorry, I know that’s probably not what you want to hear! Maybe different people have different intuitions.

Isaac Dunn Oct 23, 2024, 11:03 PM
1 point
0 ∶ 0
in reply to: Ebenezer Dukakis’s comment on: chinscratch’s Quick takes
I don’t think OpenAI’s near term ability to make money (e.g. because of the quality of its models) is particularly relevant now to its valuation. It’s possible it won’t be in the lead in the future, but I think OpenAI investors are betting on worlds where OpenAI does clearly “win”, and the stickiness of its customers in other worlds doesn’t really affect the valuation much.

So I don’t agree that working on this would be useful compared with things that contribute to safety more directly.

How much do you think customers having 0 friction to switching away from OpenAI would reduce its valuation? I think it wouldn’t change it much, less than 10%.

(Also note that OpenAI’s competitors are incentivised to make switching cheap, e.g. Anthropic’s API is very similar to OpenAI’s for this reason.)

Isaac Dunn Oct 23, 2024, 4:14 PM
1 point
1 ∶ 0
in reply to: Ebenezer Dukakis’s comment on: chinscratch’s Quick takes
I think investors want to invest in OpenAI so badly almost entirely because it’s a bet on OpenAI having better models in the future, not because of sticky customers. So it seems that the effect of this on OpenAI’s cost of capital would be very small?

Isaac Dunn Aug 21, 2024, 4:06 PM
1 point
0 ∶ 0
on: Results of an informal survey on AI grantmaking
Interesting exercise, thanks! The link to view the questions doesn’t work though. It says:
The form AI Grantmaking Priorities Survey is no longer accepting responses.
Try contacting the owner of the form if you think that this is a mistake.

Isaac Dunn Jul 2, 2024, 6:31 PM
1 point
0 ∶ 0
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Quick takes
Interesting!
I think my worry is people who don’t think they need advice about what the future should look like. When I imagine them making the bad decision despite having lots of time to consult superintelligent AIs, I imagine them just not being that interested in making the “right” decision? And therefore their advisors not being proactive in telling them things that are only relevant for making the “right” decision.
That is, assuming the AIs are intent aligned, they’ll only help you in the ways you want to be helped:
- Thoughtful people might realise the importance of getting the decision right, and might ask “please help me to get this decision right” in a way that ends up with the advisors pointing out that AI welfare matters and the decision makers will want to take that into account.
- But unthoughtful or hubristic people might not ask for help in that way. They might just ask for help in implementing their existing ideas, and not be interested in making the “right” decision or in what they would endorse on reflection.
I do hope that people won’t be so thoughtless as to impose their vision of the future without seeking advice, but I’m not confident.

Isaac Dunn Jul 2, 2024, 1:51 PM
42 points
11 ∶ 1
on: LLMs cannot usefully be moral patients
I agree that the text an LLM outputs shouldn’t be thought of as communicating with the LLM “behind the mask” itself.
But I don’t agree that it’s impossible in principle to say anything about the welfare of a sentient AI. Could we not develop some guesses about AI welfare by getting a much better understanding of animal welfare? (For example, we might learn much more about when brains are suffering, and this could be suggestive of what to look for in artificial neural nets)
It’s also not completely clear to me what the relationship between the sentient being “behind the mask” is, and the “role-played character”, especially if we imagine conscious, situationally-aware future models. Right now, it’s for sure useful to see the text output by an LLM as simulating a character, which is nothing to do with the reality of the LLM itself, but could that be related to the LLM not being conscious of itself? I feel confused.
Also, even if it was impossible in principle to evaluate the welfare of a sentient AI, you might still want to act differently in some circumstances:
- Some ethical views see creating suffering as worse than creating the same amount of pleasure.
- Empirically, in animals, it seems to me that the total amount of suffering is probably more than the total amount of pleasure. So we might worry that this could also be the case for ML models.

Isaac Dunn Jul 2, 2024, 9:50 AM
20 points
0 ∶ 0
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Quick takes
Why does “lock-in” seem so unlikely to you?
One story:
- Assume AI welfare matters
- Aligned AI concentrates power in a small group of humans
- AI technology allows them to dictate aspects of the future / cause some “lock in” if they want. That’s because:
  - These humans control the AI systems that have all the hard power in the world
  - Those AI systems will retain all the hard power indefinitely; their wishes cannot be subverted
  - Those AI systems will continue to obey whatever instructions they are given indefinitely
- Those humans decide to dictate some or all of what the future looks like, and lots of AIs end up suffering in this future because their welfare isn’t considered by the decision makers.
  - (Also, the decision makers could pick a future which isn’t very good in other ways.)
You could imagine AI welfare work now improving things by putting AI welfare on the radar of those people, so they’re more likely to take AI welfare into account when making decisions.
I’d be interested in which step of this story seems implausible to you—is it about AI technology making “lock in” possible?