nonn

Karma: 303

nonn 7 May 2026 2:45 UTC
3 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Unfalsifiable stories of doom
sorry, I don’t check the forum much. And yeah, your revised picture sounds like what I meant.

Still interested in something like this, though might punt on it for a month or two (busy times), partly because I have to think about what makes sense to me/what my probabilities are. The above was just intended to illustrate the point about betting, I didn’t think much about the numbers. I’m definitely not an “AI 2027” person, but also definitely not an “AI 2070″ person re: medians.

If you know your numbers that would help—e.g. it sounds like you could suggest numbers that are a good deal to you, but also sound like an obviously good deal to me, despite my less-clear picture? You could also hash that if desired until I come back :P

nonn 28 Apr 2026 3:32 UTC
3 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Unfalsifiable stories of doom

the one above is not worth it for me

That’s what I meant by interest-indexed, unless that isn’t capturing your concern?

Also to be clear that was just an example possible bet, would have to think about what actually makes sense (2035 is somewhat near my median tho). My point was just that money isn’t valuable to me after crazy AI stuff already happened, so me winning has no value in a conventional bet & it has to be constructed giving money beforehand to be returned with interest + winnings.

Would be open to confirming identity, and/or likely have other folks who would be keen to make the same bet.

What is your P(existentially bad outcomes) in the next 10 years? As maybe a starting point for finding a bet that sounds good to you.

nonn 22 Apr 2026 0:36 UTC
3 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Unfalsifiable stories of doom
How are you operationalizing this? No matter the odds, it doesn’t make sense to make bets of the form
- heads we’re dead OR there’s infinite abundance + I get some money from you
- tails I give your some money
Maybe would be open to “you transfer 1k to me now (2026), I give you interest-indexed 2k in 2035” or whatever odds make sense. Though I understand you’d need to trust me and/or some trusted way to make sure I give it back

I also don’t think short timelines is actually cruxy to my argument above, which is mainly about their argument being wrong + pointing at other arguments for misalignment, not timelines.

nonn 20 Apr 2026 20:59 UTC
3 points
0 ∶ 0
on: Unfalsifiable stories of doom
The theoretical section seems weak, and basically just sidesteps all sorts of arguments like this. And theoretical arguments are a key crux imo

And their main theoretical argument doesn’t seem right either. They say “evolution only touches DNA whereas gradient descent touches the whole network”… but the way we ‘touch’ the whole network is via a simple loss function + default weight update rule (which would easily fit into DNA)! You could just as easily say DNA does this, by defining the algorithm/architecture/update process that creates/updates the brain.

They then imply we have much more fine-grained control than DNA/evolution could, but in fact we our current method implies very little fine control. Like yes, in principle, we have access to all the weights. If we understood more, maybe we could have an extremely-complicated loss function and sophisticated update process, and then DNA couldn’t code for something analogously sophisticated. But that’s not remotely what we do, and best practice is essentially “define some loss functions on vibes and see what works”.

If anything, it seems like DNA’s training algorithm exerts more fine-grain control & has a more complicated ‘loss function/update rule’ than gradient descent.

(And evolution had far more time to try out adaptions to novel behavior at ‘near-human capability’ and still failed on inner alignment, albeit unclear how to compare “tons of random random tries” v.s. “a few vibes-based tries”)

nonn 20 Apr 2026 19:47 UTC
11 points
1 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Unfalsifiable stories of doom
Not this person, but many AI risk arguments are necessarily logical rather than empirical—there are good reasons to believe the relevant behaviors won’t appear, or be trivially easy to counter (at least re: harmful outputs), until you have very capable systems.

Like, if I can construct a deceptive response-to-training strategy (but current models can’t), that’s enough evidence to be concerned future superhuman models might do similar deceptive alignment. Other concerns like inner optimizers (e.g. humans stopped being kid-maxxers at high capability, because our proxy decoupled from evolution’s target) might not show up, or change in character, as models become less limited. And even when you can demonstrate the behavior empirically, people dismiss it as overly-induced or a toy environment—which was the whole point, just to show plausibility not prove it.

More fundamentally: If I argue that a future thing logically implies certain risks arise, responding with “there’s no empirical evidence” is silly. Logical chains and structural arguments are still valid epistemic tools.

nonn 19 Mar 2026 17:03 UTC
5 points
0 ∶ 0
on: The case for AI safety capacity-building work
I think a major blocker to this kind of thing is that people feel like ‘it’s not a real career’ and worry what would happen if they tried to leave, or just didn’t see success in their fieldbuilding startup.

IMO this is very incorrect above a certain threshold of ability, especially for people already working in EA or AIS technical/policy/generalist roles. But it would be very helpful if your team could offer some stronger guarantees to these people!

Here’s one basic idea (common and probably far from optimal): ‘failed-fieldbuilding-attempt insurance’ - For people you think should do this, you agree to give a 5 year stipend of 2-5k/month if they try & fail & can’t find another decent job. Likely you wouldn’t even have to pay this out much, because most people that you’re excited to see try fieldbuilding are IMO incorrect about not being able to transition back. So in practice, you’d give them the stipend for a few months before they found a new job. And many of them would actually succeed & you’d pay nothing!

nonn 22 Dec 2025 18:54 UTC
1 point
0 ∶ 0
in reply to: AGB 🔸’s comment on: Insects Are Not Moderately Important
I think this is a good point about precise phrasing, but I think the argument still basically goes through that insects should be treated as extremely important in expectation. You can eliminate the two envelope problem by either make the numbers fixed/concrete, or you can use conditional probabilities.

Intuitively: suppose you thought there was an 50% chance you prevent a holocaust-level (10,000,000 lives) event happening to humans, but a 50% chance that this intervention would be completely useless. Alternately, you could do a normal intervention to save 1000 lives.

You could say “the normal intervention as a 50% chance to be ~infinitely more valuable than the holocaust-prevention thing”

But it’s obvious you should do the holocaust prevention thing. Because here it’s more obvious what the comparative/conditional stakes are. In one possible world, the ‘world you can affect’ is vastly larger, and that world should be prioritized.

Caveats: ignoring longtermist arguments, and the probability insects matter is << 50% imo

nonn 18 Nov 2025 9:38 UTC
1 point
0 ∶ 0
in reply to: Yarrow Bouchard 🔸’s comment on: nonn’s Shortform
I don’t know what you mean? You can look at existing interventions that primarily help very young people (neonatal or childhood vitamin supplementation) v.s. a comparably-effective interventions that target adults or older people (e.g. cash grants, schistosomiasis)

There are multiple GiveWell charities in both categories, so this is just saying you should weight towards the ones that target older folks by maybe a factor of 2x or more, v.s. what givewell says (they assume the world won’t change much)

nonn 18 Nov 2025 7:40 UTC
1 point
0 ∶ 0
in reply to: Yarrow Bouchard 🔸’s comment on: nonn’s Shortform
Assuming two interventions are around similarly effective in life-years saved, interventions saving old lives must (necessarily) save more lives in the short run. E.g. save 4 lives granting 10 life-years v.s. save 1 life granting 40 life years.

nonn 17 Nov 2025 20:22 UTC
3 points
0 ∶ 0
on: nonn’s Shortform
Some fraction of people who don’t work on AI risk cite “wanting to have more certainty of impact”as their main reason. But I think many of them are running the same risk anyway: namely, that what they do won’t matter because transformative AI will make their work irrelevant, or dramatically lower value.

This is especially obvious if they work on anything that primarily returns value after a number of years. E.g. building an academic career or any career where most impact is realized later, working toward policy changes, some movement-building things, etc.

But also applies somewhat to things like nutrition or vaccination or even preventing deaths, where most value is realized later (by having better life outcomes, or living an extra 50 years). Though this category does still have certainty of impact, just the amount of impact might be cut by whatever fraction of worlds are upended in some way by AI. And this might affect what they should prioritize… e.g. they should prefer saving old lives over young ones, if the interventions are pretty-close on naive effectiveness measures.

nonn 11 Nov 2025 22:32 UTC
13 points
3 ∶ 0
in reply to: Henry Howard🔸’s comment on: Humanizing Expected Value
Feels like there’s some line where your numbers are getting so tiny and speculative that many other considerations start dominating, like “are your numbers actually right?” E.g. I’d be pretty skeptical of many proposed ”.000001% of huge number” interventions (especially skeptical on the on the .000001% side).

In practice, the line could be where “are your numbers actually right” starts becoming the dominant consideration. At that point, proving your numbers are plausible is the main challenge that needs to be overcome—and is honestly where I suspect most people’s anti-low-probabilities intuitions come from in the first place.

nonn 7 Sep 2023 18:44 UTC
1 point
0 ∶ 0
on: New Princeton course on longtermism
Very cool!

random thought: could include some of Yoshua Bengio’s or Geoffrey Hinton’s writings/talks on AI risks concerns in week 10 (& could include Lecun for counterpoint to get all 3), since they’re very-well cited academics & Turing Award Winners for deep learning

I haven’t looked through their writings/talks to find the most directly relevant, but some examples: https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/ https://yoshuabengio.org/2023/06/24/faq-on-catastrophic-ai-risks/

nonn 4 Aug 2023 18:24 UTC
7 points
1 ∶ 0
in reply to: quinn’s comment on: University EA Groups Need Fixing
My experience is that it’s more that group leaders & other students in EA groups might reward poor epistemics in this way.

And that when people are being more casual, it ‘fits in’ to say AI risk & people won’t press for reasons in those contexts as much, but would push if you said something unusual.

Agree my experience with senior EAs in the SF Bay was often the opposite–I was pressed to explain why I’m concerned about AI risk & to respond to various counterarguments.

nonn 27 Dec 2022 23:37 UTC
2 points
0 ∶ 0
in reply to: Ariel Simnegar 🔸’s comment on: A Case for Voluntary Abortion Reduction
No, though maybe you’re using the word “intrinsically” differently? For the (majority) consequentialist part of my moral portfolio: The main intrinsic bad is suffering, and wellbeing (somewhat broader) is intrinsically good.

I think any argument about creating people/etc is instrumental—will they or won’t they increase wellbeing? They can both potentially contain suffering/wellbeing themselves, and affect the world in ways that affect wellbeing/suffering now & in the future. This includes effects before they are born (e.g. on women’s lives). TBH given your above arguments, I’m confused about the focus on abortion—it seems like you should be just as opposed to people choosing not to have children, and focus on encouraging/supporting people having kids.

For now, I think the ~main thing that matters is from a total-view longtermist perspective is making it through “the technological precipice”, where risks of permanent loss of sentient life/our values is somewhat likely, so other total-view longtermist arguments flow through effects on this + influencing for good trajectory arguably. Since abortion access seems good for civilization trajectory (women can have children when the want, don’t have their lives & health derailed, etc), more women involved in the development of powerful technology probably makes these fields more cautious/less rash, fewer ‘unwanted children’ [probably worse life outcomes], etc. Then abortion access seems good.

Maybe related: in general when maximizing, I think it’s probably best to finding the most important 1-3 things, then focus on those things. (e.g. for temp of my house, focus on temp of thermostat + temp of outside + insulation quality, ignore body heat & similar small things)

nonn 27 Dec 2022 22:04 UTC
4 points
1 ∶ 0
in reply to: Ariel Simnegar 🔸’s comment on: A Case for Voluntary Abortion Reduction
I don’t think near-term population is helpful for long-term population or wellbeing, e.g. in >10,000 years from now. More likely negative effect than positive effect imo, especially if the mechanism of trying to increase near-term population is to restrict abortion (this is not a random sample of lives!)

I also think it seems bad for general civilization trajectory (partially norm-damaging, but mostly just direct effects on women & children), probably bad for ability to make investments in resilience & be careful with powerful new technology. These seem like the most important effects from a longtermist perspective, so I think abortion-restriction is bad from a total-longtermist perspective.

nonn 27 Dec 2022 21:51 UTC
1 point
0 ∶ 0
in reply to: lastmistborn’s comment on: A Case for Voluntary Abortion Reduction
I guess I did mean aggregate in the ‘total’ well-being sense. I just feel pretty far from neutral about creating people who will live wonderful lives, and also pretty strongly disagree with the belief that restricting abortion will create more total well-being in the long run (or short tbh).

For total-view longtermism, I think the most important things are ~civilization is on a good trajectory, people are prudent/careful with powerful new technology, the world is lower conflict, investments are made to improve resilience to large catastrophes, etc. Restricting abortion seems kinda bad for several of those things, and positive for none. So it seems like total-view longtermism, even ignoring all other reasons to think this, says abortion-restriction is bad.

I guess part of this is a belief that in the long-run, the number of morally-valuable lives & total wellbeing (e.g. in a 10 million years) is very uncorrelated or anti-correlated with near-term world population. (though I also think restricting abortion is one of the worst ways to go about increasing near-term population, even for those who do think near-term & very-long-term are pretty positively correlated)

nonn 27 Dec 2022 1:35 UTC
4 points
1 ∶ 0
in reply to: lastmistborn’s comment on: A Case for Voluntary Abortion Reduction

abortion is morally wrong is a direct logical extension of a longtermist view that highly values maximizing the number of people on assumption that the average existing persons life will have positive value

I’m a bit confused by this statement. Is a world where people don’t have access to abortion likely to have more aggregate well-being in the very long run? Naively, it feels like the opposite to me

To be clear I don’t think it’s worth discussing abortion at length, especially considering bruce’s comment. But I really don’t think the number of people currently existing says much about well-being in the very long run (arguably negatively correlated). And even if you wanted to increase near-term population, reducing access to abortion is a very bad way to that, with lots of negative knock-on effects.

nonn 1 Sep 2022 22:03 UTC
1 point
0 ∶ 0
in reply to: Linch’s comment on: Open EA Global
Agree that was a weird example.

Other people around the group (e.g. many of the non-Stanford people who sometimes came by & worked at tech companies) are better examples. Several weren’t obviously promising at the time, but are doing good work now.

nonn 29 Jul 2022 19:48 UTC
2 points
0 ∶ 0
in reply to: Callum Dyer’s comment on: Leaning into EA Disillusionment
typo, imo. (in my opinion)

nonn 22 Jul 2022 18:36 UTC
27 points
1 ∶ 0
on: Leaning into EA Disillusionment
I’m somewhat more pessimistic that disillusioned people have useful critiques, at least on average. EA asks people to swallow a hard pill “set X is probably the most important stuff by a lot”, where X doesn’t include that many things. I think this is correct (i.e. the set will be somewhat small), but it means that a lot of people’s talents & interests probably aren’t as [relatively] valuable as they previously assumed.

That sucks, and creates some obvious & strong motivated reasons to lean into not-great criticisms of set X. I don’t even think this is conscious, just vague ‘feels like this is wrong’ when people say [thing I’m not the best at/dislike] is the most important. This is not to say set X doesn’t have major problems

They might more often have useful community critiques imo, e.g. more likely to notice social blind spots that community leaders are oblivious to.

Also, I am concerned about motivated reasoning within the community, but don’t really know how to correct for this. I expect the most-upvoted critiques will be the easy-to-understand plausible-sounding ones that assuage the problem above or social feelings, but not the correct ones about our core priorities. See some points here: https://forum.effectivealtruism.org/posts/pxALB46SEkwNbfiNS/the-motivated-reasoning-critique-of-effective-altruism