Derek Shiller

Karma: 2,652

Derek Shiller 29 Jan 2026 20:42 UTC
4 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Digital Consciousness Model Results and Key Takeaways

AIs could have negligible welfare (in expectation) even if they are conscious. They may not be sentient even if they are conscious, or have negligible welfare even if they are sentient. I would say the (expected) total welfare of a group (individual welfare times population) matters much more for its moral consideration than the probability of consciousness of its individuals. Do you have any plans to compare the individual (expected hedonistic) welfare of AIs, animals, and humans? You do not mention this in the section “What’s next”.

This is an important caveat. While our motivation for looking at consciousness is largely from its relation to moral status, we don’t think that establishing that AIs were conscious would entail that they have significant states that counted strongly one way or the other for our treatment of them, and establishing that they weren’t conscious wouldn’t entail that we should feel free to treat them however we like.

We think that it estimates of consciousness still play an important practical role. Work on AI consciousness may help us to achieve consensus on reasonable precautionary measures and motivate future research directions with a more direct upshot. I don’t think the results of this model can be directly plugged into any kind of BOTEC, and should be treated with care.

Do you have any ideas for how to decide on the priors for the probability of sentience? I agree decisions about priors are often very arbitrary, and I worry they will have significantly different implications.

We favored a ¹⁄₆ prior for consciousness relative to every stance and we chose that fairly early in the process. To some extent, you can check the prior against what you update to on the basis of your evidence. Given an assignment of evidence strength and an opinion about what it should say about something that satisfies all of the indicators, you can backwards infer the prior needed to update to the right posterior. That prior is basically implicit in your choices about evidential strength. We didn’t explicitly set our prior this way, but we would probably have reconsidered our choice of ¹⁄₆ if it was giving really implausible results for humans, chickens, and ELIZA across the board.

The right conclusion would be that the respondants have no idea about the right exponent, or how to weight the various models because they would not be able to adequately justify their picks.

There is a tension here between producing probabilities we think are right and producing probabilities which could reasonably act as a consensus conclusion. I have my own favorite stance, and I think I have good reason for it, but I didn’t try to convince anyone to give it more weight in our aggregation. Insofar as we’re aiming in the direction of something that could achieve broad agreement, we don’t want to give too much weight to our own views (even if we think we’re right). Unfortunately,among people with significant expertise in this area, there is broad and fairly fundamental disagreement. We think that it is still valuable to shoot for consensus, even if that means everyone will think it is flawed (by giving too much weight to different stances.)

Derek Shiller 13 Jan 2026 14:46 UTC
4 points
0 ∶ 0
on: One reason why we might be especially uncertain about AI welfare

This last part carries a lot of weight; a simulacrum, when dormant in the superposition from which it can be sampled, is nonexistent. A simulacrum only exists during the discrete processing event which correlates with its sampling.

There seems to me to be a sensible view on which a simulacrum exists to the extent that computations relevant to making decisions on its behalf are carried out, regardless of what the token sampler chooses. This would suggest that there could conceivably be vast numbers of different simulacra instantiated even in a single forward pass.

One odd upshot of requiring the token sampler is that in contexts in which no tokens get sampled (prefill, training) you can get all of the same model computations but have no simulacra at all.

Derek Shiller 20 Nov 2025 19:36 UTC
4 points
0 ∶ 0
in reply to: Zach Stein-Perlman’s comment on: Longview’s Emerging Challenges Fund can effectively absorb marginal funding
I find this distinction kind of odd. If we care about what digital minds we produce in the future, what should we be doing now?

I expect that what minds we build in large numbers in the future will be largely depend on how we answer a political question. The best way to prepare now for influencing how we as a society answer that question (in a positive way) is to build up a community with a reputation for good research, figure out the most important cruxes and what we should say about them, create a better understanding of what we should actually be aiming for, initiate valuable relationships with potential stakeholders based on mutual respect and trust, creating basic norms about human-ai relationships, and so on. To me, that looks like engaging with whether near-future AIs are conscious (or have other morally important traits) and working with stakeholders to figure out what policies make sense at what times.

Though I would have thought the posts you highlighted as work you’re more optimistic about fit squarely within that project, so maybe I’m misunderstanding you.

Derek Shiller 7 Nov 2025 17:51 UTC
10 points
4 ∶ 0
on: Incomparability Implies Nihilism
I think this is basically right (I don’t think the upshot is that incomparability implies nihilism, but rather the moral irrelevance of most choices). I don’t really understand why this is a reason to reject incomparability. If values are incomparable, it turns out that the moral implications are quite different from what we thought. Why change your values rather than your downstream beliefs about morally appropriate action?

Derek Shiller 21 Aug 2025 12:25 UTC
2 points
0 ∶ 0
in reply to: MinusGix’s comment on: Deep Democracy as a promising target for positive AGI futures
Thanks for the suggestion. I’m interested in the issue of dealing with threats in bargaining.

I don’t think we ever published anything specifically on the defaults issue.

We were focused on allocating a budget that respects the priorities of different worldviews. The central thing we were encountering was that we started by taking the defaults to be the allocation you get by giving everyone their own slice of the total budget and spending it as they wanted. Since there are often options that are well-suited to each different worldview, there is no way to get good compromises. Everyone is happier with the default than any adjustment of it. (More here.) On the other hand, if you switch the default to be some sort of neutral 0 value (assuming that can be defined), then you will get compromises, but many bargainers would rather that they just be given their own slice of the total budget to allocate.

I think the importance of defaults comes through just by playing around with some numbers. Consider the difference between setting the default to be the status quo trajectory we’re currently on and setting the default to be the worst possible outcome. Suppose we have two worldviews, one of which cares about suffering in all other people linearly, and the other of which is very locally focused and doesn’t care about immense suffering elsewhere. For the two worldviews, relative to the status quo, option A might give (worldview1: 2,worldview2: 10) value and option B might give (4,6) value. Against this default, option B has a higher product (24 vs 20) and is preferred by Nash bargaining. However, relative to the worst possible value default, option A might give (10,002, 12) and option B (10,004, 8), then option A would be preferred to option B (~120k vs 80k).

Derek Shiller 20 Aug 2025 19:31 UTC
9 points
3 ∶ 0
in reply to: tylermjohn’s comment on: Deep Democracy as a promising target for positive AGI futures
We implemented a Nash bargain solution in our moral parliament and I came away the impression that the results of Nash bargaining are very sensitive to your choice of defaults and for plausible defaults true bargains can be pretty rare. Anyone who is happy with defaults gets disproportionate bargaining power. One default might be ‘no future at all’, but that’s going to make it hard to find any bargain with the anti-natalists. Another default might be ‘just more of the same’, but again, someone might like that and oppose any bargain that deviates much. Have you given much thought to picking the right default against which to measure people’s preferences? (Or is the thought that you would just exclude obstinate minorities?)

Derek Shiller 9 Aug 2025 17:23 UTC
4 points
1 ∶ 1
on: Why I’m A Fanatic!

Keeping the world around probably does that, so you should donate to Longtermist charities (especially because they potentially increase the number of people ever born, thus giving more people a chance of getting into heaven).

I often get the sense that people into fanaticism think that it doesn’t much change what they actually should support. That seems implausible to me. Maybe you should support longtermist causes. (You probably have to contort yourself to justify giving any money to shrimp welfare.) But I would think the longtermist causes you should support will also be fairly different from ‘mainstream’ causes, and look rather weird close up. You don’t really care if the species colonizes the stars and the future is full of happy people living great lives. If some sort of stable totalitarian hellscape offers a marginally better (but still vanishingly small) chance of producing infinite value, that is where you should put your money.

Maybe the best expected value would be to tile the universe with computers trying to figure out the best way to produce infinite value under every conceivable metaphysical scheme consistent with what we know and run them all until the heat death of the universe before trying to act. Given that most people are almost certainly not going to do that, you might think that we shouldn’t be looking to build an aligned AI, we should want to build a fanatical AI.

Has your fanaticism changed your mind much about what is worth supporting?

Derek Shiller 10 Jul 2025 21:36 UTC
4 points
2 ∶ 0
on: Fish Feel Pain

But even a 10% chance that fish feel pain—and that we annually painfully slaughter a population roughly ten times the number of humans who have ever lived—is enough to make it a serious issue. Given the mind-bending scale of the harm we inflict on fish, even a modest chance that they feel pain is enough.

Completely in agreement here.

And while it’s possible that evolution produced some kind of non-conscious signal that produces identical behavior to pain, such a thing is unlikely. If a creature didn’t feel pain, it’s unlikely it would respond to analgesics, seek out analgesic drugs, and get distracted by bodily damage.

This is where I would disagree. I expect moderately-complicated creatures would develop traits like these under evolutionary pressures (except seeking out analgesic drugs). The question then is how likely is it that the best / only / easiest-to-evolve way to produce this slate of behaviors involves having a conscious experience with the relevant pain profile.

We know that human brains have undergone massive changes since our most recent common ancestor with fish, that terrestrial environments place very different demands on our bodies, that human beings have an unparalleled behavioral flexibility to address injuries, etc. so it is plausible that we do have fairly different nociceptive faculties. It seems to me like a pretty open question precisely how neurologically or algorithmically similar our faculties are and how similar they would need to be to for fish to qualify as having pain. The fact that we can’t even tell how important the cortex is for pain in humans seems like strong evidence that we shouldn’t be too confident about attributing pain to fish. We just know so little. Of course, we shouldn’t be confident about denying it to them either, but much confidence either way seems unjustifiable.

Derek Shiller 1 Jul 2025 20:43 UTC
4 points
0 ∶ 0
in reply to: David Mathers🔸’s comment on: Debate: Depopulation Matters
I would think the trend would also need to be evenly distributed. If some groups have higher-than-replacement birth rates, they will simply come to dominate over time.

Derek Shiller 24 Jun 2025 18:50 UTC
14 points
3 ∶ 0
in reply to: Bentham's Bulldog’s comment on: Debate: Morality is Objective
I think of moral naturalism as a position where moral language is supposed to represent things, and it represents certain natural things. The view I favor is a lot closer to inferentialism: the meaning of moral language is constituted by the way it is used, not what it is about. (But I also don’t think inferentialism is quite right, since I’m not into realism about meaning either.)

I guess I don’t quite see what your puzzlement is with morality. There are moral norms which govern what people should do. Now, you might deny there in fact are such things, but I don’t see what’s so mysterious.

Another angle on the mystery: it is possible that there are epistemic norms, moral norms, prudential norms, and that’s it. But if you’re a realist, it seems like it should also be possible that there are hundreds of other kinds of norms that we’re completely unaware of, such that we act in all sorts of wrong ways all the time. Maybe there are special norms governing how you should brush your teeth (that have nothing to do with hygiene or our interests), or how to daydream. Maybe these norms hold more weight than moral norms, in something like the way moral norms may hold more weight than prudential norms. If you’re a non-naturalist, then apart from trust in a loving God, I’m not sure how you address this possibility. But it also seems absurd that I should have to worry about such things.

Derek Shiller 24 Jun 2025 16:44 UTC
25 points
7 ∶ 0
on: Debate: Morality is Objective
I consider myself a pretty strong anti-realist, but I find myself accepting a lot of the things you take to be problems for anti-realism. For instance:

But lots of moral statements just really don’t seem like any of these. The wrongness of slavery, the holocaust, baby torture, stabbing people in the eye—it seems like all these things really are wrong and this fact doesn’t depend on what people think about it.

I think that these things really are wrong and don’t depend on what people think about it. But I also think that that statement is part of a language game dictated by complex norms and expectations. The significance of thought experiments. The need to avoid inconsistency. The acceptance of implications. The reliance on gut evaluations. Endorsement of standardly accepted implications. Etc. I live my life according to those norms and expectations, and they lead me to condemn slavery and think quite poorly of slavers and say things like ‘slavery was a terrible stain on our nation’. I don’t feel inclined to let people off the hook by virtue of having different desires. I’m quite happy with a lot of thought and talk that looks pretty objective.

I’m an anti-realist because I have no idea what sort of thing morality could be about that would justify the norms and expectations that govern our thoughts about morality. Maybe this is a version of the queerness argument. There aren’t any sorts of entities or relations that seem like appropriate truth-makers for moral claims. I have a hard time understanding what they might be such that I would have any inclination to shift what I care about were I to learn that the normative truths themselves were different (holding fixed all of the things that currently guide my deployment of moral concepts). If my intuitions about cases were the same, if all of the theoretical virtues were the same, if the facts in the world were the same, but an oracle were to tell me that moral reality were different in some way—turns out, baby torture is good! -- I wouldn’t be inclined to change my moral views at all. If I’m not inclined to change my views except when guided by things like gut feelings, consistency judgments, etc. then I don’t see how anything about the world can be authoritative in the way that realism should require.
What links here?
- Richard Y Chappell🔸's comment on Debate: Morality is Objective by Bentham's Bulldog (24 Jun 2025 18:19 UTC; 28 points)

Derek Shiller 26 Apr 2025 20:10 UTC
5 points
2 ∶ 0
on: It’s time to say goodbye to “functionally equivalent artificial neurons”

I don’t think it’s even necessary to debate whether quantum phenomena manifest somehow at the macro level of the brain

You might think it is important that the facts about consciousness contribute to our beliefs about them in some way. Our beliefs about consciousness are surely a phenomenon of the macro level. So if our beliefs are somehow sensitive to the facts, and the facts consist of quantum effects, we should expect those quantum effects to generate some marcoscopic changes.

This is the sticking point for me with quantum theories: there doesn’t seem to be any obvious mechanism for general quantum level truths to exert the kinds of very targeted influences that would be necessary for them to explain our beliefs about consciousness. And if they don’t, then it seems like we’re left with beliefs insensitive to the truth, and that is deeply unintuitive. What do you think?

Derek Shiller 18 Apr 2025 17:02 UTC
17 points
8 ∶ 1
in reply to: Chris Leong’s comment on: casebash’s Shortform
Also, it is worrying if the optimists easily find financial opportunities that depend on them not changing their minds. Even if they are honest and have the best of intentions, the disparity in returns to optimism is epistemically toxic.

Derek Shiller 14 Apr 2025 14:21 UTC
4 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Rethink Priorities’ Cross-Cause Cost-Effectiveness Model: Introduction and Overview
Yeah, that’s right. Some kinds of mitigation will increase risks later (e.g. a pause), and the model doesn’t accommodate such nuance.

Derek Shiller 14 Apr 2025 14:19 UTC
4 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Rethink Priorities’ Digital Consciousness Project Announcement

Could you link the most relevant piece you are aware of? What do you mean by “independently”? Under hedonism, I think the probability of consciousness only matters to the extent it informs the probability of valences experiences.

The idea is more aspirational. I’m not really sure of what to recommend in the field, but this is a pretty good overview: https://arxiv.org/pdf/2404.16696

Interesting! How?

Perhaps valence requires something like the assignment of weights to alternative possibilities. If you can look inside the AI and confirm that it is making decisions in a different way, you can conclude that it doesn’t have valenced experiences. Valence plausibly requires such assignments of weights (most likely with a bunch of other constraints), and the absence of one requirement is enough to disconfirm something. Of course, this sort of requirement is likely to be controversial, but it is less open to radically different views than consciousness itself.

Derek Shiller 28 Mar 2025 12:48 UTC
4 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: Rethink Priorities’ Digital Consciousness Project Announcement
Not at the moment. Consciousness is tricky enough as it is. The field is interested in looking more closely at valence independently of consciousness, given that valence seems more tractable and you could at least confirm that AIs don’t have valenced experience, but that lies a bit outside our focus for now.

Independently, we’re also very interested in how to capture the difference between positive and negative experiences in alien sorts of minds. It is often taken for granted based on human experience, but it isn’t trivial to say what it is.

Derek Shiller 19 Mar 2025 12:54 UTC
10 points
2 ∶ 0
in reply to: tylermjohn’s comment on: ...but is increasing the value of futures tractable?
This more or less conforms to why I think trajectory changes might be tractable, but I think the idea can be spelled out in a slightly more general way: as technology develops (and especially AI), we can expect to get better at designing institutions that perpetuate themselves. Past challenges to affecting a trajectory change come from erosion of goals due to random and uncontrollable human variation and the chaotic intrusion of external events. Technology may help us make stable institutions that can continue to promote goals for long periods of time.

Derek Shiller 17 Mar 2025 15:46 UTC
2 points
0 ∶ 0
in reply to: OllieBase’s comment on: Discussion Thread: Existential Choices Debate Week
Lots of people think about how to improve the future in very traditional ways. Assuming the world keeps operating under the laws it has been for the past 50 years, how do we steer it in a better direction?

I suppose I was thinking of this in terms of taking radical changes from technology development seriously, but not in the sense of long timelines or weird sources of value. Far fewer people are thinking about how to navigate a time when AGI becomes commonplace than are thinking about how to get to that place, even though there might not be a huge window of time between them.

Derek Shiller 17 Mar 2025 13:05 UTC
6 points
0 ∶ 2
on: Discussion Thread: Existential Choices Debate Week
People in general, and not just longtermist altruists, have reason to be concerned with extinction. It may turn out not to be a problem or not be solvable and so the marginal impact seems questionable here. In contrast, few people are thinking about how to navigate our way to a worthwhile future. There are many places where thoughtful people might influence decisions that effectively lock us into a trajectory.

Derek Shiller 14 Mar 2025 13:27 UTC
6 points
0 ∶ 0
in reply to: zdgroff’s comment on: Worrisome Trends for Digital Mind Evaluations

While secrecy makes it difficult or impossible to know if a system is a moral patient, it also prevents rogue actors from quickly making copies of a sentient system or obtaining a blueprint for suffering.

There is definitely a scenario in which secrecy works out for the best. Suppose AI companies develop recognizably conscious systems in secret that they don’t deploy, or deploy only with proper safeguards. If they had publicized how to build them, then it is possible that others would go ahead and be less responsible. The open source community raises some concerns. I wouldn’t want conscious AI systems to be open-sourced if it was feasible to run them on hardware anyone could afford. Still, I think the dangers here are relatively modest: it seems unlikely that rogue actors will run suffering AI on a large scale in the near future.

The scenario I’m most worried about is one in which the public favors policies about digital minds that are divorced from reality. Perhaps they grant rights and protections to all and only AIs that behave in sufficiently overt human-like ways. This would be a problem if human-likeness is not a good guide to moral status, either because many inhuman systems have moral status or many human-like systems lack it. Hiding the details from experts would make it more likely that we attribute moral status to the wrong AIs: AIs that trigger mind-recognizing heuristics from our evolutionary past, or AIs that the creators want us to believe are moral subjects.

2 and 3) If I understand correctly, the worry here is that AI multiplies at a speed that outpaces our understanding, making it less likely that humanity handles digital minds wisely. Some people are bullish on digital minds (i.e., think they would be good in and of themselves). Some also think other architectures would be more likely to be sentient than transformers. Wider exploration and AI-driven innovation plausibly have the effect of just increasing the population of digital minds. How do you weigh this against the other considerations?

My primary worry is getting ahead of ourselves and not knowing what to say about the first systems that come off as convincingly conscious. This is mostly a worry in conjunction with secrecy, but the wider we explore and the quicker we do it, the less time there will be for experts to process the details, even if they have access in principle. There are other worries for exploration even if we do have proper time to assess the systems we build, but it may make it more likely that we will see digital minds and I’m an optimist that any digital minds we create will be more likely to have good lives than bad.

If experts don’t know what to say about new systems, the public may make up its own mind. There could be knee-jerk reactions from skepticism in LLMs that are unwarranted in the context of new systems. Or there could be a credulity about the new systems that would be as inappropriate as it is for LLMs if you knew the details and not just the marketing.

The more experts are forced to throw up their hands and say “we’ve got no idea what to say about these things” the more likely we are to adopt commitments in ignorance that would turn out bad in the long run.

I think it may be quite hard to contract the moral circle once it includes agentic, social, and immortal AI systems. If we give them political and legal rights. If we welcome them into our homes and friend circles, etc. it may prove difficult to say “whoops, we were too taken in by your charms, no rights for you anymore!”. Similarly, if companies build an industry off the back of conscious AIs without recognizing it, they may be much more resistant to adopting new regulations that threaten their interests. The pressures against recategorizing existing AIs might also count against properly categorizing novel AIs, so if the justification for protecting new systems would undermine the justification for respecting existing systems, it may turn out to be a difficult argument to make.