Derek Shiller

• Derek Shiller 29 May 2026 11:32 UTC

  2 points

  0 ∶ 0

  on: Pope Leo XIV Just Made Digi­tal Minds Ad­vo­cacy a Lot Harder

  This tracks what I would strongly expect to be the church’s position, and I can’t imagine that we would get to the point of passing any kind of laws protecting the status of digital minds without the church having a position. This strikes me as reason more to work to figure out how our goals align with the church, rather than try to fight them on this.

• Derek Shiller 18 May 2026 13:31 UTC

  2 points

  0 ∶ 0

  on: Why only con­scious prefer­ences matter

  To manage its own limited processing power, the brain uses attention to focus on specific signals.

  What do you think about AI systems that don’t have the kinds of limitations we do, and that might not need to restrict focus as much, or might be able to devote cognitive resources to many different problems in parallel. It is plausible that they wouldn’t count as having attention in the way that Graziano thinks we do, and that they wouldn’t therefore satisfy his theory of consciousness. But it seems weird to suggest that the reason our preferences matter is because of our cognitive limitations, no?

• Derek Shiller 24 Apr 2026 14:18 UTC

  4 points

  0 ∶ 0

  in reply to: Vasco Grilo🔸’s comment on: In­com­pa­ra­bil­ity Im­plies Nihilism

  From your perspective now, given your uncertainty, you might have no reason to think that the drive changes things in a better or worse way, and so the expected value derived from taking the expected value of each possible consequence considered piecemeal and summed up might be neutral.

  I think if you accept some degree of objective chance, this might completely resolve the problem. (I explore this idea, without direct connection to incomparability, here: https://​​philpapers.org/​​rec/​​SHICAT-11)

  It is consistent with a purely subjective neutral expected value, however, that you might be quite sure that the results, if revealed in their full detail, would be incomparable.

  This creates an intuitive tension. There is something called the “Principle of Reflection”—it is a bit controversial, but very intuitive—that says roughly that if you know you would believe something if you were put into a better epistemic position, you should believe it now. The challenge Bulldog raises relies on taking reflection seriously and thinking that we would almost surely see incomparability if we looked closely at all the details, so we should accept incomparability now.

• Derek Shiller 23 Apr 2026 14:46 UTC

  4 points

  0 ∶ 2

  in reply to: Jim Buhler’s comment on: Dis­cus­sions of (p)sen­tience of small an­i­mals miss the point

  What I defend is that, in the absence of a good argument based on welfare ranges and not p(sentience), we don’t know if the welfare range of simpler animals is below or above the bar above which their welfare would dominate over that of more complex animals (not that it is below!).

  I suspect we disagree about how it is reasonable to apportion probabilities in the absence of empirical evidence. I think that conditioning on p(sentience) is sufficient to justify a non-negligible probability of similar levels of welfare range in the absence of any further empirical evidence. The empirical evidence that shrimp have small brains detracts from this probability, but not by much. That evidence makes it much more likely that they are not sentient at all, not much more likely that they only feel small feelings.

• Derek Shiller 21 Apr 2026 18:07 UTC

  4 points

  0 ∶ 0

  in reply to: Vasco Grilo🔸’s comment on: Dis­cus­sions of (p)sen­tience of small an­i­mals miss the point

  Which kind of functionalism? I am very sceptical of at least computational functionalism (CF). Any algorithm run by a digital computer can be executed with pen and paper (although it may take a super long time), and I have a hard time imagining how such process would itself be conscious.

  I’m skeptical of functionalism about consciousness (though I don’t know any alternative that fares better.) But functionalism about valence seems much harder to avoid. Maybe if you have a benevolent God? Or some sort of dualism? Otherwise, it seems to me that you’re going to be hard-pressed to explain why it is that the functional role of valence aligns with whatever properties constitute the fact that valence matters. Why is it that pain is bad and we avoid it, or pleasure is good and we seek it, if it is not just the case that the things we’re inclined to avoid count as pain, and the things we’re inclined to seek count as pleasure (very roughly).

• Derek Shiller 21 Apr 2026 15:32 UTC

  5 points

  1 ∶ 0

  in reply to: Vasco Grilo🔸’s comment on: Dis­cus­sions of (p)sen­tience of small an­i­mals miss the point

  However, I think you mean that the expected welfare range will be significant (for example, at least 1 % of that of humans) as long as there is one plausible model (for example, which gets 10 % weight) which predicts a significant welfare range (for example, 10 % of that of humans).

  Yeah, basically.

  I wonder whether electromagnetic (EM) field theories of consciousness could shed some light on it. I assume the maximum intensity of the EM fields generated by brain activity depends on the number of neurons, at least when assessed across species (there is little variance in the number of neurons of humans, which means the maximum intensity of EM fields may not vary much in humans).

  This makes me think we’re inclined towards a different basic perspective on the determinants of valence. This kind of sounds like you’re thinking of pain as a sort of physical magnitude, like weight or charge. Then it is reasonable to think it is likely to scale with size, so that much smaller brains are likely to have much smaller magnitudes. I’m more inclined towards functionalist interpretations of welfare, on which something like relative functional significance determines welfare levels. E.g. something’s attention-grabbing capacity helps to determine its welfare significance. In that case, you might be deeply skeptical that small animals have the right functional role at all, but once you grant they do, it is much more plausible that welfare ranges are similar to humans. However, for my point to be right, I think you just need to treat these kinds of functionalist views as in the running. You don’t have to be confident that they’re true.

• Derek Shiller 21 Apr 2026 13:48 UTC

  15 points

  4 ∶ 1

  on: Dis­cus­sions of (p)sen­tience of small an­i­mals miss the point

  . For the conclusion to follow, (ii) needs to be: evidence of a welfare range that is not too insignificant to crucially undermine the importance of their large numbers.

  I disagree with this. You don’t need evidence of a welfare range that is not too insignificant—you need a presumption of a reasonable probability of a welfare range that is not too small and no significant evidence against it. Without a theory of how you get from neurons to welfare, it seems to me to be very unreasonable to be confident that simpler brains most likely have much smaller welfare ranges. And we don’t have a good theory of how to get from neurons to welfare.

• Derek Shiller 2 Mar 2026 15:15 UTC

  13 points

  8 ∶ 0

  on: Don’t scale up your lifestyle

  One thing I’ve found for myself: it is easy to think of lifestyle changes across the board, but different areas have very different impacts on your budget. Opting for the slightly nicer apartment may end up costing way more than switching to the fancy vegan yogurt. Insofar as lifestyle creep is reasonable (it is probably healthy to be increasing your quality of life a bit over time), it is better to focus on the little things than the big ones.

• Derek Shiller 29 Jan 2026 20:42 UTC

  6 points

  0 ∶ 0

  in reply to: Vasco Grilo🔸’s comment on: Digi­tal Con­scious­ness Model Re­sults and Key Takeaways

  AIs could have negligible welfare (in expectation) even if they are conscious. They may not be sentient even if they are conscious, or have negligible welfare even if they are sentient. I would say the (expected) total welfare of a group (individual welfare times population) matters much more for its moral consideration than the probability of consciousness of its individuals. Do you have any plans to compare the individual (expected hedonistic) welfare of AIs, animals, and humans? You do not mention this in the section “What’s next”.

  This is an important caveat. While our motivation for looking at consciousness is largely from its relation to moral status, we don’t think that establishing that AIs were conscious would entail that they have significant states that counted strongly one way or the other for our treatment of them, and establishing that they weren’t conscious wouldn’t entail that we should feel free to treat them however we like.

  We think that it estimates of consciousness still play an important practical role. Work on AI consciousness may help us to achieve consensus on reasonable precautionary measures and motivate future research directions with a more direct upshot. I don’t think the results of this model can be directly plugged into any kind of BOTEC, and should be treated with care.

  Do you have any ideas for how to decide on the priors for the probability of sentience? I agree decisions about priors are often very arbitrary, and I worry they will have significantly different implications.

  We favored a ¹⁄₆ prior for consciousness relative to every stance and we chose that fairly early in the process. To some extent, you can check the prior against what you update to on the basis of your evidence. Given an assignment of evidence strength and an opinion about what it should say about something that satisfies all of the indicators, you can backwards infer the prior needed to update to the right posterior. That prior is basically implicit in your choices about evidential strength. We didn’t explicitly set our prior this way, but we would probably have reconsidered our choice of ¹⁄₆ if it was giving really implausible results for humans, chickens, and ELIZA across the board.

  The right conclusion would be that the respondants have no idea about the right exponent, or how to weight the various models because they would not be able to adequately justify their picks.

  There is a tension here between producing probabilities we think are right and producing probabilities which could reasonably act as a consensus conclusion. I have my own favorite stance, and I think I have good reason for it, but I didn’t try to convince anyone to give it more weight in our aggregation. Insofar as we’re aiming in the direction of something that could achieve broad agreement, we don’t want to give too much weight to our own views (even if we think we’re right). Unfortunately,among people with significant expertise in this area, there is broad and fairly fundamental disagreement. We think that it is still valuable to shoot for consensus, even if that means everyone will think it is flawed (by giving too much weight to different stances.)

Digi­tal Con­scious­ness Model Re­sults and Key Takeaways

Digi­tal Con­scious­ness Model launch events

• Derek Shiller 13 Jan 2026 14:46 UTC

  4 points

  0 ∶ 0

  on: One rea­son why we might be es­pe­cially un­cer­tain about AI welfare

  This last part carries a lot of weight; a simulacrum, when dormant in the superposition from which it can be sampled, is nonexistent. A simulacrum only exists during the discrete processing event which correlates with its sampling.

  There seems to me to be a sensible view on which a simulacrum exists to the extent that computations relevant to making decisions on its behalf are carried out, regardless of what the token sampler chooses. This would suggest that there could conceivably be vast numbers of different simulacra instantiated even in a single forward pass.

  One odd upshot of requiring the token sampler is that in contexts in which no tokens get sampled (prefill, training) you can get all of the same model computations but have no simulacra at all.

• Derek Shiller 20 Nov 2025 19:36 UTC

  4 points

  0 ∶ 0

  in reply to: Zach Stein-Perlman’s comment on: Longview’s Emerg­ing Challenges Fund can effec­tively ab­sorb marginal funding

  I find this distinction kind of odd. If we care about what digital minds we produce in the future, what should we be doing now?

  I expect that what minds we build in large numbers in the future will be largely depend on how we answer a political question. The best way to prepare now for influencing how we as a society answer that question (in a positive way) is to build up a community with a reputation for good research, figure out the most important cruxes and what we should say about them, create a better understanding of what we should actually be aiming for, initiate valuable relationships with potential stakeholders based on mutual respect and trust, creating basic norms about human-ai relationships, and so on. To me, that looks like engaging with whether near-future AIs are conscious (or have other morally important traits) and working with stakeholders to figure out what policies make sense at what times.

  Though I would have thought the posts you highlighted as work you’re more optimistic about fit squarely within that project, so maybe I’m misunderstanding you.

[Question] Cause prio cruxes in 2026?

• Derek Shiller 7 Nov 2025 17:51 UTC

  10 points

  4 ∶ 0

  on: In­com­pa­ra­bil­ity Im­plies Nihilism

  I think this is basically right (I don’t think the upshot is that incomparability implies nihilism, but rather the moral irrelevance of most choices). I don’t really understand why this is a reason to reject incomparability. If values are incomparable, it turns out that the moral implications are quite different from what we thought. Why change your values rather than your downstream beliefs about morally appropriate action?

• Derek Shiller 21 Aug 2025 12:25 UTC

  2 points

  0 ∶ 0

  in reply to: MinusGix’s comment on: Deep Democ­racy as a promis­ing tar­get for pos­i­tive AGI futures

  Thanks for the suggestion. I’m interested in the issue of dealing with threats in bargaining.

  I don’t think we ever published anything specifically on the defaults issue.

  We were focused on allocating a budget that respects the priorities of different worldviews. The central thing we were encountering was that we started by taking the defaults to be the allocation you get by giving everyone their own slice of the total budget and spending it as they wanted. Since there are often options that are well-suited to each different worldview, there is no way to get good compromises. Everyone is happier with the default than any adjustment of it. (More here.) On the other hand, if you switch the default to be some sort of neutral 0 value (assuming that can be defined), then you will get compromises, but many bargainers would rather that they just be given their own slice of the total budget to allocate.

  I think the importance of defaults comes through just by playing around with some numbers. Consider the difference between setting the default to be the status quo trajectory we’re currently on and setting the default to be the worst possible outcome. Suppose we have two worldviews, one of which cares about suffering in all other people linearly, and the other of which is very locally focused and doesn’t care about immense suffering elsewhere. For the two worldviews, relative to the status quo, option A might give (worldview1: 2,worldview2: 10) value and option B might give (4,6) value. Against this default, option B has a higher product (24 vs 20) and is preferred by Nash bargaining. However, relative to the worst possible value default, option A might give (10,002, 12) and option B (10,004, 8), then option A would be preferred to option B (~120k vs 80k).

• Derek Shiller 20 Aug 2025 19:31 UTC

  9 points

  3 ∶ 0

  in reply to: tylermjohn’s comment on: Deep Democ­racy as a promis­ing tar­get for pos­i­tive AGI futures

  We implemented a Nash bargain solution in our moral parliament and I came away the impression that the results of Nash bargaining are very sensitive to your choice of defaults and for plausible defaults true bargains can be pretty rare. Anyone who is happy with defaults gets disproportionate bargaining power. One default might be ‘no future at all’, but that’s going to make it hard to find any bargain with the anti-natalists. Another default might be ‘just more of the same’, but again, someone might like that and oppose any bargain that deviates much. Have you given much thought to picking the right default against which to measure people’s preferences? (Or is the thought that you would just exclude obstinate minorities?)

• Derek Shiller 9 Aug 2025 17:23 UTC

  4 points

  1 ∶ 1

  on: Why I’m A Fa­natic!

  Keeping the world around probably does that, so you should donate to Longtermist charities (especially because they potentially increase the number of people ever born, thus giving more people a chance of getting into heaven).

  I often get the sense that people into fanaticism think that it doesn’t much change what they actually should support. That seems implausible to me. Maybe you should support longtermist causes. (You probably have to contort yourself to justify giving any money to shrimp welfare.) But I would think the longtermist causes you should support will also be fairly different from ‘mainstream’ causes, and look rather weird close up. You don’t really care if the species colonizes the stars and the future is full of happy people living great lives. If some sort of stable totalitarian hellscape offers a marginally better (but still vanishingly small) chance of producing infinite value, that is where you should put your money.

  Maybe the best expected value would be to tile the universe with computers trying to figure out the best way to produce infinite value under every conceivable metaphysical scheme consistent with what we know and run them all until the heat death of the universe before trying to act. Given that most people are almost certainly not going to do that, you might think that we shouldn’t be looking to build an aligned AI, we should want to build a fanatical AI.

  Has your fanaticism changed your mind much about what is worth supporting?

• Derek Shiller 10 Jul 2025 21:36 UTC

  4 points

  2 ∶ 0

  on: Fish Feel Pain

  But even a 10% chance that fish feel pain—and that we annually painfully slaughter a population roughly ten times the number of humans who have ever lived—is enough to make it a serious issue. Given the mind-bending scale of the harm we inflict on fish, even a modest chance that they feel pain is enough.

  Completely in agreement here.

  And while it’s possible that evolution produced some kind of non-conscious signal that produces identical behavior to pain, such a thing is unlikely. If a creature didn’t feel pain, it’s unlikely it would respond to analgesics, seek out analgesic drugs, and get distracted by bodily damage.

  This is where I would disagree. I expect moderately-complicated creatures would develop traits like these under evolutionary pressures (except seeking out analgesic drugs). The question then is how likely is it that the best /​ only /​ easiest-to-evolve way to produce this slate of behaviors involves having a conscious experience with the relevant pain profile.

  We know that human brains have undergone massive changes since our most recent common ancestor with fish, that terrestrial environments place very different demands on our bodies, that human beings have an unparalleled behavioral flexibility to address injuries, etc. so it is plausible that we do have fairly different nociceptive faculties. It seems to me like a pretty open question precisely how neurologically or algorithmically similar our faculties are and how similar they would need to be to for fish to qualify as having pain. The fact that we can’t even tell how important the cortex is for pain in humans seems like strong evidence that we shouldn’t be too confident about attributing pain to fish. We just know so little. Of course, we shouldn’t be confident about denying it to them either, but much confidence either way seems unjustifiable.

• Derek Shiller 1 Jul 2025 20:43 UTC

  4 points

  0 ∶ 0

  in reply to: David Mathers🔸’s comment on: De­bate: De­pop­u­la­tion Matters

  I would think the trend would also need to be evenly distributed. If some groups have higher-than-replacement birth rates, they will simply come to dominate over time.