I am a senior researcher at Rethink Priorities’ Worldview Investigations Team. Previously I did work for Oxford’s Global Priorities Institute and was a research analyst at the Forethought Foundation for Global Priorities Research. I took that role after completing the MPhil in Economics at Oxford University. Before that, I studied Mathematics and Philosophy at the University of St Andrews.
Find out more about me here.
Hi Oscar, thanks. Yes! Indicator evidence is inserted at the bottom and flows upward. The diagram priors flows down to set expectations, so it’s a common convention to draw it so. Informally you see it as each node splitting different nodes into subnodes, but the arrows don’t mean to imply that information doesn’t travel up.
On to your main point: there’s no question that this is a preparadigmatic field where progress and consensus are difficult to find, and rightly so given the state of the evidence.
A few thoughts on why pursue this research now despite the uncertainty:
First, we want a framework in place that can help transition towards a more paradigmatic science of digital minds as the field progresses. Even if you’re sceptical about current reliability, we think that having a model that can modularly incorporate new evidence and judgements could serve as valuable infrastructure for organising future findings.
Second, whilst which theory is true remains uncertain, we found that operationalising specific stances was often less fuzzy than expected. Many theories make fairly specific predictions about conscious versus non-conscious systems, giving us firmer ground for stance-by-stance analysis. (Though you might still disagree with any given stance, of course!)
Your concern about theory quality might itself be worth modelling as a stance—we thought of something like “Stance X” but it could be “Theoretical Scepticism”—where all features provide very weak support. That would yield small updates from the prior regardless of the system. Already, you can see the uncertainty playing out in the model: notice how wide the bands are for chickens vs humans (Figure 3). Several theories disagree substantially about animal consciousness, which arguably reflects concerns about theory quality.
That said, we’re deliberately cautious about making pronouncements on consciousness probability or evidence strength. We see this as a promising way to start characterising these values rather than offering definitive answers.
(Oh and regarding priors: setting them is hard, and robustness important. it might be helpful to see Appendix E and Figures 9-11. The key finding is that while absolute posteriors are highly prior-dependent (as you’d expect), the comparative results and direction of updating are pretty robust across priors.)
Hi Andrew, I definitely agree. Tracking how these systems evolve over time — both what the models reveal about their changes and how we can formulate projections based on those insights — has been one of our main motivations for this work. We’re in fact hoping to gather data on newer systems and develop some more informed forecasts based on what we find.
Thank you for your comment Kuhanj. I share your belief that the EA movement would benefit from the type of suggestions you outlined on your quick take. I particularly valued seeing more discussions on heuristics, for they are often as limited as they are useful!
Regarding your ‘Being slow to re-orient’ suggestion, an important nuance comes to mind: movements can equally falter by pivoting too rapidly. When a community glimpses promise in a new X direction, there’s a risk of hastily redirecting significant resources, infrastructure, and attention toward it prematurely. The wisdom accumulated through longer reflection and careful evidence collection often contains (at least some) genuine insight, and we should be cautious about abandoning established priorities to chase every emerging “crucial consideration” that surfaces.
As ever, the challenge lies in finding that delicate balance between responsiveness and steadfastness — being neither calcified in thinking nor swept away by every new intellectual current.
Thanks for your comment Richard, I think the discussion is better for it. I agree with your clarification that there are key differences that distinguish EA from more traditional attitudes and that defending cause incommensurability and personal taste are two relevant dimensions.
Like you, it does seem to us that in the early days of EA, many people doing prioritisation of GHD interventions went beyond traditional intervention clusters (e.g. education) and did some cross-cause prioritisation (identifying the best interventions simpliciter).
That said, the times feel different now and we think that, increasingly, people are doing within-cause prioritisation by only trying to identify the best interventions within a given area without it being clearly ‘done in service of the ultimate goal of “cross-cause prioritization”’ (e.g. because they are working for an institution or project with funds dedicated exclusively to be allocated within a certain cause).
Thank you Jack, yes it’s absolutely about acting on it too, research is just step 1.
If you’ve found the ‘snapshot of EA’ section particularly valuable, please flag it under this comment so we can gauge how much we should invest in updating it or expanding it in the future. To clarify:
- Vote agree for “particularly valuable”.
- Vote disagree for “not that valuable”.
Feel free to add any comments.
I agree with you Oscar, and we’ve highlighted this in the summary table where I borrowed your ‘contrasting project preferences’ terminology. Still, I think it could still be worth drawing the conceptual distinctions because it might help identify places where bargains can occur.
I liked your example too! We tried to add a few (GCR-focused agent believes AI advances are imminent, while a GHD agent is skeptical; AI safety view borrows resources from a Global Health to fund urgent AI research; meat-eater; gun rights and another supporting gun control both fund a neutral charity like Oxfam...) but we could have done better in highlighting them. I’ve also added these to the table.
I found your last mathematical note a bit confusing because I originally read A,B,C as projects they might each support. But if it’s outcomes (i.e. pairs of projects they would each support), then I think I’m with you!
Just to flag that Derek posted on this very recently. It’s directly connected to both the present post and Michael’s.
That’s fair. The main thought that came to mind, which might not be useful, is developing the patience (eagerness to get to conclusions is often incompatible with the work required) and choosing your battles early. As you say, it can be hard and time-consuming. So people in the community asking narrower questions and focusing on one or two is probably the way to go.
Thanks for looking through our work and for your comment, Deborah. We recognise that different parts of our models are often interrelated in practice. In particular, we’re concerned about the problem of correlations between interventions too, as we flag here. This is an important area for further work. That being said, it isn’t clear that the cases you have in mind are problems for our tools. If you think, for instance, that environmental interventions are particularly good because they have additional (quantifiable or non-quantifiable) benefits, you can update the tool inputs (including the cause or project name) to reflect that and increase the estimated impact of that particular cause area. We certainly don’t mean to imply that climate change is an unimportant issue.
I think another common pitfall is not working through things from first principles. I appreciate that it’s challenging and that any model is unrealistic. Still, BOTECs, pre-established boundaries between cause-areas/worldviews and our first instincts more broadly are likely to (and often do) lead us astray. Separately, I’m glad EA is so self-aware and worried about healthier epistemics, but I think we could do more to guard against echo-chamber thinking.
I was personally struck by how sensitive portfolios are to even modest levels of risk aversion. I don’t know what “correct” level of risk aversion is, or what the optimal decision procedure is in practice (even though most of my theoretical sympathies lie with expected value maximisation). Even so, seeing how introducing bits of risk aversion, even when using parameters relatively generous towards x-risk, still points towards spending most resources on animals (and sometimes global health) has led me to believe that type of work is robustly better than I used to think. There are many uncertainties and I don’t think EA should be reduced to any one of its cause-areas but, especially given this update, I would be sad to see the animal space shrink in relative size any more than it has.
Thanks for the question Carter! Would you mind saying a bit more about the kind of empirical work you have in mind? Are you thinking about empirical research into the inputs to the tools? Or are you thinking about using the tools to conduct research on people’s views about cause prioritization? Do you have any concrete empirical projects you’d like to see WIT do?
Thanks for your question, Chris. We hear you about the importance of making the content accessible. We’ve aimed to include the main takeaways in intro and conclusion posts that can be easily skimmed. We also provide an executive summary at the beginning of each post. We hope that these help, but we take the point that it may not be obvious that we’ve taken these steps, and we’ll revisit this suggestion in future sequences to make sure the purposes of those posts and introductory materials are clear. It may also be useful for us to consider more visual summaries of some of our results, as we provided for our discussion of human extinction. Do you have any concrete suggestions given the approach we’ve adopted so far?
Thank you for your kind words Ben. A substantial amount of in-house software work went into both tools. We used react and vite to create these, and python for the server running the maths behind the scenes. If the interest in this type of work and the value-added are high enough, we’d likely want to do more of it.
On your last point, we haven’t done open-source development from scratch for the projects we’ve done so far, but it might be a good strategy for future ones. That said, for transparency, we’ve made all our code accessible here.
Thank you for your comment Lukas, we agree that this tool, and more generally this approach, could be useful even in that case, when all considerations are known. The ideas we built on and the language we used came from the literature on moral parliaments as an approach to better understand and tackle moral uncertainty, hence us borrowing from that framing.
Thank you for adding various threads to the conversation Arepo! I don’t disagree with what I take to be your main point: benign AI and interstellar travel are likely to have a big impact. I will say though, while their success might significantly reduce risk, and for a long time, any given intervention is unlikely to make major progress towards them. Hence, at the intervention level, I’m tempted to remain sceptical about the abundance of interventions that dramatically reduce risk for a long time.
Thank you for deeply engaging with our work and for laying out your thoughts on what you think are the most promising paths forward, like searching for contingent and persistent interventions, applying a medium-term lens to global health and animal welfare, or investigating fanaticism. I thought your post was well-written, to the point and enjoyable.
Hi Siebe, yes, all the scenarios of this report assume positive value at all times. I don’t think it’s certain that this will happen which is why the concluding remarks mention “investigating value trajectories that feature negative value” as a possible extension. So, yes, I completely agree this is something to look into in more depth.
Thanks vasco. And thanks for helping us think through what we can do better. Some thoughts on this:
We considered several framings, scales and options to give experts. Since they were evaluating a lot of stances and we wanted experts to really know what we meant, we prioritised giving them context and then asking them the simplified general question of plausibility, with an intuitive scale. The exact question was: ‘how plausible do you find X stance?’, just after having fully describing X. We also asked them for general notes and comments and they didn’t seem to find that part of the survey particularly confusing (perhaps to your and my surprise). More broadly, I agree with you that sometimes perfectly defining terms and scales can help some people think through it but not everyone, and the science on how much it helps points is mixed.
We didn’t find that people were responding with zero plausibility very much at all. As you can see from the results, almost all respondents found most, if not all, stances at least a little bit plausible. I agree that had we found a lot of concentration around the very high or very low plausibility, having some sort of logarithmic scale could help distinguish results.
I’m not sure what you have in mind in terms of modelling the stances’ weight as distributions instead of point estimates. Perhaps you mean something like leveraging those distributions above via some sort of Monte Carlo where weights are drawn from these distributions and the process is repeated many times, then aggregated. That indeed sounds more sophisticated and could possibly help track uncertainty but I suspect it would very little difference. In particular, I think so because we observed that unweighted pooling of results across all stances is surprisingly similar to the pool when weighted by experts; the same if you squint.