I agree that things could be such that the reflective equilibrium was that, but it would really surprise me. In particular, Open Philanthropy has a sole funder. It’s one guy. It would surprise me if his preference about things was such that he wanted to leave Pareto improvements on the table. Not impossible, but it feels like a stretch. Of course I’m missing a whole lot of context, so ¯\_(ツ)_/¯.
Or, in other words, if things were such as in your post, I would expect to see trade (even though you could set up things such that it doesn’t happen). I would not expect to see a U-turn on criminal justice. Instead, I think the worldview diversification thing as implemented by Open Phil is more like a historical accident.
The Pareto improvements aren’t about worldview diversification, though. You can see this because you have exactly the same problem under a single worldview, if you keep the amount of funding constant per year. You can solve this by letting each worldview donate to, or steal from, its own budget in other years.
I do think trade between worldviews is good in addition to that, to avoid the costs of lending/borrowing; the issue is that you need to be very careful when you’re relying on the worldviews themselves to tell you how much weight to put on them. So for example, if every year the AI risk worldview gets more and more alarmed, then it might “borrow” more and more money from the factory farming worldview, with the promise to pay back whenever it starts getting less alarmed. But the whole point of doing the bucketing in the first place is so that the factory farming worldview can protect itself from the possibility of the AI risk worldview being totally wrong/unhinged, and so you can’t assume that the AI risk worldview is just as likely to update down as to update upwards.
You could avoid this by having the AI risk worldview make concrete predictions about what would make it more or less alarmed. But now this is far from a “simple” scheme, it requires doing a bunch of novel intellectual work to pin down the relevant predictions.
Re the U-turn on criminal justice: no matter what scheme you use, you should expect it to change whenever you have novel thoughts about the topic. Worldview diversification isn’t a substitute for thinking!
So for example, if every year the AI risk worldview gets more and more alarmed, then it might “borrow” more and more money from the factory farming worldview, with the promise to pay back whenever it starts getting less alarmed. But the whole point of doing the bucketing in the first place is so that the factory farming worldview can protect itself from the possibility of the AI risk worldview being totally wrong/unhinged, and so you can’t assume that the AI risk worldview is just as likely to update down as to update upwards.
Suppose that as the AI risk worldview becomes more alarmed, you are paying more and more units of x-risk prevention (according to the AI risk worldview) for every additional farmed animal QALY (as estimated by the farmed animal worldview). I find that very unappealing.
I agree that this, and your other comment below, both describe unappealing features of the current setup. I’m just pointing out that in fact there are unappealing outcomes all over the place, and that just because the equilibrium we’ve landed on has some unappealing properties doesn’t mean that it’s the wrong equilibrium. Specifically, the more you move towards pure maximization, the more you run into these problems; and as Holden points out, I don’t think you can get out of them just by saying “let’s maximize correctly”.
(You might say: why not a middle ground between “fixed buckets” and “pure utility maximization”? But note that having a few buckets chosen based on standard cause prioritization reasoning is already a middle ground between pure utility maximization and the mainstream approach to charity, which does way less cause prioritization.)
I think the post from Holden that you point to isn’t really enough to go from “we think really hardcore estimation is perilous” to “we should do worldview diversification”. Worldview diversification is fairly specific, and there are other ways that you could rein-in optimization even if you don’t have worldviews, e.g., adhering to deontological constraints, reducing “intenseness”, giving good salaries to employees, and so on.
You can see this because you have exactly the same problem under a single worldview, if you keep the amount of funding constant per year. You can solve this by letting each worldview donate to, or steal from, its own budget in other years.
Good point! I think my objection then is that I do want relative values to not be “dangling”. Also, I do care about expected values, so there are some interesting trades where you bet all of your budget which you couldn’t do just by borrowing from your future stream.
One reason to see “dangling” relative values as principled: utility functions are equivalent (i.e. produce the same preferences over actions) up to a positive affine transformation. Hence why we often use voting systems to make decisions in cases where people’s preferences clash, rather than trying to extract a metric of utility which can be compared across people.
I’m not sure whether the following example does anything for you; it could be that our intuitions about what is “elegant” are just very different:
Imagine that I kill all animals except one Koala. Then if your worldview diversification which had some weight on animals, you would spend the remaining of that worldview on the Koala. But you could buy way more human QALYs or units of x-risk work per unit of animal welfare.
More generally, setting up things such that sometimes you would end up valuing e.g., a salmon as 0.01% of a human and other times as 10% of a human just seems pretty inelegant.
This is one reason why you should use normatively defined worldviews to define your buckets, not cause areas. Animal welfare interventions currently dominate on some worldviews that include nonhuman animals and assign them substantial weight. Those worldviews aren’t (almost ever) going to assign an average nonhuman animal many times more average weight than the average human. The animal-inclusive worldviews also care terminally at least about as much about any human as the panda, and assuming they are impartial and maximizing in some way, the panda won’t get any help from us (unless it benefits humans substantially; humans might care more about the last panda than the average human).
Each worldview (and buckets may reflect aggregates of such worldviews) has its own claim on the relative average value of humans and salmons. Those claims can change with new information, but in the usual ways, not because there are fewer salmons or because fewer salmon are helped per $ now than before.
There can be worldviews where the marginal cost-effectiveness of GiveWell interventions are close to the marginal cost-effectiveness of the best nonhuman animal-focused interventions, and small changes to the situation can shift resources from humans towards nonhumans or vice versa. I think most worldviews won’t be like that, though, and will instead have large gaps in marginal cost-effectiveness (going either way).
I agree that things could be such that the reflective equilibrium was that, but it would really surprise me. In particular, Open Philanthropy has a sole funder. It’s one guy. It would surprise me if his preference about things was such that he wanted to leave Pareto improvements on the table. Not impossible, but it feels like a stretch. Of course I’m missing a whole lot of context, so ¯\_(ツ)_/¯.
Or, in other words, if things were such as in your post, I would expect to see trade (even though you could set up things such that it doesn’t happen). I would not expect to see a U-turn on criminal justice. Instead, I think the worldview diversification thing as implemented by Open Phil is more like a historical accident.
The Pareto improvements aren’t about worldview diversification, though. You can see this because you have exactly the same problem under a single worldview, if you keep the amount of funding constant per year. You can solve this by letting each worldview donate to, or steal from, its own budget in other years.
I do think trade between worldviews is good in addition to that, to avoid the costs of lending/borrowing; the issue is that you need to be very careful when you’re relying on the worldviews themselves to tell you how much weight to put on them. So for example, if every year the AI risk worldview gets more and more alarmed, then it might “borrow” more and more money from the factory farming worldview, with the promise to pay back whenever it starts getting less alarmed. But the whole point of doing the bucketing in the first place is so that the factory farming worldview can protect itself from the possibility of the AI risk worldview being totally wrong/unhinged, and so you can’t assume that the AI risk worldview is just as likely to update down as to update upwards.
You could avoid this by having the AI risk worldview make concrete predictions about what would make it more or less alarmed. But now this is far from a “simple” scheme, it requires doing a bunch of novel intellectual work to pin down the relevant predictions.
Re the U-turn on criminal justice: no matter what scheme you use, you should expect it to change whenever you have novel thoughts about the topic. Worldview diversification isn’t a substitute for thinking!
Suppose that as the AI risk worldview becomes more alarmed, you are paying more and more units of x-risk prevention (according to the AI risk worldview) for every additional farmed animal QALY (as estimated by the farmed animal worldview). I find that very unappealing.
I agree that this, and your other comment below, both describe unappealing features of the current setup. I’m just pointing out that in fact there are unappealing outcomes all over the place, and that just because the equilibrium we’ve landed on has some unappealing properties doesn’t mean that it’s the wrong equilibrium. Specifically, the more you move towards pure maximization, the more you run into these problems; and as Holden points out, I don’t think you can get out of them just by saying “let’s maximize correctly”.
(You might say: why not a middle ground between “fixed buckets” and “pure utility maximization”? But note that having a few buckets chosen based on standard cause prioritization reasoning is already a middle ground between pure utility maximization and the mainstream approach to charity, which does way less cause prioritization.)
I think the post from Holden that you point to isn’t really enough to go from “we think really hardcore estimation is perilous” to “we should do worldview diversification”. Worldview diversification is fairly specific, and there are other ways that you could rein-in optimization even if you don’t have worldviews, e.g., adhering to deontological constraints, reducing “intenseness”, giving good salaries to employees, and so on.
Good point! I think my objection then is that I do want relative values to not be “dangling”. Also, I do care about expected values, so there are some interesting trades where you bet all of your budget which you couldn’t do just by borrowing from your future stream.
One reason to see “dangling” relative values as principled: utility functions are equivalent (i.e. produce the same preferences over actions) up to a positive affine transformation. Hence why we often use voting systems to make decisions in cases where people’s preferences clash, rather than trying to extract a metric of utility which can be compared across people.
I’m not sure whether the following example does anything for you; it could be that our intuitions about what is “elegant” are just very different:
Imagine that I kill all animals except one Koala. Then if your worldview diversification which had some weight on animals, you would spend the remaining of that worldview on the Koala. But you could buy way more human QALYs or units of x-risk work per unit of animal welfare.
More generally, setting up things such that sometimes you would end up valuing e.g., a salmon as 0.01% of a human and other times as 10% of a human just seems pretty inelegant.
This is one reason why you should use normatively defined worldviews to define your buckets, not cause areas. Animal welfare interventions currently dominate on some worldviews that include nonhuman animals and assign them substantial weight. Those worldviews aren’t (almost ever) going to assign an average nonhuman animal many times more average weight than the average human. The animal-inclusive worldviews also care terminally at least about as much about any human as the panda, and assuming they are impartial and maximizing in some way, the panda won’t get any help from us (unless it benefits humans substantially; humans might care more about the last panda than the average human).
Each worldview (and buckets may reflect aggregates of such worldviews) has its own claim on the relative average value of humans and salmons. Those claims can change with new information, but in the usual ways, not because there are fewer salmons or because fewer salmon are helped per $ now than before.
There can be worldviews where the marginal cost-effectiveness of GiveWell interventions are close to the marginal cost-effectiveness of the best nonhuman animal-focused interventions, and small changes to the situation can shift resources from humans towards nonhumans or vice versa. I think most worldviews won’t be like that, though, and will instead have large gaps in marginal cost-effectiveness (going either way).
Open Phil has worldview buckets (https://www.openphilanthropy.org/research/update-on-cause-prioritization-at-open-philanthropy/), including an animal-inclusive bucket, but no animal-only bucket. This might look like cause area buckets instead, but what they’re trying to approximate are in fact worldview buckets.
Good point!