Thanks for working on this! As a grantmaker I’ve found it a bit puzzling to evaluate fellowship programs and I hope you continue the project.
tylermjohn
Karma: 1,033
Yup. The constitution of the democratic community is inherently value laden. Even prioritizing conscious beings or beings with preferences is a value judgment. I don’t think there’s any option here but to debate and argue over who gets a seat at the table in a realpolitik kind of way, and then use deep democracy to extend standing to other beings. If, for example, only adult humans vote, you and I can still use our votes to extend political standing to animals, and if there is no tyranny of the majority and democracy does justice to the cardinality in people’s preferences, then people who care a lot about animals can give them a correspondingly large amount of status.
Makes sense! There’s some old writers in the utilitarian tradition like James Griffin that define utilitarianism in the broader way, but I do think your articulation is probably more common.
Huh, I mean it just is formally equivalent to the sum of log utilities in the bargaining situation! But “utilitarianism” is fuzzy :)
Yes, the idea of finding a preference aggregation mechanism that does much better than modern electoral systems at capturing the cardinality of societal preferences is, I think, really core to what I’m doing here, so I probably should have brought this out a bit more than I did!
I’m only saying it’s in tension with the diagnosis as “emphasis on individual action, behavior & achievement over collective.”
I agree with all of your concrete discussion and think it’s important.
In the traditional Nash bargaining setup you evaluate people’s utilities in options relative to the default scenario, and only consider options that make everyone at least as well off. This makes it individually rational for everyone to participate because they will be made better off by the bargain. That’s different from, say, range voting.
Thanks very much for the added last four paragraphs! We’re in strong agreement re: trade being a great way to approximate granular, deep preference aggregation, particularly if you have a background of economic equality.
I’m excited to read the linked section of No Easy Eutopia. I agree that there’s no fully neutral way to aggregate people’s preferences and preserve cardinality. But I do think there are ways that are much more neutral, and that command much broader consent, and that they can be a big improvement over alternative mechanisms.
No problem on the chaotically written thoughts, to be fair to you my post was (due to its length) very unspecific. And that meant we could hammer out more of the details in the comments, which seems appropriate.
And then I guess both of as are in some kind of agreement that this kind of stuff (deliberate structured initatives to inject some democracy into the models) ends up majorly determining outcomes from AGI.
Yeah I think this is plausible and a good point of agreement, plus a promising leverage point. But I do kind of expect normal capitalist incentives will dominate anything like this, and that governments won’t intervene except for issues of safety, as you seem to.I find this somewhat confusing
Nash is formally equivalent to the sum of log utilities (preferences) in the disagreement set, so it’s a prioritarian transformation of preference utilitarianism over a particular bargain.
I agree that it can come drastically far apart from totalist utilitarianism. What I actually like about it is that it’s a principled way to give everyone’s values equal weight that preserves the cardinality in people’s value functions and is arbitrarily sensitive to changes in individuals’ values, and that it doesn’t require interpersonally comparable utilities, making it very workable. I also like that it maximizes a certain weighted sum of efficiency and equality. As an antirealist who thinks I have basically unique values, I like that it guarantees that my values have some sway over the future.
One thing I don’t like about Nash is that it’s a logistic form of prioritarianism, and over preferences rather than people. That means that my strongest preferences don’t get that much more weight over my weakest preferences. Perhaps for that reason simple quadratic voting does better. It’s in some ways less elegantly grounded, but it’s also more well-understood by the broader world.
I’m seeing the position as a principled way to have a fair compromise across different people’s moral viewpoints, which also happens to do pretty well by the lights of my own values. It’s not attempting to approximate classical utilitarianism directly, but instead to give me some control over the future in the areas that matter the most to me, and thereby allow me to enact classical utilitarianism. There might be better such approaches, but so far this is the one that seems most promising to me at the moment.
Nice. I encountered a similar crux the other week in a career advice chat when someone said “successful people find the skills with which they really excel and exploit that repeatedly to get compounding returns” to which I responded with “well, people aren’t the only things that can have compounding returns, organizations can also have compounding returns, so maybe I should keep helping organizations succeed to capture their compounding returns.”
On the flip side, the fact that EA has focused so much on community building and talent seems like a certain kind of communitarianism, putting the success of the whole above any individual.
Nice! I’ll have to read this.
I agree defaults are a problem, especially with large choice problems involving many people. I honestly haven’t given this much thought, and assume we’ll just have to sacrifice someone or some desideratum to get tractability, and that will kind of suck but such is life.
I’m more wedded to Nash’s preference prioritarianism than the specific set-up, but I do see that once you get rid of Pareto efficiency relative to the disagreement point it’s not going to be individually rational for everyone to participate. Which is sad.
One further thing that might help you get in my brain a bit is that I really am thinking this as more like “what values should we be aiming at to guide the future” and being fairly agnostic on mechanism rather than something like “let’s put democracy in The AGI’s model spec”. And I really am envisioning the argument as something like: “Wow, it really seems like the future is not a utilitarian one. Maybe sensitivity to minority values like utilitarianism is the best thing we can ask for.” — rather than something like “democracy good!” And that could mean a lot of different things. On Mondays, Wednesdays, and Fridays I think it means avoiding too much centralization and aiming for highly redistributive free market based futures as an approximation of Deep Democracy.
Always appreciate your “normal technology” foil! It’s shaped my views a lot. One simple thought that you foreshadow is: we’ll have some laws about what AIs can do (can they harm animals?) and we’ll have new ways to decide on these laws enabled by AI technology. This could be a kind of discontinuity where a lot of change could happened, and where we could do things in a different and better way.
I foreshadow a few other options, like nationalization and democratization of AI, which I expect will happen *to some degree*, but likely not that much. If you have something more singleton shaped, where one lab is leading everyone else, maybe because Google or Nvidia has so much more compute, this kind of outcome looks much more likely.
Of course, all the labs say things about democratic inputs into AI model specs and are trying stuff to this end, and these could be deeper or shallow. But I share your skepticism that this is actually going to happen in any meaningful way...
You might find my response to Michael’s comment helpful re: from whose perspective this makes sense. The classic idea of bargaining theory and formal defenses of democracy is that it’s in most everyone’s rational self interest because it expands the Pareto frontier, eliminating arbitrary dictatorship, avoiding wasteful races, and getting people positive sum options better than the status quo. Or see CB’s remarks: it’s really unlikely that a utilitarian thing could happen, but this might actually work! And it gets a lot of what we want, and much more than things that aren’t sensitive to niche values like utilitarianism.
Re: what deep democracy, I like Nash bargaining! Kalai-Smorodinsky also seems fine. Quadratic voting with AI agents finding compromise solutions seems like a good approximation.
(One side-bar is that you should ideally do a global Nash bargain, over all of your values rather than bargaining over particular issues like the suffering of wildlife. This is so that people can rank all of their values and get the stuff that is most important to them. If you care a lot about wild animal suffering and nothing about hedonium and I care a lot about hedonium and nothing about WAS, a good trade is that we have no wild animal suffering and lots of hedonium. This is very hard to do but theoretically optimal.)
I have a slide deck on this solution if you’d like to see it!
Thank you, Michael!
How does this work mechanically? Say 1% of people care about wild animal suffering, 49% care about spreading nature, and 50% don’t care about either. How do you satisfy both the 1% and the 49%? How do the 1%—who have the actually correct values—not get trampled?
The view I like is something like Nash bargaining. You elicit values from people on a cardinal scale, give everyone’s values equal weight, and find compromise solution that maximizes group values. On Nash’s solution this means something like: everyone rates outcomes on a Likert scale (1-10) and then you pick whatever solution gives you the highest product of everyone’s numbers. (There are other approaches than taking the product, and I’m fudging a few details, but the underlying principle is the same: people rank outcomes, then you find the compromise solution that maximizes a weighted or adjusted sum of their values.) You can imagine just doing preference utilitarianism to see what this looks like in practice.
If you have a literal conflict between values (some people want to minimize animal suffering a 10 and some people want to maximize animal suffering a 10) then they will cancel out and there’s no positive sum trade that you can make. Still, the system will be sensitive to the 1%’s values. So maybe we’ll spread nature 48% hard instead of 49% hard because of the 1% canceling out. (Not literally this, but directionally this.)
But usually people don’t have literally opposed values and they can find positive sum compromises that have more value overall. Like, say, spreading nature but without sentience, or spreading nature but with welfare interventions, etc.
You could also picture it as an idealized marketplace: the 1% who care about wild animal suffering pay the 49% who care about spreading nature a price that they see as fair to reduce wild animal suffering in the nature that they spread.
Lots of different methods to consider here, but I hope the underlying idea is now less opaque.
If I align AGI to my own values, then the AGI will be nice to everyone—probably nicer than if it’s aligned to some non-extrapolated aggregate of the values of all currently-living humans.
It also seems a bit circular because if you want to build a Deep Democracy AGI, then that means you value Deep Democracy, so you’re still aligning AGI to your values, it’s just that you value including everyone else’s values.
I agree that Deep Democracy is not value neutral. It presupposes some values you might not like, and will get you outcomes worse than what you value. The hope is to find a positive sum compromise that makes sense from the individual rationality of lots of people, so instead of fighting over what to maximize you maximize something that captures what a lot of people care about, expanding the Pareto frontier by finding better solutions than wasteful conflict or risking a chance of a dictator that lots of people oppose.
Or, put another way, the idea is that it’s “better than the default outcome” not just for you but for a large majority of people, and so it’s in our interests to band together and push for this over the default outcome.
If you have a lot of power, then gambling on a chance at dictatorship (say, proportional to your power) could be worth it and incentivized, and I think it’s important to be realistic about that to understand how the world could unfold. But there are a lot of other downsides to it, like wasteful races and the chance of being cut out of the pie as punishment for your societal defection, which do favour more democratic approaches.
Yes! That is very close to the kind of idea that drove me from utilitarian supervillain towards deep democracy enjoyer. I do think it’s worth reading the whole post (it’s short), but in brief:
On a bunch of moral worldviews there are some futures that are astronomically more valuable than others, and they are not valued to nearly that extent in the world today, leading to the possibility of losing out on almost all value
For example, maybe you’re a hedonium fan and you think hedonium is 10^20 times more valuable than whatever matter is turned into by default; if you can’t get any hedonium, then the future you expect is like 10^20 times worse than it could be… ~all value has been lost
One way to hedge against this possibility is essentially a kind of worldview diversification, where you give power to a bunch of different moral views that then each maximize their own goals
Then if you’re someone with a minority viewpoint you at least get some of the world to maximize according to your values, so maybe you capture 1/1000th of value instead of 1 over 10^20
This only works if you have a democratic procedure that is genuinely sensitive to minority views and not democratic procedures that, say, maximize whatever 51% of people want, which leads to zero hedonium
So if you have extremely scope sensitive, fragile values that society doesn’t have (which… probably all of us do?) then you do much better with Deep Democracy than with Normal Democracy and, arguably, than you do with a coup from an arbitrary dictator.
Morality is Objective
It’s epistemically inaccessible, explanatorily redundant, unnecessary for any pragmatic aim, just a relic of the way our language and cooperative schemes work. I’m not sure the idea can even really be made clear. Empirically, convergence through cooperative, argumentative means looks incredibly unlikely in any normal future. I voted for the strongest position because relative to my peers I have the most relativistic view I know and because of my high (>.99) credence in antirealism. But obviously morality is sort of kind of objective in certain contexts and among certain groups of interlocutors, given socially ingrained shared values.
The next existential catastrophe is likelier than not to wipe off all animal sentience from the planet
Edit: OK almost done being nerdsniped by this, I think it basically comes down to:
What’s the probability that the existential catastrophe comes from a powerful optimizer or something that turns into a powerful optimizer, which is arbitrarily close to paperclipping?
Maybe something survives a paperclipper. It wants to turn all energy into data centers but it’s at least conceivable that something survives this. The optimizer might, say, dissassemble Mercury and Venus to turn it into a Matryoshka brain but not need further such materials from Earth. Earth still might get some emanent heat from the sun despite all of the solar panels nested around it, and be the right temperature to turn the whole thing into data centers. But not all materials can be turned into data centers, so maybe some of the ocean is left in place. Maybe the Earth’s atmosphere is intentionally cooled for faster data centers, but there’s still geothermal heat for some bizarre animals.
But probably not. As @Davidmanheim points out (who changed my mind on this), you’ll probably still want to disassemble the Earth to mine out all of the key resources for computing, whether for the Matryoshka brain or the Jupiter brain, and the most efficient way to do this probably isn’t cautious precision mining.
Absent a powerful optimizer you’d expect some animals to survive. There’s a lot of fish, some of them very deep in the ocean, and ocean life seems pretty wildly adaptive, particularly down at the bottom where they do crazy stuff like feeding off volcanic heat vents to turn their bodies into iron and withstand pressures that crumble submarines.
So by far the biggest parameter is going to be how much you expect the world to end from a powerful optimizer. This is the biggest threat in the near term, though if we don’t build ASI or build it safely other existential threats loom larger.
@Toby Tremlett🔹 would a better focus be “tractable interventions on democratic backsliding” to focus on concretely useful things and get around many of the content policy issues?