Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
I think this post misses one of the concerns I have in the back of the mind about AI: How much is current pluralism, liberalism and democracy based on the fact that governance can’t be automized yet?
Currently, policymakers need the backing of thousands of bureaucrats to execute policy, this same bureaucracy provides most of the information to the policymaker. I am fairly sure that this makes the policymaker more accountable and ensures that some truly horrible ideas do not get implemented. If we create AI specifically to help with governance and automate a large amount of this kind of labor, we will find out how important this dynamic is…
I think this dynamic was better explained in this post.
I think this is an interesting idea—that “a large bureaucracy pressures the leaders to be more aligned”—but it really doesn’t sit well with me.
In my experience, small bureaucracies often behave better than large ones. Startups infamously get worse and less efficient as they scale. I haven’t seen many analyses of, “the extra managers that come in help make sure that the newer CEO is more aligned.”
I generally expect similar with national governments.
Again though, it is key that “better governance” is actually better. It’s clearly possible to use AI for all kinds of things, many of which are negative. My main argument above is that some of those possibilities include “making the top levels of governance better and more aligned”, and that smart actors can pressure these situations to happen.
I don’t disagree with your final paragraph, and I think this is worth pursuing generally.
However, I do think we must consider the long-term implications of replacing long-established structures with AI. These structures have evolved over decades or centuries, and their dismantling carries significant risks.
Regarding startups: to me, it seems like their decline in efficiency as they scale is a form of regression to the mean. Startups that succeed do so because of their high-quality decision-making and leadership. As they grow, the decision-making pool expands, often including individuals who haven’t undergone the same rigorous selection process. This dilution can reduce overall alignment of decisions with those the founders would have made (a group already selected for decent decision-making quality, at least based on the limited metrics which cause startup survival).
Governments, unlike startups, do not emerge from such a competitive environment. They inherit established organizations with built-in checks and balances designed to enhance decision-making. These checks and balances, although contributing to larger bureaucracies, are probably useful for maintaining accountability and preventing poor decisions, even though they also prevent more drastic change when this is necessary. They also force the decision-maker to take into account another large group of stakeholders within the bureaucracy.
I guess part of my point is that there is a big difference between alignment with the decision-maker and the quality of decision-making.
I don’t see my recommendations as advocating for a “dismantling”—it’s more like an “augmenting.”
I’m not at all recommend we move to replace our top executives with AIs anytime soon. I’m not sure if/when that might ever be necessary.
Rather, I think we could use AIs to help assist and oversee the most sensitive parts of what already exists. Like, top executives and politicians can use AI systems to give them advice, and can separately work with auditors who would use advanced AI tools to help them with oversight.
In my preferred world, I could even see it being useful to have more people in government, not less. Productivity could improve a lot in this sector, but also, this sector still seems like a very important one relative to others, and perhaps expectations could rise a lot too.
> I guess part of my point is that there is a big difference between alignment with the decision-maker and the quality of decision-making.
I agree these are both quite separate. I think AI systems could help with both though, and would prefer that they do.
So I’ve been working in a very adjacent space to these ideas for the last 6 months and I think that the biggest problems that I have with this is just the feasibility of it.
That being said we have thought about some ways of approaching a GTM for a very similar system. Thr system I’m talking about here is an algorithm to improve interpretability and epistemics of organizations using AI.
One is to sell it as a way to “align” management teams lower down in the organization for the C-suite level since this actually incentivises people to buy it.
A second one is to start doing the system fully on AI to prove that it increases interpretability of AI agents.
A third way is to prove it for non-profits by creating an open source solution and directing it to them.
At my startup we’re doing number two and at a non-profit I’m helping we’re doing number three. After doing some product market fit people weren’t really that excited about number 1 and so we had a hard time getting traction which meant a hard time building something.
Yeah, that’s about it really, just reporting some of the experience in working on a very similar problem
That sounds really neat, thanks for sharing!
Hi, Jonas! Do you have a link to more info about your project?
Startup: https://thecollectiveintelligence.company/
Democracy non-profit: https://digitaldemocracy.world/
I think these comments (thanks for them, all), have made me think it would be interesting to have more surveys on opinions here.
I’d be curious to see the matrix of the questions:
1. How likely do you think AI is to be an existential threat?
2. How valuable do you think AIs will be, pre-transformative-AI?
My guess is that there’s a lot of diversity here, even among EAs.
I think some form of AI-assited governance have great potential.
However, it seems like several of these ideas are (in theory) possible in some format today—yet in practice don’t get adopted. E.g.
I think it’s very hard to get even the most basic forms of good epistemic practices (e.g. putting probabilities on helpful, easy-to-forecast statements) embedded at the top levels of organizations (for standard moral maze-type reasons).
As such I think the role of AI here is pretty limited—the main bottleneck to adoption is political / bureacratic, rather than technological.
I’d guess the way to make progress here is in aligning [implementation of AI-assisted governance] with [incentives of influential people in the organization] - i.e. you first have to get the organization to actually care about good goverance (perhaps by joining it, or using external levers).
[Of course, if we go through crazy explosive AI-driven growth then maybe the existing model of large organizations being slow will no longer be true—and hence there would be more scope for AI-assisted governance]
I definitely agree that it’s difficult to get organizations to improve governance. External pressure seems critical.
As stated in the post, I think that it’s possible that external pressure could come to AI capabilities organizations, in the form of regulation. Hard, but possible.
I’d (gently) push back against this part:
> I think it’s very hard to get even the most basic forms of good epistemic practices
I think that there are clearly some practices that seem good that don’t get used. But there are many ones that do get used, especially at well-run companies. In fact, I’d go so far to say that at least for the issue of “performance and capability” (rather than alignment/oversight), I’d trust the best-run organizations today a lot more than EA ideas of good techniques.
These organizations are often highly meritocratic, very intelligent, and leaders are good at cutting out the BS and honing in on key problems (at least, when doing so is useful to them).
I expect that our techniques like probabilities and forecastable statements just aren’t that great at these top levels. If much better practices come out, using AI, I’d feel good about them being used.
Or, at least for the part of “AIs helping organizations make tons of money by suggesting strategies and changes”, I’d expect businesses to be fairly efficient.
My main objection is that people working in government need to be able to get away with a mild level of lying and scheming to do their jobs (eg broker compromises, meet with constituents). AI could upset this equilibrium in a couple ways, making it harder to govern.
If the AI is just naive, it might do things like call out a politician for telling a harmless white lie, jeopardizing eg an international agreement that was about to be signed.
One response is that human overseers will discipline these naive mistakes, but the more human oversight is required, the more you run into the typical problems of human oversight you outlined above. “These evaluators can do so while not seeing critical private information” is not always true. (Eg if the AI realizes that Biden is telling a minor lie to Xi based on classified information, revealing the existence of the lie to the overseer would necessarily reveal classified information).
Even if the AI is not naive, and can distinguish white lies from outright misinformation, say, I still worry that it undermines the current equilibrium. The public would call for stricter and stricter oversight standards, while government workers will struggle to fight back because
That’s a bad look, and
The benefits of a small level of deception are hard to identify and articulate.
TLDR: Government needs some humans in the loop making decisions and working together. To work together, humans need some latitude to behave in ways that would become difficult with greater AI integration.
Thanks for vocalizing your thoughts!
1. I agree that these systems having severe flaws, especially if over-trusted, in ways that are similar to what I feel about managements. Finding ways to make sure they are reliable will be hard, though many organizations might want it enough to do much of that work anyway. I obviously wouldn’t suggest or encourage bad implementations.
2. I feel comfortable that we can choose where to apply it. I assume these systems should/can/would be rolled out gradually.
At the same time, I would hope that we could move towards an equilibrium with much less lying. A lot of lies in business are both highly normalized and definitely not white lies.
All in all, I’m proposing powerful technology. This is a tool, and it’s possible use almost any tool in incompetent or malicious ways, if you really want.
Welsh government commits to making lying in politics illegal
This sounds awesome at first blush, would love to see it battle-tested.
I am very pessimistic about this—my assumption is the state will use it to attack its enemies, even when they make true statements, while ignoring the falsehoods of its allies.
I think the example in the article is pretty strong evidence of this. They claim a politician is guilty of a “direct lie” for saying that a policy would entitle illegal immigrants to £1,600/month of welfare. The claim is somewhat misleading, because illegal immigrants would not be the only ones entitled. But it is true that some of those entitled would be illegal immigrants, and their inclusion was a deliberate policy choice.
If this is the canonical example they’re using to illustrate the rule, rather than something more objective and clear-cut, this makes me very pessimistic it will actually be applied in a truth-seeking manner. Rather this seems like it will undermine rational decision making and democracy—the state can simply declare opposition politicians to be liars and remove them from the ballot, preventing voters from course-correcting.
Note: I wrote a lot more about one specific technology for this here:
https://forum.effectivealtruism.org/posts/piAQ2qpiZEFwdKtmq/llm-secured-systems-a-general-purpose-tool-for-structured
robot domination of humans is a good thing, actually
aasimov wrote this first btw
Executive summary: AI-assisted governance could enable organizations to reach superhuman standards of integrity and competence in the coming decades, and warrants further investigation by the effective altruism community.
Key points:
AI-enhanced governance could allow organizations to achieve “99.99” reliability in avoiding corruption and deception.
Most risks of AI-assisted governance can likely be managed with reasonable implementation.
AI governance tools could complement AI company regulation and help ensure alignment with public interests.
Rapidly advancing AI capabilities, lack of promising alternatives, and potential for reliable oversight make AI-assisted governance promising.
A recursive “bootstrapping” approach, with increasingly stringent AI-enforced standards, could enable a positive feedback loop of safer and more capable AI governance over time.
Risks of increased complexity, overconfidence, and AI misuse in governance are serious but likely manageable with careful design and oversight.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
It is more than desirable, it is necessary:
https://forum.effectivealtruism.org/posts/6j6qgNa3uGmzJEMoN/artificial-intelligence-as-exit-strategy-from-the-age-of
This seems like a generally bad idea. I feel like the entire field of algorithmic bias is dedicated to explaining why this is generally a bad idea.
Neural networks, at least at the moment, are for the most part functionally black boxes. You feed in your data (say, the economic data of each state), and it does a bunch of calculations and spits out a recommendation (say, the funding you should allocate). But you can’t look inside and figure out the actual reasons that, say, Florida got X$ in funding and alaska got Y$. It’s just “what the algorithm spit out”. This is all based on your training data, which can be biased or incorrect.
Essentially, by relegating things to inscrutable AI systems, you remove all accountability from your decision making. If a person is racially biased, and making decisions on that front, you can interrogate them and analyse their justifications, and remove the rot. If an algorithm is racist, due to being trained on biased data, you can’t tell (you can observe unequal outcomes, but how do you know it was a result of bias, and not the other 21 factors that went into the model?).
And of course, we know that current day AI suffers from hallucinations, glitches, bugs, and so on. How do you know that a decision was made genuinely, or was just a glitch somewhere in the giant neural network matrix?
Rather than making things fairer and less corrupt, it seems like this just concentrates power in whoever is building the AI. Which also makes it an easier target for attacks by malevolent entities, of course.
This seems really overstated to me. My impression was that this field researches ways that algorithms could have significant issues. I don’t get the impression that the field is making the normative claim that these issues thereby mean that large classes of algorithms will always be net-negative.
I’ll also flag that humans also are very often black-boxes with huge and gaping flaws. And again—I’m not recommending replacing humans with AIs—I think often the thing to do is to augment them with AIs. I’d trust decisions recommended both by AIs and humans more than either one, at this point.
>it seems like this just concentrates power in whoever is building the AI
Again, much of my point in this piece is that AIs could be used to do the opposite of this.
Is your argument that it’s impossible for these sorts of algorithms to be used in positive ways? Or that you just expect that they will be mostly used in negative ways? If it’s the latter, do you think it’s a waste of time or research to try to figure out how to use them in positive ways, because any gains of that will be co-opted for negative use?
I have no problem with AI/machine learning being used in areas where the black box nature does not matter very much, and the consequences of hallucinations or bias are small.
My problem is with the idea of “superhuman governance”, where unaccountable black box machines make decisions that affect peoples lives significantly for reasons that cannot be dissected and explained.
Far from preventing corruption, I think this is a gift wrapped opportunity for the corrupt to hide their corruption behind the veneer of a “fair algorithm”. I don’t think it would be particularly hard to train a neural network to appear to be neutral while actually subtly favoring one outcome or the other, by manipulating the training data or the reward function. There would be no way to tell this manipulation occurred in the code, because the actual “code” of a neural network involves multiplying ginormous matrices of inscrutable numbers.
Of course, the more likely outcome is that this just happens by accident, and whatever biases and quirks occurred by accident due to inherently non-random data sampling get baked into the decisions affecting everybody.
Human decision making is spread over many, many people, so the impact of any one person being flawed is minimized. Taking humans out of the equation reduces the number of points of failure significantly.
Sure, if this is the case, then it’s not clear to me if/where we disagree.
I’d agree that it’s definitely possible to use these systems in poor ways, using all the downsides that you describe and more.
My main point is that I think we can also point to some worlds and some solutions that are quite positive in these cases. I’d also expect that there’s a lot of good that could be done by improving the positive AI/governance workflows, even if there also other bad AI/governance workflows in the world.
This is similar to how I think some technology is quite bad, but there’s also a lot of great technology—and often the solution to bad tech is good tech, not no tech.
I’ll also quickly flag that:
1. “AI” does cover a wide field.
2. LLMs and similar could be used to write high-reliability code with formal specifications, if desired, instead of being used directly. So we use lots of well-trusted code for key tasks, instead of more black-box-y methods.