This seems like a generally bad idea. I feel like the entire field of algorithmic bias is dedicated to explaining why this is generally a bad idea.
Neural networks, at least at the moment, are for the most part functionally black boxes. You feed in your data (say, the economic data of each state), and it does a bunch of calculations and spits out a recommendation (say, the funding you should allocate). But you can’t look inside and figure out the actual reasons that, say, Florida got X$ in funding and alaska got Y$. It’s just “what the algorithm spit out”. This is all based on your training data, which can be biased or incorrect.
Essentially, by relegating things to inscrutable AI systems, you remove all accountability from your decision making. If a person is racially biased, and making decisions on that front, you can interrogate them and analyse their justifications, and remove the rot. If an algorithm is racist, due to being trained on biased data, you can’t tell (you can observe unequal outcomes, but how do you know it was a result of bias, and not the other 21 factors that went into the model?).
And of course, we know that current day AI suffers from hallucinations, glitches, bugs, and so on. How do you know that a decision was made genuinely, or was just a glitch somewhere in the giant neural network matrix?
Rather than making things fairer and less corrupt, it seems like this just concentrates power in whoever is building the AI. Which also makes it an easier target for attacks by malevolent entities, of course.
This seems like a generally bad idea. I feel like the entire field of algorithmic bias is dedicated to explaining why this is generally a bad idea.
This seems really overstated to me. My impression was that this field researches ways that algorithms could have significant issues. I don’t get the impression that the field is making the normative claim that these issues thereby mean that large classes of algorithms will always be net-negative.
I’ll also flag that humans also are very often black-boxes with huge and gaping flaws. And again—I’m not recommending replacing humans with AIs—I think often the thing to do is to augment them with AIs. I’d trust decisions recommended both by AIs and humans more than either one, at this point.
>it seems like this just concentrates power in whoever is building the AI Again, much of my point in this piece is that AIs could be used to do the opposite of this.
Is your argument that it’s impossible for these sorts of algorithms to be used in positive ways? Or that you just expect that they will be mostly used in negative ways? If it’s the latter, do you think it’s a waste of time or research to try to figure out how to use them in positive ways, because any gains of that will be co-opted for negative use?
I have no problem with AI/machine learning being used in areas where the black box nature does not matter very much, and the consequences of hallucinations or bias are small.
My problem is with the idea of “superhuman governance”, where unaccountable black box machines make decisions that affect peoples lives significantly for reasons that cannot be dissected and explained.
Far from preventing corruption, I think this is a gift wrapped opportunity for the corrupt to hide their corruption behind the veneer of a “fair algorithm”. I don’t think it would be particularly hard to train a neural network to appear to be neutral while actually subtly favoring one outcome or the other, by manipulating the training data or the reward function. There would be no way to tell this manipulation occurred in the code, because the actual “code” of a neural network involves multiplying ginormous matrices of inscrutable numbers.
Of course, the more likely outcome is that this just happens by accident, and whatever biases and quirks occurred by accident due to inherently non-random data sampling get baked into the decisions affecting everybody.
Human decision making is spread over many, many people, so the impact of any one person being flawed is minimized. Taking humans out of the equation reduces the number of points of failure significantly.
I have no problem with AI/machine learning being used in areas where the black box nature does not matter very much, and the consequences of hallucinations or bias are small.
Sure, if this is the case, then it’s not clear to me if/where we disagree.
I’d agree that it’s definitely possible to use these systems in poor ways, using all the downsides that you describe and more.
My main point is that I think we can also point to some worlds and some solutions that are quite positive in these cases. I’d also expect that there’s a lot of good that could be done by improving the positive AI/governance workflows, even if there also other bad AI/governance workflows in the world.
This is similar to how I think some technology is quite bad, but there’s also a lot of great technology—and often the solution to bad tech is good tech, not no tech.
1. “AI” does cover a wide field. 2. LLMs and similar could be used to write high-reliability code with formal specifications, if desired, instead of being used directly. So we use lots of well-trusted code for key tasks, instead of more black-box-y methods.
This seems like a generally bad idea. I feel like the entire field of algorithmic bias is dedicated to explaining why this is generally a bad idea.
Neural networks, at least at the moment, are for the most part functionally black boxes. You feed in your data (say, the economic data of each state), and it does a bunch of calculations and spits out a recommendation (say, the funding you should allocate). But you can’t look inside and figure out the actual reasons that, say, Florida got X$ in funding and alaska got Y$. It’s just “what the algorithm spit out”. This is all based on your training data, which can be biased or incorrect.
Essentially, by relegating things to inscrutable AI systems, you remove all accountability from your decision making. If a person is racially biased, and making decisions on that front, you can interrogate them and analyse their justifications, and remove the rot. If an algorithm is racist, due to being trained on biased data, you can’t tell (you can observe unequal outcomes, but how do you know it was a result of bias, and not the other 21 factors that went into the model?).
And of course, we know that current day AI suffers from hallucinations, glitches, bugs, and so on. How do you know that a decision was made genuinely, or was just a glitch somewhere in the giant neural network matrix?
Rather than making things fairer and less corrupt, it seems like this just concentrates power in whoever is building the AI. Which also makes it an easier target for attacks by malevolent entities, of course.
This seems really overstated to me. My impression was that this field researches ways that algorithms could have significant issues. I don’t get the impression that the field is making the normative claim that these issues thereby mean that large classes of algorithms will always be net-negative.
I’ll also flag that humans also are very often black-boxes with huge and gaping flaws. And again—I’m not recommending replacing humans with AIs—I think often the thing to do is to augment them with AIs. I’d trust decisions recommended both by AIs and humans more than either one, at this point.
>it seems like this just concentrates power in whoever is building the AI
Again, much of my point in this piece is that AIs could be used to do the opposite of this.
Is your argument that it’s impossible for these sorts of algorithms to be used in positive ways? Or that you just expect that they will be mostly used in negative ways? If it’s the latter, do you think it’s a waste of time or research to try to figure out how to use them in positive ways, because any gains of that will be co-opted for negative use?
I have no problem with AI/machine learning being used in areas where the black box nature does not matter very much, and the consequences of hallucinations or bias are small.
My problem is with the idea of “superhuman governance”, where unaccountable black box machines make decisions that affect peoples lives significantly for reasons that cannot be dissected and explained.
Far from preventing corruption, I think this is a gift wrapped opportunity for the corrupt to hide their corruption behind the veneer of a “fair algorithm”. I don’t think it would be particularly hard to train a neural network to appear to be neutral while actually subtly favoring one outcome or the other, by manipulating the training data or the reward function. There would be no way to tell this manipulation occurred in the code, because the actual “code” of a neural network involves multiplying ginormous matrices of inscrutable numbers.
Of course, the more likely outcome is that this just happens by accident, and whatever biases and quirks occurred by accident due to inherently non-random data sampling get baked into the decisions affecting everybody.
Human decision making is spread over many, many people, so the impact of any one person being flawed is minimized. Taking humans out of the equation reduces the number of points of failure significantly.
Sure, if this is the case, then it’s not clear to me if/where we disagree.
I’d agree that it’s definitely possible to use these systems in poor ways, using all the downsides that you describe and more.
My main point is that I think we can also point to some worlds and some solutions that are quite positive in these cases. I’d also expect that there’s a lot of good that could be done by improving the positive AI/governance workflows, even if there also other bad AI/governance workflows in the world.
This is similar to how I think some technology is quite bad, but there’s also a lot of great technology—and often the solution to bad tech is good tech, not no tech.
I’ll also quickly flag that:
1. “AI” does cover a wide field.
2. LLMs and similar could be used to write high-reliability code with formal specifications, if desired, instead of being used directly. So we use lots of well-trusted code for key tasks, instead of more black-box-y methods.