This seems like a generally bad idea. I feel like the entire field of algorithmic bias is dedicated to explaining why this is generally a bad idea.
This seems really overstated to me. My impression was that this field researches ways that algorithms could have significant issues. I don’t get the impression that the field is making the normative claim that these issues thereby mean that large classes of algorithms will always be net-negative.
I’ll also flag that humans also are very often black-boxes with huge and gaping flaws. And again—I’m not recommending replacing humans with AIs—I think often the thing to do is to augment them with AIs. I’d trust decisions recommended both by AIs and humans more than either one, at this point.
>it seems like this just concentrates power in whoever is building the AI Again, much of my point in this piece is that AIs could be used to do the opposite of this.
Is your argument that it’s impossible for these sorts of algorithms to be used in positive ways? Or that you just expect that they will be mostly used in negative ways? If it’s the latter, do you think it’s a waste of time or research to try to figure out how to use them in positive ways, because any gains of that will be co-opted for negative use?
I have no problem with AI/machine learning being used in areas where the black box nature does not matter very much, and the consequences of hallucinations or bias are small.
My problem is with the idea of “superhuman governance”, where unaccountable black box machines make decisions that affect peoples lives significantly for reasons that cannot be dissected and explained.
Far from preventing corruption, I think this is a gift wrapped opportunity for the corrupt to hide their corruption behind the veneer of a “fair algorithm”. I don’t think it would be particularly hard to train a neural network to appear to be neutral while actually subtly favoring one outcome or the other, by manipulating the training data or the reward function. There would be no way to tell this manipulation occurred in the code, because the actual “code” of a neural network involves multiplying ginormous matrices of inscrutable numbers.
Of course, the more likely outcome is that this just happens by accident, and whatever biases and quirks occurred by accident due to inherently non-random data sampling get baked into the decisions affecting everybody.
Human decision making is spread over many, many people, so the impact of any one person being flawed is minimized. Taking humans out of the equation reduces the number of points of failure significantly.
I have no problem with AI/machine learning being used in areas where the black box nature does not matter very much, and the consequences of hallucinations or bias are small.
Sure, if this is the case, then it’s not clear to me if/where we disagree.
I’d agree that it’s definitely possible to use these systems in poor ways, using all the downsides that you describe and more.
My main point is that I think we can also point to some worlds and some solutions that are quite positive in these cases. I’d also expect that there’s a lot of good that could be done by improving the positive AI/governance workflows, even if there also other bad AI/governance workflows in the world.
This is similar to how I think some technology is quite bad, but there’s also a lot of great technology—and often the solution to bad tech is good tech, not no tech.
1. “AI” does cover a wide field. 2. LLMs and similar could be used to write high-reliability code with formal specifications, if desired, instead of being used directly. So we use lots of well-trusted code for key tasks, instead of more black-box-y methods.
This seems really overstated to me. My impression was that this field researches ways that algorithms could have significant issues. I don’t get the impression that the field is making the normative claim that these issues thereby mean that large classes of algorithms will always be net-negative.
I’ll also flag that humans also are very often black-boxes with huge and gaping flaws. And again—I’m not recommending replacing humans with AIs—I think often the thing to do is to augment them with AIs. I’d trust decisions recommended both by AIs and humans more than either one, at this point.
>it seems like this just concentrates power in whoever is building the AI
Again, much of my point in this piece is that AIs could be used to do the opposite of this.
Is your argument that it’s impossible for these sorts of algorithms to be used in positive ways? Or that you just expect that they will be mostly used in negative ways? If it’s the latter, do you think it’s a waste of time or research to try to figure out how to use them in positive ways, because any gains of that will be co-opted for negative use?
I have no problem with AI/machine learning being used in areas where the black box nature does not matter very much, and the consequences of hallucinations or bias are small.
My problem is with the idea of “superhuman governance”, where unaccountable black box machines make decisions that affect peoples lives significantly for reasons that cannot be dissected and explained.
Far from preventing corruption, I think this is a gift wrapped opportunity for the corrupt to hide their corruption behind the veneer of a “fair algorithm”. I don’t think it would be particularly hard to train a neural network to appear to be neutral while actually subtly favoring one outcome or the other, by manipulating the training data or the reward function. There would be no way to tell this manipulation occurred in the code, because the actual “code” of a neural network involves multiplying ginormous matrices of inscrutable numbers.
Of course, the more likely outcome is that this just happens by accident, and whatever biases and quirks occurred by accident due to inherently non-random data sampling get baked into the decisions affecting everybody.
Human decision making is spread over many, many people, so the impact of any one person being flawed is minimized. Taking humans out of the equation reduces the number of points of failure significantly.
Sure, if this is the case, then it’s not clear to me if/where we disagree.
I’d agree that it’s definitely possible to use these systems in poor ways, using all the downsides that you describe and more.
My main point is that I think we can also point to some worlds and some solutions that are quite positive in these cases. I’d also expect that there’s a lot of good that could be done by improving the positive AI/governance workflows, even if there also other bad AI/governance workflows in the world.
This is similar to how I think some technology is quite bad, but there’s also a lot of great technology—and often the solution to bad tech is good tech, not no tech.
I’ll also quickly flag that:
1. “AI” does cover a wide field.
2. LLMs and similar could be used to write high-reliability code with formal specifications, if desired, instead of being used directly. So we use lots of well-trusted code for key tasks, instead of more black-box-y methods.