Consider these types of questions that AI systems might help address:
What strategic missteps is Microsoft making in terms of maximizing market value?
What metrics could better evaluate the competence of business and political leaders?
Which public companies would be best off by firing their CEOs?
<...>
I’m open to the possibility that a future AI may well be able to answer these questions more quickly and more effectively than the typical human who currently handles those questions.
The tricky thing is how to test this.
Given that these are not easily testable things, I think it might be hard for people to gain enough confidence in the AI to actually use it. (I guess that too might be surmountable, but it’s not immediately obvious to me how)
I agree that testing it is difficult. I partially addressed this above in the section on “Strategy and Verifiability”.
I would flag that people should arguably be equally suspicious of most humans. As we come up with various tests and evals, I expect that mostly the best AIs will have mediocre results, and most prominent humans will just refuse to be tested (we can still do lighter evals on public intellectuals and such using their available works, but this will be more limited).
Prediction markets seem like a pretty good test to me, though they are only one implementation.
I expect that with decent systems, we should have new “epistemic table stakes” of things like:
Can do forecasting in a wide variety of fields, at least roughly as good as Metaculus forecasters with say 10hrs per question
In extensive simulations, has low amounts of logical inconsistencies
Flags all claims that users might not agree with
Very low rates of hallucinations
Biases have been extensively tested under different situations
Extensive red-teaming by other top AI systems
Predictions of how well this AI will hold up, in comparison to better AI intellectual systems in 10 to 40 years.
Full oversight/visibility of potential conflicts of interest.
(I’m not saying that these systems will be broadly-trusted, just that they will exist. I would expect the smarter people at least to trust them, in accordance to their evals.)
You wrote:
I’m open to the possibility that a future AI may well be able to answer these questions more quickly and more effectively than the typical human who currently handles those questions.
The tricky thing is how to test this.
Given that these are not easily testable things, I think it might be hard for people to gain enough confidence in the AI to actually use it. (I guess that too might be surmountable, but it’s not immediately obvious to me how)
Thanks for raising the concern!
I agree that testing it is difficult. I partially addressed this above in the section on “Strategy and Verifiability”.
I would flag that people should arguably be equally suspicious of most humans. As we come up with various tests and evals, I expect that mostly the best AIs will have mediocre results, and most prominent humans will just refuse to be tested (we can still do lighter evals on public intellectuals and such using their available works, but this will be more limited).
Prediction markets seem like a pretty good test to me, though they are only one implementation.
I expect that with decent systems, we should have new “epistemic table stakes” of things like:
Can do forecasting in a wide variety of fields, at least roughly as good as Metaculus forecasters with say 10hrs per question
In extensive simulations, has low amounts of logical inconsistencies
Flags all claims that users might not agree with
Very low rates of hallucinations
Biases have been extensively tested under different situations
Extensive red-teaming by other top AI systems
Predictions of how well this AI will hold up, in comparison to better AI intellectual systems in 10 to 40 years.
Full oversight/visibility of potential conflicts of interest.
(I’m not saying that these systems will be broadly-trusted, just that they will exist. I would expect the smarter people at least to trust them, in accordance to their evals.)