I don’t mind you using LLMs for elucidating discussion, although I don’t think asking it to rate arguments is very valuable.
The additional details of having subfield specific auditors that are opt-in does lessen my objections significantly. Of course, the issue of what counts as a subfield is kinda thorny. It would make most sense for, as claude suggests, journals to have an “auditor verified” badge, but then maybe you’re giving too much power over content to the journals, which usually stick to accept/reject decisions (and even that can get quite political).
Coming back to your original statement, ultimately I just don’t buy that any of this can lead to “incredibly low rates of fraud/bias”. If someone wants to do fraud or bias, they will just game the tools, or submit to journals with weak/nonexistent auditors. Perhaps the black box nature of AI might even make it easier to hide this kind of thing.
Next: there are large areas of science where a tool telling you the best techniques to use will never be particularly useful. On the one hand there is research like mine, where it’s so frontier that the “best practices” to put into such an auditor don’t exist yet. On the other, you have statistics stuff that is so well known that there already exist software tools that implement the best practices: you just have to load up a well documented R package. What does an AI auditor add to this?
If I was tasked with reducing bias and fraud, I would mainly push for data transparency requirements in journal publications, and in beefing up the incentives for careful peer review, which is currently unpaid and unrewarding labour. Perhaps AI tools could be useful in parts of that process, but I don’t see it as anywhere near as important than those other two things.
Looking back, I think this part of my first comment was poorly worded: > I imagine that scientists will soon have the ability to be unusually transparent and provide incredibly low rates of fraud/bias, using AI.
I meant > I imagine that scientists will [soon have the ability to] be unusually transparent and provide incredibly low rates of fraud/bias], using AI.
So it’s not that this will lead to low rates of fraud/bias, but that AI will help enable that for scientists willing to go along with it—but at the same time, there’s a separate question of if scientists are willing to go along with it.
But I think even that probably is not fair. A a better description of my beliefs is something like,
I think that LLM auditing tools could be useful for some kinds of scientific research for communities open to them.
I think in the short-term, sufficiently-motivated groups could develop these tools and use them to help decrease the levels of statistical and algorithmic accidents that happen. Correspondingly, I’d expect this to help with fraud.
In the long-run, whenever AI approaches human-level intelligence (which I think will likely happen in the next 20 years, but I realize others disagree), I expect that more and more of the scientific process will be automated. I think there are ways this could go very well using things like AI auditing, whereby the results will be much more reliable than those currently made by humans. There are of course also worlds in which humans do dumb things with the AIs and the opposite happens.
I think that at least, AI safety researchers should consider using these kinds of methods, and that the AI safety landscape should investigate efforts to make decent auditing tools.”
My core hope with the original message is to draw attention to AI science auditing tools as things that might be interesting/useful, not to claim that they’re definitely a major game changer.
I don’t mind you using LLMs for elucidating discussion, although I don’t think asking it to rate arguments is very valuable.
The additional details of having subfield specific auditors that are opt-in does lessen my objections significantly. Of course, the issue of what counts as a subfield is kinda thorny. It would make most sense for, as claude suggests, journals to have an “auditor verified” badge, but then maybe you’re giving too much power over content to the journals, which usually stick to accept/reject decisions (and even that can get quite political).
Coming back to your original statement, ultimately I just don’t buy that any of this can lead to “incredibly low rates of fraud/bias”. If someone wants to do fraud or bias, they will just game the tools, or submit to journals with weak/nonexistent auditors. Perhaps the black box nature of AI might even make it easier to hide this kind of thing.
Next: there are large areas of science where a tool telling you the best techniques to use will never be particularly useful. On the one hand there is research like mine, where it’s so frontier that the “best practices” to put into such an auditor don’t exist yet. On the other, you have statistics stuff that is so well known that there already exist software tools that implement the best practices: you just have to load up a well documented R package. What does an AI auditor add to this?
If I was tasked with reducing bias and fraud, I would mainly push for data transparency requirements in journal publications, and in beefing up the incentives for careful peer review, which is currently unpaid and unrewarding labour. Perhaps AI tools could be useful in parts of that process, but I don’t see it as anywhere near as important than those other two things.
This context is useful, thanks.
Looking back, I think this part of my first comment was poorly worded:
> I imagine that scientists will soon have the ability to be unusually transparent and provide incredibly low rates of fraud/bias, using AI.
I meant
> I imagine that scientists will [soon have the ability to] be unusually transparent and provide incredibly low rates of fraud/bias], using AI.
So it’s not that this will lead to low rates of fraud/bias, but that AI will help enable that for scientists willing to go along with it—but at the same time, there’s a separate question of if scientists are willing to go along with it.
But I think even that probably is not fair. A a better description of my beliefs is something like,
I think that LLM auditing tools could be useful for some kinds of scientific research for communities open to them.
I think in the short-term, sufficiently-motivated groups could develop these tools and use them to help decrease the levels of statistical and algorithmic accidents that happen. Correspondingly, I’d expect this to help with fraud.
In the long-run, whenever AI approaches human-level intelligence (which I think will likely happen in the next 20 years, but I realize others disagree), I expect that more and more of the scientific process will be automated. I think there are ways this could go very well using things like AI auditing, whereby the results will be much more reliable than those currently made by humans. There are of course also worlds in which humans do dumb things with the AIs and the opposite happens.
I think that at least, AI safety researchers should consider using these kinds of methods, and that the AI safety landscape should investigate efforts to make decent auditing tools.”
My core hope with the original message is to draw attention to AI science auditing tools as things that might be interesting/useful, not to claim that they’re definitely a major game changer.