On your first point, if researchers found any of the critiques raised by the AI to be credible, we would follow up in the same chat to ask for sourcing or additional information (like for the water turbidity point). We found this more successful than asking for heavy citations in the initial output, which led to lower quality critiques (presumably because we were giving too many instructions). The most recent generation of models are better at following complex instructions, so I expect we should revisit this.
Human researchers spent ~75 minutes reviewing AI output per intervention. We haven’t rigorously compared this to having researchers look into the literature themselves, but my sense is that currently using AI is worth it for areas where we haven’t done as much research (family planning, syphilis treatment) and not worth it in areas where we’ve done a lot of research (bed nets, vaccinations). Given the pace of improvement we’ve seen over the past year, I expect a preliminary AI red teaming pass will be worthwhile in almost all cases by the end of 2026.
Thanks for the feedback!
On your first point, if researchers found any of the critiques raised by the AI to be credible, we would follow up in the same chat to ask for sourcing or additional information (like for the water turbidity point). We found this more successful than asking for heavy citations in the initial output, which led to lower quality critiques (presumably because we were giving too many instructions). The most recent generation of models are better at following complex instructions, so I expect we should revisit this.
Human researchers spent ~75 minutes reviewing AI output per intervention. We haven’t rigorously compared this to having researchers look into the literature themselves, but my sense is that currently using AI is worth it for areas where we haven’t done as much research (family planning, syphilis treatment) and not worth it in areas where we’ve done a lot of research (bed nets, vaccinations). Given the pace of improvement we’ve seen over the past year, I expect a preliminary AI red teaming pass will be worthwhile in almost all cases by the end of 2026.