I would agree that LLMs are much stronger at finding and summarizing information than original thought (which is a big limitation for red teaming). However, weâve gotten a lot of utility out of having âSuperGoogleâ research a topic and then look for ways that our intervention reports differ from published literature. You could argue this is still SuperGoogle behavior (search + comparison) rather than genuine critical thinking, but for our purposes, thatâs been enough to surface a handful of worthwhile leads per intervention.
This is why weâve found that AI red teaming works best for well-researched interventions where we at GiveWell havenât done as much research (like syphilis) and doesnât work well for interventions where weâve done a lot of research (like insecticide-treated bed nets) or that are relatively new and donât have as much published literature (like malaria vaccines).
Thanks for your comment!
I would agree that LLMs are much stronger at finding and summarizing information than original thought (which is a big limitation for red teaming). However, weâve gotten a lot of utility out of having âSuperGoogleâ research a topic and then look for ways that our intervention reports differ from published literature. You could argue this is still SuperGoogle behavior (search + comparison) rather than genuine critical thinking, but for our purposes, thatâs been enough to surface a handful of worthwhile leads per intervention.
This is why weâve found that AI red teaming works best for well-researched interventions where we at GiveWell havenât done as much research (like syphilis) and doesnât work well for interventions where weâve done a lot of research (like insecticide-treated bed nets) or that are relatively new and donât have as much published literature (like malaria vaccines).