Hi — I took you up on the invitation to try an alternative AI red teaming approach.
I built a multi-agent pipeline (decomposition → investigation → verification → quantification → adversarial testing → synthesis) and ran it against all three interventions where you published detailed AI output: water chlorination, ITNs, and SMC.
Results across three runs:
Signal rates: 84% (water), 100% (ITNs), 82% (SMC) — vs your reported ~15-30%
Zero hallucinated citations (the key architectural change: Investigators generate hypotheses without citing evidence, then a separate Verifier searches for real evidence)
Each surviving critique includes parameter mappings to specific CEA spreadsheet cells with computed sensitivity ranges
The most interesting finding was cross-intervention: three structural patterns appeared independently in all three analyses. All three CEAs model dynamic phenomena with static parameters (adherence decay, resistance evolution, efficacy degradation). All three collapse meaningful within-category variation into single aggregate parameters. And the two malaria interventions both lack mechanisms to capture biological adaptation by the target organism.
Your post mentions you covered six grantmaking areas total. The other three — CMAM, syphilis, and malaria vaccines — could be run through the pipeline as well. It doesn’t strictly require your AI output to function; that feeds into novelty filtering and baseline comparison, but the critiques themselves are generated independently. I’ve reached out separately about this.
Happy to discuss methodology, and happy to hear where you think the pipeline’s findings miss the mark — several of the cross-intervention patterns may reflect deliberate modeling choices rather than oversights, and I’d be interested to know which.
Hi — I took you up on the invitation to try an alternative AI red teaming approach.
I built a multi-agent pipeline (decomposition → investigation → verification → quantification → adversarial testing → synthesis) and ran it against all three interventions where you published detailed AI output: water chlorination, ITNs, and SMC.
Results across three runs:
Signal rates: 84% (water), 100% (ITNs), 82% (SMC) — vs your reported ~15-30%
Zero hallucinated citations (the key architectural change: Investigators generate hypotheses without citing evidence, then a separate Verifier searches for real evidence)
Each surviving critique includes parameter mappings to specific CEA spreadsheet cells with computed sensitivity ranges
The most interesting finding was cross-intervention: three structural patterns appeared independently in all three analyses. All three CEAs model dynamic phenomena with static parameters (adherence decay, resistance evolution, efficacy degradation). All three collapse meaningful within-category variation into single aggregate parameters. And the two malaria interventions both lack mechanisms to capture biological adaptation by the target organism.
I wrote up the full results here: tsondo.com/blog/three-interventions-same-structural-patterns/
Phase 1 write-up (methodology explanation): tsondo.com/blog/give-well-red-team/
The full pipeline, prompts, and results are open source: github.com/tsondo/givewell_redteam
Your post mentions you covered six grantmaking areas total. The other three — CMAM, syphilis, and malaria vaccines — could be run through the pipeline as well. It doesn’t strictly require your AI output to function; that feeds into novelty filtering and baseline comparison, but the critiques themselves are generated independently. I’ve reached out separately about this.
Happy to discuss methodology, and happy to hear where you think the pipeline’s findings miss the mark — several of the cross-intervention patterns may reflect deliberate modeling choices rather than oversights, and I’d be interested to know which.