Tsondo

• Tsondo 16 Apr 2026 14:17 UTC

  1 point

  0 ∶ 0

  in reply to: Morgan Fairless’s comment on: Three struc­tural pat­terns across eleven AIM cost-effec­tive­ness analyses

  Thanks Morgan! I appreciate the detailed response from you, Filip, and Vicky. Glad the patterns were useful. I’ll look out for your email and am happy to dig into whatever direction would be most useful on your end.

  Tsondo

• Tsondo 11 Apr 2026 13:17 UTC

  1 point

  0 ∶ 0

  in reply to: Toby Tremlett🔹’s comment on: Three struc­tural pat­terns across eleven AIM cost-effec­tive­ness analyses

  Thanks Toby, both the specific guidance and the offer are super helpful.

  The asymmetric-reach point is something I had not thought of, but makes perfect sense. Even when a misunderstanding would be the org’s to own, the critique travels further than the correction, so the pre-share is doing real work beyond politeness. I’ll treat it as the default going forward, and I’ll take you up on the DM route next time rather than trying to find a warm lead myself. That removes my main hesitation.

  And good catch on the link-post vs cross-post distinction. I didn’t get that, either, somewhat conflated them. For the next one I’ll put the full text on the Forum with a canonical link back. You’re right that the comments are where the methodology actually gets pressure-tested, and honestly, my blog has no commenting enabled, so the Forum is clearly where the conversation should live. Appreciate you taking the time to flag both of these.

• Tsondo 8 Apr 2026 17:51 UTC

  3 points

  0 ∶ 0

  in reply to: Toby Tremlett🔹’s comment on: Three struc­tural pat­terns across eleven AIM cost-effec­tive­ness analyses

  I sent Vicky an email at the time of publishing rather than sharing a draft in advance. I didn’t pre-share for two reasons: I don’t have an existing relationship with anyone at AIM, so cold-emailing a stranger and asking them to review a draft on a deadline felt like a heavier ask than a heads-up at publication, and I tried to write the post so that any factual claim can be verified in a few minutes from the public spreadsheets (the cell references in each finding are there for that purpose). Vasco Grilo’s recent methodology posts on AIM CEAs are part of the same public conversation, and Vicky has engaged with those publicly on the Forum, which is part of why the published-with-heads-up route felt appropriate for a contribution in the same vein.

  That said, I’m new to posting on the Forum and I’d take your read on the convention seriously. If the norm here is that pre-sharing should happen even when the analysis is grounded in public data, I’d want to know for next time.

• Tsondo 8 Apr 2026 17:48 UTC

  1 point

  0 ∶ 0

  in reply to: tobycrisford 🔸’s comment on: GiveWell’s AI red-team­ing limi­ta­tions aren’t a model prob­lem — they’re an ar­chi­tec­ture problem

  If you read my blog post, I go into detail about why this is not a model issue. It’s about how you frame the question much more than what the model contains. For this purpose any decent model would have had the same result. The main benefit that Claude gives is direct in terminal code writing and execution.

• Tsondo 8 Apr 2026 17:46 UTC

  1 point

  0 ∶ 0

  in reply to: titotal’s comment on: GiveWell’s AI red-team­ing limi­ta­tions aren’t a model prob­lem — they’re an ar­chi­tec­ture problem

  The model I use did have an earlier cut off for it’s data, but that isn’t relevant for what I am doing. My write up actually surfaced several things that they didn’t see at all. That’s the point, really. And for the verication, it did not look at GiveWell at all. My verification sources are all listed in the code if you are concerned. All reputable sources.

Three struc­tural pat­terns across eleven AIM cost-effec­tive­ness analyses

• Tsondo 31 Mar 2026 14:29 UTC

  −1 points

  1 ∶ 0

  in reply to: Brendan Phillips🔸’s comment on: GiveWell’s AI red-team­ing limi­ta­tions aren’t a model prob­lem — they’re an ar­chi­tec­ture problem

  Good to hear! All of my work is there on github. Please have a look at the results. If my pipeline found something that yours didn’t, it might be worth integrating the methodology.
  
  I’d be very happy to discuss with you at your convenience. I’m in Central EU time (Italy.) I also sent you an email via research@GiveWell.org. Hannah says she will pass it on to you.
  

• Tsondo 31 Mar 2026 14:24 UTC

  1 point

  0 ∶ 0

  in reply to: Brendan Phillips🔸’s comment on: AI Red Team­ing at GiveWell: What We’ve Learned (and Where We’d Wel­come Your In­put)

  If you want help converting to a database let me know. It looks like a weekend project. We could also develop a front end for easier input, if you want. I would be happy to assist you.

• Tsondo 31 Mar 2026 14:22 UTC

  1 point

  0 ∶ 0

  in reply to: david_reinstein’s comment on: AI Red Team­ing at GiveWell: What We’ve Learned (and Where We’d Wel­come Your In­put)

  When I ran my multi-agent pipeline on the data, I had to create unique parsing rules for each data set due to inconsistencies in the spreadsheets. One of my suggestions going forward would be to standardize how they collect and store the data to make it more machine readable. But you are right. Claude Code can sift through it and sort it out, with the right prompting, at the right stage. It is just one more context to manage.

• Tsondo 31 Mar 2026 14:18 UTC

  1 point

  0 ∶ 0

  on: AI Red Team­ing at GiveWell: What We’ve Learned (and Where We’d Wel­come Your In­put)

  Hi — I took you up on the invitation to try an alternative AI red teaming approach.

  I built a multi-agent pipeline (decomposition → investigation → verification → quantification → adversarial testing → synthesis) and ran it against all three interventions where you published detailed AI output: water chlorination, ITNs, and SMC.

  Results across three runs:

  • Signal rates: 84% (water), 100% (ITNs), 82% (SMC) — vs your reported ~15-30%

  • Zero hallucinated citations (the key architectural change: Investigators generate hypotheses without citing evidence, then a separate Verifier searches for real evidence)

  • Each surviving critique includes parameter mappings to specific CEA spreadsheet cells with computed sensitivity ranges

  The most interesting finding was cross-intervention: three structural patterns appeared independently in all three analyses. All three CEAs model dynamic phenomena with static parameters (adherence decay, resistance evolution, efficacy degradation). All three collapse meaningful within-category variation into single aggregate parameters. And the two malaria interventions both lack mechanisms to capture biological adaptation by the target organism.

  I wrote up the full results here: tsondo.com/​blog/​three-interventions-same-structural-patterns/​

  Phase 1 write-up (methodology explanation): tsondo.com/​blog/​give-well-red-team/​

  The full pipeline, prompts, and results are open source: github.com/​tsondo/​givewell_redteam

  Your post mentions you covered six grantmaking areas total. The other three — CMAM, syphilis, and malaria vaccines — could be run through the pipeline as well. It doesn’t strictly require your AI output to function; that feeds into novelty filtering and baseline comparison, but the critiques themselves are generated independently. I’ve reached out separately about this.

  Happy to discuss methodology, and happy to hear where you think the pipeline’s findings miss the mark — several of the cross-intervention patterns may reflect deliberate modeling choices rather than oversights, and I’d be interested to know which.

GiveWell’s AI red-team­ing limi­ta­tions aren’t a model prob­lem — they’re an ar­chi­tec­ture problem