Thanks for the context but putting these caveats in the disclaimers doesn’t change how the tool works.
The underlying studies are solid. The problem is the layer on top of them. The scores and weights are AI set guesses on a slider, not field data. Again, that’s most true for the civic engagement, equity and justice buckets, which is where most of Scott’s money goes. The author says himself those buckets get “wide skeptical priors” because no health pathway has credible evidence. That isn’t a real effect adjusted for uncertainty. It’s a number filling in for missing data. Since those are the biggest buckets, that number is doing a lot of the work behind the headline frontier multiple. The comparison is weakest exactly where most of her money sits.
That’s my real concern. The tool takes her hardest to measure work and makes it look like it buys almost nothing, when that low number is a measurement gap, not a fact about the work. A model is only as strong as its weakest layer. For most of this portfolio, that layer is a guess standing in for how funding actually works on the ground, like whether a local coalition can absorb a sudden influx of cash, or the fact that structural change doesn’t run in a straight line to a health outcome.
Flagging that in the fine print is good practice, but it doesn’t fix it. The calculator is still running precise maths on top of guesses.
Hm, I empathise with your underlying concern, but don’t the sliders help with that?
(Tangentially, “precise maths on top of guesses” is basically how I’d describe all CEAs, including GiveWell’s which move hundreds of millions annually. The guesses get better in all kinds of ways, but at least when I last checked the AMF CEA it wasn’t using field data like you’d like. So in a sense you seem to be asking for more than whole teams of specialists can deliver, let alone one person and a bunch of subscriptions can do in their limited spare time?)
Thanks for the context but putting these caveats in the disclaimers doesn’t change how the tool works.
The underlying studies are solid. The problem is the layer on top of them. The scores and weights are AI set guesses on a slider, not field data. Again, that’s most true for the civic engagement, equity and justice buckets, which is where most of Scott’s money goes. The author says himself those buckets get “wide skeptical priors” because no health pathway has credible evidence. That isn’t a real effect adjusted for uncertainty. It’s a number filling in for missing data. Since those are the biggest buckets, that number is doing a lot of the work behind the headline frontier multiple. The comparison is weakest exactly where most of her money sits.
That’s my real concern. The tool takes her hardest to measure work and makes it look like it buys almost nothing, when that low number is a measurement gap, not a fact about the work. A model is only as strong as its weakest layer. For most of this portfolio, that layer is a guess standing in for how funding actually works on the ground, like whether a local coalition can absorb a sudden influx of cash, or the fact that structural change doesn’t run in a straight line to a health outcome.
Flagging that in the fine print is good practice, but it doesn’t fix it. The calculator is still running precise maths on top of guesses.
Hm, I empathise with your underlying concern, but don’t the sliders help with that?
(Tangentially, “precise maths on top of guesses” is basically how I’d describe all CEAs, including GiveWell’s which move hundreds of millions annually. The guesses get better in all kinds of ways, but at least when I last checked the AMF CEA it wasn’t using field data like you’d like. So in a sense you seem to be asking for more than whole teams of specialists can deliver, let alone one person and a bunch of subscriptions can do in their limited spare time?)