Perhaps helpful to disambiguate what a person cares about from that person’s decision-making processes (and from the incentive gradients which compose the environment wherein those processes are trained).
Yeah to be clear I meant that the decision making processes are probably informed by these things even if the metrics presented to donors are not, and from the looks of Ben’s comment above this is indeed the case.
We make the impact evaluation I note above available to donors (and our donors also do their own version of it). We also publish top line results publicly in our annual reviews (e.g. number of impact-adjusted plan changes) , but don’t publish the case studies since they involve a ton of sensitive personal information.
Because the orgs in question have literally said so, because I think the people working there genuinely care about their impact and are competent enough to have heard of Goodhart’s law, and because in several cases there have been major strategy changes which cannot be explained by a model of “everyone working there has a massive blindspot and is focused on easy to meet targets”. As one concrete example, 80k’s focus has switched to be very explicitly longtermist, which it was not originally. They’ve also published several articles about areas of their thinking which were wrong or could have been improved, which again I would not expect for an organisation merely focused on gaming its own metrics.
I think there’s likely a difference here between:
and
I’d be extremely surprised if 80k didn’t care about the impact of their advisees, or AMF didn’t care about reducing malaria.
Perhaps helpful to disambiguate what a person cares about from that person’s decision-making processes (and from the incentive gradients which compose the environment wherein those processes are trained).
Yeah to be clear I meant that the decision making processes are probably informed by these things even if the metrics presented to donors are not, and from the looks of Ben’s comment above this is indeed the case.
We make the impact evaluation I note above available to donors (and our donors also do their own version of it). We also publish top line results publicly in our annual reviews (e.g. number of impact-adjusted plan changes) , but don’t publish the case studies since they involve a ton of sensitive personal information.
Why do you think the decision-making processes are informed by things that aren’t presented to donors / supporters?
Because the orgs in question have literally said so, because I think the people working there genuinely care about their impact and are competent enough to have heard of Goodhart’s law, and because in several cases there have been major strategy changes which cannot be explained by a model of “everyone working there has a massive blindspot and is focused on easy to meet targets”. As one concrete example, 80k’s focus has switched to be very explicitly longtermist, which it was not originally. They’ve also published several articles about areas of their thinking which were wrong or could have been improved, which again I would not expect for an organisation merely focused on gaming its own metrics.