But at what level should that standardised set of outcome-related indicators operate?
As you mention, we already have indicators for ultimate impact (QALYs, etc). And the indicators at the opposite end of the spectrum are pretty simple (completion rates, NPS, etc.).
It feels like you’re looking for indicators that occupy the space in between? Something like 80k’s old DIPY metric or AAC’s ICAP?
I thiiiiink both organisations tried these metrics and then discontinued them because they weren’t so useful?
Yes, as you sugget I think the biggest gap is indicators that say something meaningful about an organisation’s impact, rather than just outputs like completion rates, while still being feasible to track continuously (it seems unrealistic for TFOs to regularly estimate their ultimate impact, e.g. QALYs).
These can be on different levels, including simpler ones like number of placements/transitions and their cost-effectiveness. Slightly more advanced ones would take counterfactual and attribution into account.
And more complex ones could assign a quantitive value of a transition. This could be something similar to DIPY or ICAP. If establishing a new standardised quantified indicator for career changes, I think there could be a lot of learning to understand what worked well and less well with these. (Not sure that AAC has stopped using ICAPs internally.)
One possible structure for a quantified indicator could include:
Placement tiers
Tier 1: Positions at high impact organisations, or an organisation with high influence in an EA prioritised cause areas
Tier 2: Positions at organisations working with an EA prioritised cause area, but not at a highly effective one
Tier 3: Positions where the person can build relevant career capital.
Role seniority levels, such as junior, senior, and leadership
Assign impact values for each combination of tier and seniority
Counterfactual and attribution adjustments
If cause area per placement is reported, funders could adjust their comparison between different orgs depending on their cause prioritisation.
Thanks for clarifying!
But at what level should that standardised set of outcome-related indicators operate?
As you mention, we already have indicators for ultimate impact (QALYs, etc). And the indicators at the opposite end of the spectrum are pretty simple (completion rates, NPS, etc.).
It feels like you’re looking for indicators that occupy the space in between? Something like 80k’s old DIPY metric or AAC’s ICAP?
I thiiiiink both organisations tried these metrics and then discontinued them because they weren’t so useful?
Yes, as you sugget I think the biggest gap is indicators that say something meaningful about an organisation’s impact, rather than just outputs like completion rates, while still being feasible to track continuously (it seems unrealistic for TFOs to regularly estimate their ultimate impact, e.g. QALYs).
These can be on different levels, including simpler ones like number of placements/transitions and their cost-effectiveness. Slightly more advanced ones would take counterfactual and attribution into account.
And more complex ones could assign a quantitive value of a transition. This could be something similar to DIPY or ICAP. If establishing a new standardised quantified indicator for career changes, I think there could be a lot of learning to understand what worked well and less well with these. (Not sure that AAC has stopped using ICAPs internally.)
One possible structure for a quantified indicator could include:
Placement tiers
Tier 1: Positions at high impact organisations, or an organisation with high influence in an EA prioritised cause areas
Tier 2: Positions at organisations working with an EA prioritised cause area, but not at a highly effective one
Tier 3: Positions where the person can build relevant career capital.
Role seniority levels, such as junior, senior, and leadership
Assign impact values for each combination of tier and seniority
Counterfactual and attribution adjustments
If cause area per placement is reported, funders could adjust their comparison between different orgs depending on their cause prioritisation.