Yes, as you sugget I think the biggest gap is indicators that say something meaningful about an organisation’s impact, rather than just outputs like completion rates, while still being feasible to track continuously (it seems unrealistic for TFOs to regularly estimate their ultimate impact, e.g. QALYs).
These can be on different levels, including simpler ones like number of placements/transitions and their cost-effectiveness. Slightly more advanced ones would take counterfactual and attribution into account.
And more complex ones could assign a quantitive value of a transition. This could be something similar to DIPY or ICAP. If establishing a new standardised quantified indicator for career changes, I think there could be a lot of learning to understand what worked well and less well with these. (Not sure that AAC has stopped using ICAPs internally.)
One possible structure for a quantified indicator could include:
Placement tiers
Tier 1: Positions at high impact organisations, or an organisation with high influence in an EA prioritised cause areas
Tier 2: Positions at organisations working with an EA prioritised cause area, but not at a highly effective one
Tier 3: Positions where the person can build relevant career capital.
Role seniority levels, such as junior, senior, and leadership
Assign impact values for each combination of tier and seniority
Counterfactual and attribution adjustments
If cause area per placement is reported, funders could adjust their comparison between different orgs depending on their cause prioritisation.
Yes, as you sugget I think the biggest gap is indicators that say something meaningful about an organisation’s impact, rather than just outputs like completion rates, while still being feasible to track continuously (it seems unrealistic for TFOs to regularly estimate their ultimate impact, e.g. QALYs).
These can be on different levels, including simpler ones like number of placements/transitions and their cost-effectiveness. Slightly more advanced ones would take counterfactual and attribution into account.
And more complex ones could assign a quantitive value of a transition. This could be something similar to DIPY or ICAP. If establishing a new standardised quantified indicator for career changes, I think there could be a lot of learning to understand what worked well and less well with these. (Not sure that AAC has stopped using ICAPs internally.)
One possible structure for a quantified indicator could include:
Placement tiers
Tier 1: Positions at high impact organisations, or an organisation with high influence in an EA prioritised cause areas
Tier 2: Positions at organisations working with an EA prioritised cause area, but not at a highly effective one
Tier 3: Positions where the person can build relevant career capital.
Role seniority levels, such as junior, senior, and leadership
Assign impact values for each combination of tier and seniority
Counterfactual and attribution adjustments
If cause area per placement is reported, funders could adjust their comparison between different orgs depending on their cause prioritisation.