Thanks for your comment James. I would define a TFO as any organisation whose explicit goal is to help people increase the impact they have through their careers. So yes, both meta EA groups and coaching and training organisations are included. I’ve now clarified this in the post too.
While I agree the theories of change between different interventions and organisations likely differ substantially, I think a set of standardised outcome-related indicators is still both relevant and necessary. Just as different organisation’s interventions in global health differ significantly, but their effects still can be estimated in comparable units like QALYs.
With that said, some organisations (e.g., national EA groups) will have additional outcome indicators that aren’t directly tied to talent pipelines or career transitions, just as your M&E framework illustrates.
Yes, as you sugget I think the biggest gap is indicators that say something meaningful about an organisation’s impact, rather than just outputs like completion rates, while still being feasible to track continuously (it seems unrealistic for TFOs to regularly estimate their ultimate impact, e.g. QALYs).
These can be on different levels, including simpler ones like number of placements/transitions and their cost-effectiveness. Slightly more advanced ones would take counterfactual and attribution into account.
And more complex ones could assign a quantitive value of a transition. This could be something similar to DIPY or ICAP. If establishing a new standardised quantified indicator for career changes, I think there could be a lot of learning to understand what worked well and less well with these. (Not sure that AAC has stopped using ICAPs internally.)
One possible structure for a quantified indicator could include:
Placement tiers
Tier 1: Positions at high impact organisations, or an organisation with high influence in an EA prioritised cause areas
Tier 2: Positions at organisations working with an EA prioritised cause area, but not at a highly effective one
Tier 3: Positions where the person can build relevant career capital.
Role seniority levels, such as junior, senior, and leadership
Assign impact values for each combination of tier and seniority
Counterfactual and attribution adjustments
If cause area per placement is reported, funders could adjust their comparison between different orgs depending on their cause prioritisation.