Could you define TFOs? Based on your backgrounds, I’m guessing you mean community building organisations like EA Sweden, EA Netherlands, etc., and coaching/training/advising organisations like Successif, 80k, Talos, AIM, Tarbell, etc.?
While both of these sets of organisations are ultimately about helping talent make a difference, I think they have quite different theories of change, and therefore require different M&E systems.
See my proposal below for how I think community building organisations should do things.
An M&E Framework for EA Community Building Based on Community Capital
The proximate objective: Community capital
EA community building organisations are ultimately aiming for impact. But measuring final impact directly is nearly impossible for community builders. How do you attribute a career transition to AI safety, or a crucial research insight, or a new organisation being founded, to your intro fellowship or community event?
Instead, the proximate objective should be increasing community capital, defined as:
Community Capital = (Sum of individual career capital) × (Coordination ability)
This formula captures something important about how EA communities actually create value. Career capital—the skills, knowledge, credentials, and connections that enable someone to have impact—matters enormously. But a collection of capable individuals who don’t coordinate is far less valuable than a community that can pool knowledge, collaborate on projects, and leverage each other’s expertise. The multiplication relationship reflects that coordination acts as a multiplier: high coordination ability means individual career capital gets leveraged far more effectively.
For an EA national group like EA Netherlands, this means success looks like: growing the number of people with relevant career capital, increasing the average career capital per person, and strengthening the community’s ability to coordinate effectively. Do this well, and impact should follow (via EA’s broader theory that career capital directed at priority problems matters).
Measuring community capital: The annual survey approach
I think this can be best measured with an annual community survey that collects data on both components simultaneously.
Individual career capital can be measured through self-assessment questions:
Specific questions about skills, credentials, etc.
“How capable do you feel of doing high-impact work in your priority cause area?” (1-7 scale)
“How much have your skills/knowledge relevant to impact grown in the past year?”
“Are you on a career path you consider high-impact?”
Coordination ability is best measured through network questions borrowed from social capital research:
“List up to 10 EA NL members you’ve had meaningful interaction with in the past year”
“Of the people you listed, how many could you collaborate with on a project?”
“How many EA NL members do you trust to give you good advice on your work?”
These network questions serve multiple purposes. First, they give you objective data about who’s connected to whom, rather than just subjective feelings about connectedness. You can map the actual network structure, identify clusters, and measure density. Second, they differentiate between mere acquaintance and genuine collaboration-readiness—knowing someone versus being able to work with them effectively.
Estimating community size
Apparently, we could then estimate total community size using capture-recapture methods. If survey respondents collectively name 150 unique people, but only 60 of those actually took the survey, the overlap pattern tells you what proportion of the community you’re reaching. This lets you estimate:
Total active community size (N)
The engaged core (people who both responded and got named multiple times)
The periphery (people named but who didn’t respond)
Combined with career capital measures, you now have all three components of the formula:
Sum of career capital ≈ N × average career capital from survey
Coordination ability ≈ function of network density, trust levels, collaboration-readiness
Community capital ≈ the product of these
Programme-level indicators
The annual survey tells you whether you’re winning overall, but you need more frequent feedback on whether specific programmes are working. Programme-level indicators provide this:
These don’t measure community capital directly, but they’re leading indicators that tell you if you’re on track between annual surveys.
Attributing changes to your programmes
Measuring community capital is one thing; showing that your programmes actually contribute to it is another. Three complementary approaches:
1. Cohort tracking: Survey fellowship participants before and after the programme. Survey attendees of major events like EAGx before and after. Track how their career capital and network connections change. This gives you programme-specific deltas, though you can’t fully prove causation without control groups.
2. Attribution questions in the annual survey: Simply ask people which EAN programmes most increased their career capital or helped them build connections. This relies on self-reported attribution, which isn’t perfect, but people generally have decent intuitions about what helped them.
3. Qualitative contribution analysis: Interview a sample of community members annually and ask them to tell the story of how they became more connected or capable. Code their responses for whether EAN programmes feature in their causal narratives. This captures unexpected pathways and avoids leading them toward giving you credit. We’re experimenting with the QUIP methodology at the moment.
Realistically, you’d use all three: cohort tracking for major programmes, attribution questions in the annual survey, and some qualitative interviews.
Connecting community capital to impact
This is the hardest link in the chain. You can measure community capital and show your programmes contribute to it, but does community capital actually produce impact?
The honest answer: you can’t measure final impact (lives saved, existential risk reduced) directly. You’re relying on a theory of change with two key assumptions:
EA’s general theory that career capital directed at priority problems leads to impact
Your theory that coordination multiplies individual effectiveness
What you can do is track intermediate outcomes that validate this theory:
Career transitions
Collaborative projects launched
Grants secured from EA funders
Research/writing produced
Organisations started
Then correlate these with community capital levels: do people with higher career capital and better networks achieve these outcomes more frequently? Do collaborative projects require the coordination infrastructure you’ve built?
What this system gives you
This M&E approach offers several advantages:
Practical: One annual survey gives you the core metrics, supplemented by programme-level data you’re probably already collecting.
Actionable: The formula highlights where to invest. If career capital is low but coordination is high, focus on upskilling and recruitment. If career capital is high but coordination is low, invest in events and infrastructure.
Honest about limitations: It doesn’t pretend you can measure final impact. Instead, it measures the proximate objective you actually control, while acknowledging the remaining uncertainty.
Theory-driven: It’s based on an explicit model of how communities create value, not just a collection of metrics. This makes it easier to explain to funders and board members why you’re measuring what you’re measuring.
Thanks for your comment James. I would define a TFO as any organisation whose explicit goal is to help people increase the impact they have through their careers. So yes, both meta EA groups and coaching and training organisations are included. I’ve now clarified this in the post too.
While I agree the theories of change between different interventions and organisations likely differ substantially, I think a set of standardised outcome-related indicators is still both relevant and necessary. Just as different organisation’s interventions in global health differ significantly, but their effects still can be estimated in comparable units like QALYs.
With that said, some organisations (e.g., national EA groups) will have additional outcome indicators that aren’t directly tied to talent pipelines or career transitions, just as your M&E framework illustrates.
But at what level should that standardised set of outcome-related indicators operate?
As you mention, we already have indicators for ultimate impact (QALYs, etc). And the indicators at the opposite end of the spectrum are pretty simple (completion rates, NPS, etc.).
It feels like you’re looking for indicators that occupy the space in between? Something like 80k’s old DIPY metric or AAC’s ICAP?
I thiiiiink both organisations tried these metrics and then discontinued them because they weren’t so useful?
Yes, as you sugget I think the biggest gap is indicators that say something meaningful about an organisation’s impact, rather than just outputs like completion rates, while still being feasible to track continuously (it seems unrealistic for TFOs to regularly estimate their ultimate impact, e.g. QALYs).
These can be on different levels, including simpler ones like number of placements/transitions and their cost-effectiveness. Slightly more advanced ones would take counterfactual and attribution into account.
And more complex ones could assign a quantitive value of a transition. This could be something similar to DIPY or ICAP. If establishing a new standardised quantified indicator for career changes, I think there could be a lot of learning to understand what worked well and less well with these. (Not sure that AAC has stopped using ICAPs internally.)
One possible structure for a quantified indicator could include:
Placement tiers
Tier 1: Positions at high impact organisations, or an organisation with high influence in an EA prioritised cause areas
Tier 2: Positions at organisations working with an EA prioritised cause area, but not at a highly effective one
Tier 3: Positions where the person can build relevant career capital.
Role seniority levels, such as junior, senior, and leadership
Assign impact values for each combination of tier and seniority
Counterfactual and attribution adjustments
If cause area per placement is reported, funders could adjust their comparison between different orgs depending on their cause prioritisation.
Thanks for doing this!
Could you define TFOs? Based on your backgrounds, I’m guessing you mean community building organisations like EA Sweden, EA Netherlands, etc., and coaching/training/advising organisations like Successif, 80k, Talos, AIM, Tarbell, etc.?
While both of these sets of organisations are ultimately about helping talent make a difference, I think they have quite different theories of change, and therefore require different M&E systems.
See my proposal below for how I think community building organisations should do things.
An M&E Framework for EA Community Building Based on Community Capital
The proximate objective: Community capital
EA community building organisations are ultimately aiming for impact. But measuring final impact directly is nearly impossible for community builders. How do you attribute a career transition to AI safety, or a crucial research insight, or a new organisation being founded, to your intro fellowship or community event?
Instead, the proximate objective should be increasing community capital, defined as:
Community Capital = (Sum of individual career capital) × (Coordination ability)
This formula captures something important about how EA communities actually create value. Career capital—the skills, knowledge, credentials, and connections that enable someone to have impact—matters enormously. But a collection of capable individuals who don’t coordinate is far less valuable than a community that can pool knowledge, collaborate on projects, and leverage each other’s expertise. The multiplication relationship reflects that coordination acts as a multiplier: high coordination ability means individual career capital gets leveraged far more effectively.
For an EA national group like EA Netherlands, this means success looks like: growing the number of people with relevant career capital, increasing the average career capital per person, and strengthening the community’s ability to coordinate effectively. Do this well, and impact should follow (via EA’s broader theory that career capital directed at priority problems matters).
Measuring community capital: The annual survey approach
I think this can be best measured with an annual community survey that collects data on both components simultaneously.
Individual career capital can be measured through self-assessment questions:
Specific questions about skills, credentials, etc.
“How capable do you feel of doing high-impact work in your priority cause area?” (1-7 scale)
“How much have your skills/knowledge relevant to impact grown in the past year?”
“Are you on a career path you consider high-impact?”
Coordination ability is best measured through network questions borrowed from social capital research:
“List up to 10 EA NL members you’ve had meaningful interaction with in the past year”
“Of the people you listed, how many could you collaborate with on a project?”
“How many EA NL members do you trust to give you good advice on your work?”
These network questions serve multiple purposes. First, they give you objective data about who’s connected to whom, rather than just subjective feelings about connectedness. You can map the actual network structure, identify clusters, and measure density. Second, they differentiate between mere acquaintance and genuine collaboration-readiness—knowing someone versus being able to work with them effectively.
Estimating community size
Apparently, we could then estimate total community size using capture-recapture methods. If survey respondents collectively name 150 unique people, but only 60 of those actually took the survey, the overlap pattern tells you what proportion of the community you’re reaching. This lets you estimate:
Total active community size (N)
The engaged core (people who both responded and got named multiple times)
The periphery (people named but who didn’t respond)
Combined with career capital measures, you now have all three components of the formula:
Sum of career capital ≈ N × average career capital from survey
Coordination ability ≈ function of network density, trust levels, collaboration-readiness
Community capital ≈ the product of these
Programme-level indicators
The annual survey tells you whether you’re winning overall, but you need more frequent feedback on whether specific programmes are working. Programme-level indicators provide this:
Fellowships: completion rates, participant satisfaction, post-programme surveys
Events: attendance, quality ratings, new connections formed
Organiser support: organiser activity levels, events run
Digital infrastructure: Chat engagement, information sharing
Marcom: brand awareness/recall/sentiment, conversion rates, etc
These don’t measure community capital directly, but they’re leading indicators that tell you if you’re on track between annual surveys.
Attributing changes to your programmes
Measuring community capital is one thing; showing that your programmes actually contribute to it is another. Three complementary approaches:
1. Cohort tracking: Survey fellowship participants before and after the programme. Survey attendees of major events like EAGx before and after. Track how their career capital and network connections change. This gives you programme-specific deltas, though you can’t fully prove causation without control groups.
2. Attribution questions in the annual survey: Simply ask people which EAN programmes most increased their career capital or helped them build connections. This relies on self-reported attribution, which isn’t perfect, but people generally have decent intuitions about what helped them.
3. Qualitative contribution analysis: Interview a sample of community members annually and ask them to tell the story of how they became more connected or capable. Code their responses for whether EAN programmes feature in their causal narratives. This captures unexpected pathways and avoids leading them toward giving you credit. We’re experimenting with the QUIP methodology at the moment.
Realistically, you’d use all three: cohort tracking for major programmes, attribution questions in the annual survey, and some qualitative interviews.
Connecting community capital to impact
This is the hardest link in the chain. You can measure community capital and show your programmes contribute to it, but does community capital actually produce impact?
The honest answer: you can’t measure final impact (lives saved, existential risk reduced) directly. You’re relying on a theory of change with two key assumptions:
EA’s general theory that career capital directed at priority problems leads to impact
Your theory that coordination multiplies individual effectiveness
What you can do is track intermediate outcomes that validate this theory:
Career transitions
Collaborative projects launched
Grants secured from EA funders
Research/writing produced
Organisations started
Then correlate these with community capital levels: do people with higher career capital and better networks achieve these outcomes more frequently? Do collaborative projects require the coordination infrastructure you’ve built?
What this system gives you
This M&E approach offers several advantages:
Practical: One annual survey gives you the core metrics, supplemented by programme-level data you’re probably already collecting.
Actionable: The formula highlights where to invest. If career capital is low but coordination is high, focus on upskilling and recruitment. If career capital is high but coordination is low, invest in events and infrastructure.
Honest about limitations: It doesn’t pretend you can measure final impact. Instead, it measures the proximate objective you actually control, while acknowledging the remaining uncertainty.
Theory-driven: It’s based on an explicit model of how communities create value, not just a collection of metrics. This makes it easier to explain to funders and board members why you’re measuring what you’re measuring.
Hot take: right now I think most regions have high coordination but low career capital but unfortunately are spending waaaaaay more on coordination
Thanks for your comment James. I would define a TFO as any organisation whose explicit goal is to help people increase the impact they have through their careers. So yes, both meta EA groups and coaching and training organisations are included. I’ve now clarified this in the post too.
While I agree the theories of change between different interventions and organisations likely differ substantially, I think a set of standardised outcome-related indicators is still both relevant and necessary. Just as different organisation’s interventions in global health differ significantly, but their effects still can be estimated in comparable units like QALYs.
With that said, some organisations (e.g., national EA groups) will have additional outcome indicators that aren’t directly tied to talent pipelines or career transitions, just as your M&E framework illustrates.
Thanks for clarifying!
But at what level should that standardised set of outcome-related indicators operate?
As you mention, we already have indicators for ultimate impact (QALYs, etc). And the indicators at the opposite end of the spectrum are pretty simple (completion rates, NPS, etc.).
It feels like you’re looking for indicators that occupy the space in between? Something like 80k’s old DIPY metric or AAC’s ICAP?
I thiiiiink both organisations tried these metrics and then discontinued them because they weren’t so useful?
Yes, as you sugget I think the biggest gap is indicators that say something meaningful about an organisation’s impact, rather than just outputs like completion rates, while still being feasible to track continuously (it seems unrealistic for TFOs to regularly estimate their ultimate impact, e.g. QALYs).
These can be on different levels, including simpler ones like number of placements/transitions and their cost-effectiveness. Slightly more advanced ones would take counterfactual and attribution into account.
And more complex ones could assign a quantitive value of a transition. This could be something similar to DIPY or ICAP. If establishing a new standardised quantified indicator for career changes, I think there could be a lot of learning to understand what worked well and less well with these. (Not sure that AAC has stopped using ICAPs internally.)
One possible structure for a quantified indicator could include:
Placement tiers
Tier 1: Positions at high impact organisations, or an organisation with high influence in an EA prioritised cause areas
Tier 2: Positions at organisations working with an EA prioritised cause area, but not at a highly effective one
Tier 3: Positions where the person can build relevant career capital.
Role seniority levels, such as junior, senior, and leadership
Assign impact values for each combination of tier and seniority
Counterfactual and attribution adjustments
If cause area per placement is reported, funders could adjust their comparison between different orgs depending on their cause prioritisation.