Probably Good is excited to share a new path profile for careers in monitoring and evaluation. Below, we’ve included a few excerpts from the full profile.
Monitoring and evaluation (M&E) specialists collect, track, and analyze data to assess the value and impact of different programs and interventions, as well as translate these assessments into actionable insights and strategies to increase the impact of an organization.
M&E specialist careers might be a promising option for some people. If you’re an exceptional fit, especially if you’re based in a low- or middle-income country where there’s lots of scope for implementing global health and development interventions, then it may be worth considering these careers.
However, the impact you’ll be able to have will be determined in large part by the organization you enter – making it particularly important to seek out the best organizations and avoid those that only superficially care about evaluating their impact. Additionally, if you’re a good fit for some of the top roles in this path, it’s likely you’ll also be a good fit for other highly impactful roles, so we’d recommend you consider other paths, too.
[...]
How promising is this path?
Monitoring and evaluation is important for any organization aiming to have an impact. Without collecting evidence and data, it’s easy to seem like an intervention or program is having an impact, even when it’s not. Here are a few ways in which M&E might be able to generate impact:
Discover effective interventions that do a lot of good. For example, rigorous evaluation by J-PAL affiliates and Evidence Action found that placing chlorine-treated water dispensers in rural African villages reduced under-5 child mortality by as much as 63%. Evidence Action has now pledged to double the size of its water-treatment program, reaching 9 million people.
Make improvements to known effective interventions. Improving the efficacy of an already-impactful intervention by even a little bit can generate a large impact, especially if the intervention is rolled out on a large scale. Consider this study run by malaria charity TAMTAM, which found that charging even a nominal price for malaria bednets decreased demand by up to 60%, leading a number of large organizations to offer them for free instead.
Identify ineffective or harmful interventions, so that an organization can change course. A great example of this is animal advocacy organization the Humane League, which determined that their current strategy of performing controversial public stunts was ineffective, and pivoted its strategy towards corporate campaigns. In doing so, they convinced Unilever to stop killing male chicks, saving millions of baby chicks from gruesome deaths.
Advantages
Clear links to effectiveness—Because M&E is explicitly concerned with measuring the impact of interventions, there’s often a clear “theory of change” for how your work might translate into positive impact.
Leverage—If you’re working in a large organization, or working on an intervention with a large pool of potential funders and implementers, your work can influence where large amounts of money is spent, or how large amounts of other resources are distributed.
Flexible skill set—the skills and qualifications you’ll need for a career in M&E are robustly useful across a range of careers. As such, it’s likely that M&E work will provide you with flexible career capital for pursuing other paths.
Varying sensitivity to evidence—The amount of influence M&E work can have is likely to vary significantly across organizations. This is in large part because it can conflict with the incentives and personalities of other people and departments within the organization. For example, some organizations may be unwilling to pull the plug on a project if it is found to be ineffective, perhaps because doing so would create bad PR, damage fundraising efforts, or because it’s a “pet project” of someone influential within the organization. These dynamics can be personally frustrating and curtail your impact. Because of this, any career in M&E should prioritize joining organizations that really care about their impact – we’ll talk more about this in the priorities section later on.
[...]
Priorities within M&E specialist roles
Where can you work as an M&E specialist? And where might you have the most impact? Very broadly, there are three categories of organizations that hire M&E staff: implementing organizations, dedicated M&E organizations and consultancies, or multilateral institutions, governments, and foundations.
Work in-house at an implementing organizations that particularly values M&E
An “implementing” organization is one that deploys interventions and programs itself. Implementing organizations often have their own internal M&E staff to track the rollout and efficacy of these programs. However, large project evaluations are likely to be outsourced to external organizations (unless it’s a particularly large nonprofit), meaning that internal M&E officers may find themselves focusing more on monitoring activities as well as conducting smaller (sometimes qualitative) evaluations. Internal M&E specialists may also find themselves developing theories of change and communicating these to the rest of the organization, as well as other strategic work – though this will likely depend on the role’s seniority.
An important point to consider here is that there is significant variance in the extent to which implementing nonprofits really value M&E processes. For some, it’s just another box to tick or a way to generate numbers that can be included in marketing materials. In these organizations, you could find that your work is rarely used to inform or actually improve the organization’s actions, substantially reducing your potential impact.
Because of this, if you pursue an M&E role within an implementing organization, it’s really important to look out for organizations that place an unusual amount of value on gathering data and responding to evidence. But how can you identify these organizations? One place to start could be to look at whether an organization has collaborated with an external evaluating partner like J-PAL or IDInsight – this is some evidence (albeit nonperfect) that a nonprofit really cares about the efficacy of their work.
Another promising option is to research which organizations receive funding from foundations and evidence-based funders like the Gates Foundation, the Global Innovation Fund, the Gates Foundation, Instiglio, or GiveWell. Again, this is no guarantee that such organizations are high-impact, but it gives reason to think they have an above-average concern for M&E processes.
A further possibility is to look at resources on the organization’s website, like annual reports and evaluation reports. Some organizations may oversell the impact they’ve had in order to please funders, which can indicate that they don’t take evidence seriously in their decision making. Instead, look to see if they make reasonable claims about the size of their impact, if they’re transparent about their numbers and data collection methods, if they’ve documented times when they’ve changed their activities in response to new evidence, and whether they own up to any mistakes they’ve made – these can be promising indicators.
As a final note on working in implementing organizations, it may be a particularly promising option to join a new, forward-thinking implementing organization that takes evidence seriously. Experts have shared that such early stage organizations can find it difficult to hire for these positions, compared to larger organizations who tend to draw more candidates, making it more likely you’ll be able to have impact at the margin.
We don’t have a clear or confident view on whether you’ll generally have more impact at such organizations than as an M&E specialist in a larger implementing nonprofit, but we imagine the tradeoff will often work like this: at a smaller organization, you might be able to bring larger marginal improvements to the organization, as you could be the only M&E specialist (or one of very few), but will naturally influence fewer resources and smaller-scale interventions than you might at a larger organization. At a large organization, the reverse is true – you might have less scope on average for large marginal improvements, but the total amount of resources spent by the organization could mean this still yields substantial impact.
Working at top dedicated M&E consultancies
There are a number of organizations that exist to help clients like nonprofits and governments to assess the efficacy of their programs. These consultancies can be broadly divided into two categories: academia-driven & client-driven. Such organizations are generally responsible for running evaluations for partners and clients, rather than focusing on day-to-day monitoring work. Because of this, roles in these organizations are likely to demand a higher level of statistical and quantitative literacy – perhaps particularly so in academia-driven organizations.
Academia-driven
Academia-driven organizations typically contain a mixture of in-house research staff as well as affiliated academic researchers (often economists) based in universities across the world. These organizations will partner with a number of implementing organizations to gather data and conduct evaluations (such as RCTs) to test the efficacy of different interventions or programs. Academia-driven organizations tend to focus on quite broad questions about interventions, like assessing the effectiveness of cash transfers, or attempting to discover generalisable insights about the efficacy of different interventions which might be implemented by a range of actors. The results of these are also often published as papers in conventional academic journals.
Prestigious organizations in this space include J-PAL, Innovations for Poverty Action (IPA), and the Center for Global Development (CGD). It’s worth noting that, because of their ties to academia, some of these institutions might be considered to be development economics research institutions rather than M&E organizations per se – and that, therefore, careers in these institutions may overlap significantly with careers in development economics.
Client-driven
Other organizations work with clients like governments and NGOs to deliver useful information (particularly evaluations) in a way that’s more tightly connected to specific organizational needs, generally on shorter timescales, and also on a smaller scale, than academic-driven consultancies. Such organizations might evaluate specific programs according to criteria produced in collaboration with the client, rather than attempting to generate highly generalizable insights about the impact of different interventions. Because such organizations are driven by the needs of their clients, rather than broader questions of academic interest, their work is highly likely to be practically applicable. A few well-respected organizations in this space include IDinsight, Mathematica, and Oxford Policy Management.
Which organization type might be more promising to work in: academia-driven or client-driven organizations? It’s hard to give a confident general conclusion, but it’s worth noting a few relevant considerations on either side. For example, with client-driven M&E work, you may be more confident your work is practically useful as you’re responding directly to organizational needs. However, the client-directed nature of this work may also constrain impact – you may find yourself helping clients who are working on problems you don’t think are the most important or pressing.
On the academic side, it’s a slightly different story. One potential problem is that academia-driven organizations may be constrained by academic incentives to produce work that is intellectually interesting and cutting edge, but might not always be as aligned with real-world needs as client-driven work. However, when academia-driven work does have an impact, it may do so on a larger scale. For instance, a large evaluation that identifies a promising new intervention, or a better way to implement an existing intervention, could pull a number of impact-inclined organizations toward changing their activities.
Multilateral institutions, governments, and foundations
Large multilateral institutions, like the UN or the World Bank, are prolific employers of M&E specialists, as they integrate M&E processes fairly ubiquitously across branches and departments. Some M&E specialists within these organizations will also be responsible for setting high-level goals and strategies for the entire organization, such as developing systems to track and evaluate whether they are meeting the Sustainable Development Goals. There are a large number of opportunities at these institutions, and as such, they are prominent employers within the M&E field.
We’ve heard that incentives within these organizations can sometimes be misaligned with real impact, meaning that M&E activities may sometimes focus on how to make individuals or branches look effective, rather than performing genuinely truth-seeking work (at least in some cases). Because of this, some caution may be warranted when considering careers in these institutions. Though you may be able to help improve these institutions from the inside, (and there are likely well-functioning, high-impact pockets within these organizations), this may be quite difficult.
Additionally, these institutions tend to have good talent pipelines in place and are able to hire many talented young people into junior roles. The competition here means it could be less likely you’ll be able to have much counterfactual impact within these roles, though they may provide strong opportunities for building career capital. However, barring a particularly promising opportunity, they may not be the best opportunities for direct impact.
On the governmental side, we’ve heard that the UK’s Foreign, Commonwealth & Development Office is particularly invested in evidence-based decision making, and specifically the Development Innovation Ventures arm of USAID. Many LMIC governments will also offer M&E roles for domestic programs and interventions, though we’re uncertain about how promising these opportunities tend to be, or which governments might be particularly promising to work in (though our profile on civil service careers within LMICs may provide other helpful information).
Relatedly, large foundations (like the Gates Foundation) may also house employees who support grantees and partners in their M&E processes. Similarly, both Instiglio and the Global Innovation Fund are two organizations that offer funding to start and scale-up programs contingent on strong evidence of effectiveness. These may be promising organizations to work at, though the roles here might not look like “typical” M&E careers.
You can read the full profile over at Probably Good.
Do you have EA or career related content that you’d like to see from Probably Good? We’re eager to hear what types of content would be useful to the broader community and/or to specific groups. Let us know by leaving a comment below or emailing contact@probablygood.org.
Monitoring & Evaluation Specialists – a new career path profile from Probably Good
Link post
Probably Good is excited to share a new path profile for careers in monitoring and evaluation. Below, we’ve included a few excerpts from the full profile.
Monitoring and evaluation (M&E) specialists collect, track, and analyze data to assess the value and impact of different programs and interventions, as well as translate these assessments into actionable insights and strategies to increase the impact of an organization.
M&E specialist careers might be a promising option for some people. If you’re an exceptional fit, especially if you’re based in a low- or middle-income country where there’s lots of scope for implementing global health and development interventions, then it may be worth considering these careers.
However, the impact you’ll be able to have will be determined in large part by the organization you enter – making it particularly important to seek out the best organizations and avoid those that only superficially care about evaluating their impact. Additionally, if you’re a good fit for some of the top roles in this path, it’s likely you’ll also be a good fit for other highly impactful roles, so we’d recommend you consider other paths, too.
[...]
How promising is this path?
Monitoring and evaluation is important for any organization aiming to have an impact. Without collecting evidence and data, it’s easy to seem like an intervention or program is having an impact, even when it’s not. Here are a few ways in which M&E might be able to generate impact:
Discover effective interventions that do a lot of good. For example, rigorous evaluation by J-PAL affiliates and Evidence Action found that placing chlorine-treated water dispensers in rural African villages reduced under-5 child mortality by as much as 63%. Evidence Action has now pledged to double the size of its water-treatment program, reaching 9 million people.
Make improvements to known effective interventions. Improving the efficacy of an already-impactful intervention by even a little bit can generate a large impact, especially if the intervention is rolled out on a large scale. Consider this study run by malaria charity TAMTAM, which found that charging even a nominal price for malaria bednets decreased demand by up to 60%, leading a number of large organizations to offer them for free instead.
Identify ineffective or harmful interventions, so that an organization can change course. A great example of this is animal advocacy organization the Humane League, which determined that their current strategy of performing controversial public stunts was ineffective, and pivoted its strategy towards corporate campaigns. In doing so, they convinced Unilever to stop killing male chicks, saving millions of baby chicks from gruesome deaths.
Advantages
Clear links to effectiveness—Because M&E is explicitly concerned with measuring the impact of interventions, there’s often a clear “theory of change” for how your work might translate into positive impact.
Leverage—If you’re working in a large organization, or working on an intervention with a large pool of potential funders and implementers, your work can influence where large amounts of money is spent, or how large amounts of other resources are distributed.
Flexible skill set—the skills and qualifications you’ll need for a career in M&E are robustly useful across a range of careers. As such, it’s likely that M&E work will provide you with flexible career capital for pursuing other paths.
Disadvantages
Narrow range of cause areas—Within our top recommended cause areas, there are far more M&E roles within global health and development than the others. This means M&E may be a promising career path if you want to work on global health, but perhaps less so if you prioritize other cause areas, like animal welfare or global catastrophic risks.
Varying sensitivity to evidence—The amount of influence M&E work can have is likely to vary significantly across organizations. This is in large part because it can conflict with the incentives and personalities of other people and departments within the organization. For example, some organizations may be unwilling to pull the plug on a project if it is found to be ineffective, perhaps because doing so would create bad PR, damage fundraising efforts, or because it’s a “pet project” of someone influential within the organization. These dynamics can be personally frustrating and curtail your impact. Because of this, any career in M&E should prioritize joining organizations that really care about their impact – we’ll talk more about this in the priorities section later on.
[...]
Priorities within M&E specialist roles
Where can you work as an M&E specialist? And where might you have the most impact? Very broadly, there are three categories of organizations that hire M&E staff: implementing organizations, dedicated M&E organizations and consultancies, or multilateral institutions, governments, and foundations.
Work in-house at an implementing organizations that particularly values M&E
An “implementing” organization is one that deploys interventions and programs itself. Implementing organizations often have their own internal M&E staff to track the rollout and efficacy of these programs. However, large project evaluations are likely to be outsourced to external organizations (unless it’s a particularly large nonprofit), meaning that internal M&E officers may find themselves focusing more on monitoring activities as well as conducting smaller (sometimes qualitative) evaluations. Internal M&E specialists may also find themselves developing theories of change and communicating these to the rest of the organization, as well as other strategic work – though this will likely depend on the role’s seniority.
An important point to consider here is that there is significant variance in the extent to which implementing nonprofits really value M&E processes. For some, it’s just another box to tick or a way to generate numbers that can be included in marketing materials. In these organizations, you could find that your work is rarely used to inform or actually improve the organization’s actions, substantially reducing your potential impact.
Because of this, if you pursue an M&E role within an implementing organization, it’s really important to look out for organizations that place an unusual amount of value on gathering data and responding to evidence. But how can you identify these organizations? One place to start could be to look at whether an organization has collaborated with an external evaluating partner like J-PAL or IDInsight – this is some evidence (albeit nonperfect) that a nonprofit really cares about the efficacy of their work.
Another promising option is to research which organizations receive funding from foundations and evidence-based funders like the Gates Foundation, the Global Innovation Fund, the Gates Foundation, Instiglio, or GiveWell. Again, this is no guarantee that such organizations are high-impact, but it gives reason to think they have an above-average concern for M&E processes.
A further possibility is to look at resources on the organization’s website, like annual reports and evaluation reports. Some organizations may oversell the impact they’ve had in order to please funders, which can indicate that they don’t take evidence seriously in their decision making. Instead, look to see if they make reasonable claims about the size of their impact, if they’re transparent about their numbers and data collection methods, if they’ve documented times when they’ve changed their activities in response to new evidence, and whether they own up to any mistakes they’ve made – these can be promising indicators.
As a final note on working in implementing organizations, it may be a particularly promising option to join a new, forward-thinking implementing organization that takes evidence seriously. Experts have shared that such early stage organizations can find it difficult to hire for these positions, compared to larger organizations who tend to draw more candidates, making it more likely you’ll be able to have impact at the margin.
We don’t have a clear or confident view on whether you’ll generally have more impact at such organizations than as an M&E specialist in a larger implementing nonprofit, but we imagine the tradeoff will often work like this: at a smaller organization, you might be able to bring larger marginal improvements to the organization, as you could be the only M&E specialist (or one of very few), but will naturally influence fewer resources and smaller-scale interventions than you might at a larger organization. At a large organization, the reverse is true – you might have less scope on average for large marginal improvements, but the total amount of resources spent by the organization could mean this still yields substantial impact.
Working at top dedicated M&E consultancies
There are a number of organizations that exist to help clients like nonprofits and governments to assess the efficacy of their programs. These consultancies can be broadly divided into two categories: academia-driven & client-driven. Such organizations are generally responsible for running evaluations for partners and clients, rather than focusing on day-to-day monitoring work. Because of this, roles in these organizations are likely to demand a higher level of statistical and quantitative literacy – perhaps particularly so in academia-driven organizations.
Academia-driven
Academia-driven organizations typically contain a mixture of in-house research staff as well as affiliated academic researchers (often economists) based in universities across the world. These organizations will partner with a number of implementing organizations to gather data and conduct evaluations (such as RCTs) to test the efficacy of different interventions or programs. Academia-driven organizations tend to focus on quite broad questions about interventions, like assessing the effectiveness of cash transfers, or attempting to discover generalisable insights about the efficacy of different interventions which might be implemented by a range of actors. The results of these are also often published as papers in conventional academic journals.
Prestigious organizations in this space include J-PAL, Innovations for Poverty Action (IPA), and the Center for Global Development (CGD). It’s worth noting that, because of their ties to academia, some of these institutions might be considered to be development economics research institutions rather than M&E organizations per se – and that, therefore, careers in these institutions may overlap significantly with careers in development economics.
Client-driven
Other organizations work with clients like governments and NGOs to deliver useful information (particularly evaluations) in a way that’s more tightly connected to specific organizational needs, generally on shorter timescales, and also on a smaller scale, than academic-driven consultancies. Such organizations might evaluate specific programs according to criteria produced in collaboration with the client, rather than attempting to generate highly generalizable insights about the impact of different interventions. Because such organizations are driven by the needs of their clients, rather than broader questions of academic interest, their work is highly likely to be practically applicable. A few well-respected organizations in this space include IDinsight, Mathematica, and Oxford Policy Management.
Which organization type might be more promising to work in: academia-driven or client-driven organizations? It’s hard to give a confident general conclusion, but it’s worth noting a few relevant considerations on either side. For example, with client-driven M&E work, you may be more confident your work is practically useful as you’re responding directly to organizational needs. However, the client-directed nature of this work may also constrain impact – you may find yourself helping clients who are working on problems you don’t think are the most important or pressing.
On the academic side, it’s a slightly different story. One potential problem is that academia-driven organizations may be constrained by academic incentives to produce work that is intellectually interesting and cutting edge, but might not always be as aligned with real-world needs as client-driven work. However, when academia-driven work does have an impact, it may do so on a larger scale. For instance, a large evaluation that identifies a promising new intervention, or a better way to implement an existing intervention, could pull a number of impact-inclined organizations toward changing their activities.
Multilateral institutions, governments, and foundations
Large multilateral institutions, like the UN or the World Bank, are prolific employers of M&E specialists, as they integrate M&E processes fairly ubiquitously across branches and departments. Some M&E specialists within these organizations will also be responsible for setting high-level goals and strategies for the entire organization, such as developing systems to track and evaluate whether they are meeting the Sustainable Development Goals. There are a large number of opportunities at these institutions, and as such, they are prominent employers within the M&E field.
We’ve heard that incentives within these organizations can sometimes be misaligned with real impact, meaning that M&E activities may sometimes focus on how to make individuals or branches look effective, rather than performing genuinely truth-seeking work (at least in some cases). Because of this, some caution may be warranted when considering careers in these institutions. Though you may be able to help improve these institutions from the inside, (and there are likely well-functioning, high-impact pockets within these organizations), this may be quite difficult.
Additionally, these institutions tend to have good talent pipelines in place and are able to hire many talented young people into junior roles. The competition here means it could be less likely you’ll be able to have much counterfactual impact within these roles, though they may provide strong opportunities for building career capital. However, barring a particularly promising opportunity, they may not be the best opportunities for direct impact.
On the governmental side, we’ve heard that the UK’s Foreign, Commonwealth & Development Office is particularly invested in evidence-based decision making, and specifically the Development Innovation Ventures arm of USAID. Many LMIC governments will also offer M&E roles for domestic programs and interventions, though we’re uncertain about how promising these opportunities tend to be, or which governments might be particularly promising to work in (though our profile on civil service careers within LMICs may provide other helpful information).
Relatedly, large foundations (like the Gates Foundation) may also house employees who support grantees and partners in their M&E processes. Similarly, both Instiglio and the Global Innovation Fund are two organizations that offer funding to start and scale-up programs contingent on strong evidence of effectiveness. These may be promising organizations to work at, though the roles here might not look like “typical” M&E careers.
You can read the full profile over at Probably Good.
Do you have EA or career related content that you’d like to see from Probably Good? We’re eager to hear what types of content would be useful to the broader community and/or to specific groups. Let us know by leaving a comment below or emailing contact@probablygood.org.