How will we know if we are doing good better: The case for more and better monitoring and evaluation

TomBill8 Feb 2023 16:31 UTC
165 points
16 comments9 min readEA link

Written by Sophie Gulliver and Thomas Billington

TL;DR—EAs want to do the most good with the resources we have. But how do we know if we are actually doing the most good? How do we know if our projects are running efficiently and effectively? How do we know we are achieving impact? How do we check we are not causing harm? Monitoring and evaluation (M&E) theories and tools can help EA organisations answer these questions—but are currently not being applied to a sufficient degree. These theories and practices will help us achieve more impact and value in the short-term and long-term.

Here, we outline a few ways to build M&E knowledge and skills within EA, including:

Introduction

EA is “a project that aims to find the best ways to help others, and put them into practice”. Specifically, EA distinguishes itself from general do-gooding through its commitment to doing the most good possible with its resources.

To these ends, the EA community is only as good at achieving its goal of “finding the best way to help others” as it is at knowing what the best way to help others is. But how can we know what the best ways to help others are? And how will we know if what we are doing is actually helping others in a cost-effective and impactful way?

Monitoring and evaluation (M&E) experts, tools and concepts are some of the best ways to realise effective altruism’s philosophy of maximising impact and doing good better.

M&E are two interrelated concepts that help us track and assess a project or organisation’s progress, impact, and value. In essence, if you really care about whether a project is making the world better, then you should also care about M&E.

However, in our experience, the integration of M&E tools and expertise within the EA community has been variable and mostly restricted to the global health and development sector.

This post argues that the broader EA movement should also be engaging with M&E more to ensure we are doing the most good possible.

Note: This post aims at being a broad overview of M&E. Our objective is to increase knowledge and awareness of M&E in the EA space, as well as give some first steps to learn more. Don’t worry if you feel like you can’t apply it immediately. Future posts will delve into details, and see our section below for what you can do right now.

What is M&E?

Monitoring and evaluation are two distinct functions that work synergistically. They can be defined as:

Monitoring: the systematic and routine collection of information to track progress and identify areas for improvement. Monitoring asks questions like:

  • Are we on track?

  • Are we reaching the right people?

  • Are we using our money and time efficiently?

  • What can we improve?

For example, if you were running a project to reduce deaths through improved water quality, you might regularly monitor chlorine availability at water points, or run quarterly surveys asking community members how often they are chlorinating their water.

Evaluation: the rigorous assessment of the value of a project or programme to inform decision-making. This assessment usually measures the performance of the project against criteria that define what a ‘valuable’ project looks like. These could be criteria like ‘impactful’ or ‘cost-effective’ or ‘sustainable’. Unlike monitoring, evaluations answer bigger questions to inform important decisions about a project’s future:

  • How well is this project doing overall?

  • How valuable is it?

  • Is it worth it?

  • Should we adapt, scale up or scale down?

Given the deeper analysis required, evaluations are typically undertaken at important decision points within a project’s life.

For example, for our above water quality project our evaluation might assess cholera and mortality rates in our program area and compare these with similar non-treatment areas. This might help us decide if our program achieves our planned outcomes and ultimate impact.

We operationalise M&E throughout the entire life of a project—from the conception of an idea, through piloting and implementation, all the way to scaling up (or down). The diagram below gives a basic illustration of the different stages of M&E and what we aim to do at each stage.

The amount of M&E and how you might go about it changes from organisation to organisation depending on the size, budget, and resources. However, a basic M&E system often has the following elements:

A Theory of Change (ToC) which maps out the step-by-step process of how you think your project will address a problem and create change. The theory starts at your inputs (e.g. chlorine and chlorine dispensers) and leads through your outputs (e.g. amount of chlorine dispensed) to intermediate outcomes (e.g. reduced cases of cholera) all the way to your eventual impact (e.g. reduced mortality). This helps you understand how your project works and helps you think about what to track and evaluate.


A Monitoring and Evaluation Framework. Now you know the steps for how you think your project will achieve change, you need to write down how you will know if these changes are happening. What data can you gather, from whom, how often, and using what methods to track progress? This includes both quantitative and qualitative data.


Evaluation questions. What questions would you want answered to know if your project was good or worth it? For example: How cost-effective is my project? How well are we reaching those in most need? How impactful is our project in terms of lives saved? You can answer these questions through different quantitative, qualitative, and mixed evaluation methods.

In smaller organisations it can be hard to find the time, money, and expertise to do all this. If you are interested in doing this, our linked resources can be a great place to start! You can also contact us for additional pro bono advice and join the Slack M&E community to learn from others. For larger organisations it is beneficial to invest in M&E and hire M&E experts internally or externally to support you with this work.

What are some common struggles with M&E?

EA is a broad movement, and so there are lots of ways organisations are approaching M&E depending on their size, maturity, cause area, and other factors. Here are some common issues we have seen some organisations face that can hold them back from achieving a rigorous M&E system:

  • No explicit theory of how their program works. It is hard to monitor and evaluate our work when we don’t have a clear theory of change that shows how certain activities should lead to outputs, outcomes, and eventual impacts. This means we don’t know what to look for to see if our program is on its way to achieving impact.

  • A theory of change exists but lacks detail like the level of evidence/​ certainty, the risks, and the assumptions we have for each causal ‘linkage’. For example: How certain are we based on existing evidence that an activity like training leads to an outcome like behaviour change? What are we assuming? What could risk this relationship?. Without sufficient detail, M&E can’t help us focus on assessing the most important linkages and thus cannot determine if our program is actually working.

  • Not fully diagnosing the problem they are trying to solve. A needs assessment helps us systematically think through the root causes and effects of the problem we are trying to solve with our program and weigh the pros and cons of all possible solutions. This helps us design a robust project that has considered all available evidence; it also helps us identify alternative pathways we can take if our project is not working as intended.

  • Conducting only monitoring, often focused on quality assurance. That is, are we delivering what we said we would deliver?. While quality assurance monitoring is necessary and often the easiest place to start, it is important in time to move beyond looking at inputs and outputs and start monitoring early outcomes and unintended consequences. It is also important not to miss the forest for the trees— a phenomenon that occurs when you are so preoccupied with monitoring specific indicators that you aren’t asking or answering bigger evaluation questions like, how valuable is this project to our communities? or is this project worth it?

  • No evaluation plan. Often, organisations want to jump straight to evaluation without having done the planning required. First, we need to know what criteria we are evaluating the program against. Within EA, we often consider impact and cost-effectiveness as valuable criteria. However, people within our teams or the communities we work with might have different ideas about what makes a program valuable (e.g. sustainability, equity, cultural acceptability). M&E can surface important, but sometimes difficult, conversations about what matters most, but we need to decide the criteria against which we will evaluate a program before we start evaluating. This is why it is important to frame some evaluation questions that indicate what is most important to answer to determine the value of your program. Even if you are only just starting, it is important to start thinking about the type of evaluative evidence you want so you can start to prepare. Many types of evaluation, like Randomised Control Trials or quasi-experimental designs, require early planning, so you have time to collect baseline data. It is also easier if your monitoring data harmonises with your evaluation questions!

  • No good examples of M&E in their field. Animal welfare and longtermist causes can struggle as there are not as many resources available in these areas on how to do M&E. This can make it harder to formulate indicators as there are no common standards to help think through different evaluation designs; for instance, those in longtermist causes can’t conduct an RCT or quasi-experimental design.

  • No clear M&E responsibilities. Smaller organisations especially struggle with allocating M&E responsibilities because they have lots to do and not enough people. These organisations also often have lots of “jacks of all trades” but not necessarily someone with expertise in M&E. Without clear M&E responsibilities and in-house expertise, it can be hard to develop a rigorous M&E system beyond basic monitoring.

  • No resources dedicated to M&E. M&E can be expensive given the required staff/​external time and tools. And every dollar that goes towards M&E is a dollar not going towards direct programming. M&E, like projects, must be cost-effective. It requires some investment, matched to the needs of the organisation.

Why should I care about doing M&E better?

Every programme that aims to change the world contains assumptions about how the world works and a theory (either explicit or implicit) about how these programmes will make change (a “theory of change”). As leaders/​supporters of these projects, it is incumbent on us to carefully monitor them and evaluate whether those assumptions and theories hold true.

M&E allows you to apply the philosophical concepts of EA to your project practically. It is beneficial for you as a practitioner, for your organisation, and for the EA movement as a whole.

For your project

Rigorous monitoring and evaluation can…

Help you make informed decisions
We all want to do the most good with the least resources. That means we need ways to rigorously assess if we are doing valuable and cost-effective work. M&E can provide robust evidence to help us decide whether a project should continue, adapt, scale up or scale down.

Keep you accountable
It is important to be accountable to ourselves, our donors, our partners and the communities we serve and show we are doing what we committed to do, generating the results we intended and not causing harm. M&E can help us verify that a project is rolling out as planned, that the activities are reaching the intended participants, that funding is being spent appropriately and that there are no adverse consequences.

Improve your project design
If You Fail to Plan, You Are Planning to Fail. M&E practices like needs assessment/​problem diagnosis and theory of change help you to think through the problem you intend to solve and the theory for how you think you will achieve change. Bringing your theory of change into the light and inspecting each aspect of it thoroughly helps you ensure your proposed project actually addresses the problem on the ground and is evidence based.

Identify risks and areas for improvement so you can learn and adapt
Even with a robust theory of change, things can change when you start implementing in the messy real world. By systematically monitoring your project’s progress and evaluating your performance, it’s possible to see where a project is or could be going wrong and proactively address this. Learning and adapting your project as you go makes it more robust and improves its quality. This can also lead to important innovations.

Motivate you, your partners and your project participants
You, your team and the people you work with need motivation to keep going. Otherwise, especially with complex problems, it can feel like nothing is changing. A powerful way to keep people motivated is to share M&E findings with them to show if and where change is happening.

Ultimately, these things make your project more effective and impactful.

For the whole movement

Practise our values.
EA is built on the foundation of doing good better. To truly live this value, we need to invest in monitoring and evaluating systems across the EA ecosystem to know if we are indeed doing good better.

Aggregate EA organisations’ impact.
The more monitoring data and rigorous evaluations we conduct, the better placed we are to conduct systematic reviews and evaluations of the impact of the EA movement (and sub-communities) as a whole. This is important evidence to prove the value of the EA movement to members and to the world.

Build our knowledge of what works.
If every organisation in the EA movement undertook regular monitoring and evaluation and shared the key learnings from these efforts, we could help other EA, and non-EA organisations avoid similar mistakes and learn from innovations, resulting in better programming for those we most want to benefit.

Help decision-makers make better allocative decisions for funding within the EA movement so we are doing the most good possible with the resources available.

What can I do right now to improve M&E within EA?

M&E is fundamental to achieving the goals of EA to ensure we do the most good possible. To improve M&E within EA, we need to build M&E knowledge and skills within the movement and build linkages and partnerships with M&E organisations and experts. To start this, we propose the following actions:

Conclusion

EAs want to do the most good possible with the resources we have. If we want to live this value, then we need ways to assess that we are actually doing good.

M&E theories and tools can help EA organisations, and the movement as a whole, monitor and evaluate what we are doing so we can create better designed projects, adapt and improve our work, hold ourselves accountable for the commitments we make and, ultimately, make better decisions about how to do the most good.

Currently, we are not using M&E skills and knowledge in EA as much as we could. We hope that this post helps to get more EA projects interested in M&E (see our pro bono support) and begin to build a community for those interested in M&E within EA (see our Slack group).

This post will be the first in a series around M&E in the EA space. If you have topics you would like us to cover in future posts, please leave a comment below.

TomBill8 Feb 2023 16:31 UTC
165 points
16 comments9 min readEA link