As you may have noticed, change is afoot in GWWC. In the past, we feel that our research has been lacking in a few ways, including: a) We haven’t made it clear, publicly, what our methodology has been for assessing charities; b) We haven’t made it clear, publicly, what role GWWC has, relative to other charity evaluators; c) The website has not been updated regularly, so there’s been a disconnect between our research—what we’ve known—and what’s been broadcast publicly; d) We have spent fewer man-hours on research than was desirable. To a large extent, this has been because, until very recently, GWWC was run on an entirely voluntary basis. Now that we have full-time staff, we hope to be able to rectify these issues over time.
As a first step, this blog post and the next will explain our methodology in a little more detail.
What metric?
The crucial question for any charity evaluator is: what metric to use? GWWC is solely focused on finding the most cost-effective charities: that is, the charities that have the highest ratio of good produced per dollar given to the charity. Along with GiveWell, we are the only charity evaluators that are solely concerned with cost-effectiveness so understood. (1)
Explanation of what ‘producing good’ consists of
Isn’t the idea of ‘good’ subjective?
The grain of truth in that question is that different people’s conceptions of the good vary. Some people think that scientific knowledge is of value in itself; other people think that scientific knowledge is only valuable insofar as it improves the lives of others.
However, the sum total of human wellbeing is a component of any plausible moral view, and, on most moral views, increasing human wellbeing is the main way to do good in the world. For these reasons, it’s this aspect that we focus on. If this is not a component of your moral view, then our research will not be of use to you (but then, if you aren’t concerned by increasing human wellbeing, it’s unlikely that giving to global poverty alleviation is your top priority). In future posts, we’ll explain how to measure increases in human wellbeing.
What does ‘producing’ good mean?
Good question. A mistake to avoid is to think of producing good as to mean causing good. To illustrate, consider two charities that one could give to. Charity A will save 10 lives with every $10 000 given to it; Charity B will save 5 lives with every $10 000 given to it. However, if you don’t give $10 000 to Charity A, it will fill up its room for more funding from other donors (who wouldn’t have given at all otherwise). Whereas if you don’t give the $10 000 to Charity B, it will not fill up its room for more funding from additional donors. In this case, you should donate to Charity B rather than Charity A: though you directly cause 10 lives to be saved by donating to Charity A, you don’t really do any additional good, because those lives would have been saved anyway. In contrast, you do make a difference (of 5 lives saved) by donating to Charity B.
Making this more precise: what producing good means is as follows. First, you consider two possibilities: (i) where you give $n to Charity X; (ii) where you burn the money. Now think: how good is the world in which (i) you give the money? And how good is the world (ii) in which you burn the money? The amount of good you produce by giving $n to charity X is the difference in the goodness of the world in which you do give, minus the goodness of the world in which you burn the money.(2)
Importantly, the above method means that all consequences are considered. So ‘good produced’ does not just mean “impact provable in a few years”; long-run consequences are just as relevant to cost-effectiveness as short-run consequences. Nor does ‘good produced’ mean ‘the effect on the direct beneficiaries of an action’; effects on third parties are just as relevant as effects on direct beneficiaries.
But surely you can never know what organisation does the most good per dollar?
That’s exactly right – we’ll never know what the best charity is. However, that doesn’t prevent charity recommendations from being possible. It just means that we need to think in terms of expected good. To see this concept in action, consider two donation opportunities. Donating $10 000 to charity A will certainly save one life. Donating $10 000 to charity B will probably (80% likelihood) save two lives; but has some chance (20%) of producing no benefit.
In general, to work out the expected value of an action X, you look at each possible outcome that could result from X; for each possible outcome, you multiply the probability of that outcome occurring by the goodness of that outcome, if it does occur; and you then take the sum of this product across all possible outcomes. So the expected value of donating $10 000 to charity A is 1 life x 100% = 1; whereas the expected value of donating $10 000 to charity B is 2 lives x 80% = 1.6.
The concept of expected value becomes especially important in situations of great uncertainty, such as charity evaluation. Further posts will discuss in more detail how to correctly apply this concept.
What evidence do you use?
In principle, we’re open to evidence from any source, and if certain sources prove to be particularly reliable or unreliable, we will alter which sources we rely on. In the past, we’ve used the latest peer-reviewed research from health and development economics, information from larger cost-effectiveness-focused academic projects such as the Disease Control Priorities in Developing Countries (2nd edition) and the Abdul Latif Jameel Poverty Action Lab, and on research from other charity evaluators, in particular from GiveWell. Like a growing number of organisations, we place particular weight on randomised controlled trials where they are available.
In the near future we will also begin publishing draft reports on topics and passing them around in order to to solicit additional information from experts and the public.
Sometimes, we have to rely more on ‘back-of-the-envelope’ calculations, and first-order research. This is especially true for some of the more conceptual, but potentially game-changing considerations.
What do you plan to work on, going forward?
We’ll work on whatever we think will generate the most benefit with our additional research hours. This can include areas where:
There has been little useful work done
We think that there are large gains from small amounts of work
We think there might be a game-changing or ‘crucial’ considerations, that could increase the cost-effectiveness of one’s donations by several times.
Where we think we have particular skills that are lacking elsewhere.
Particular questions we will address in the future include:
Should one give now, or invest and give later?
This is an issue that satisfies (1), (3) and (4). It’s potentially game-changing because it’s at least plausible that one can have significantly greater impact with one’s donations by investing one’s money, and donating when we have better evidence regarding charity cost-effectiveness.
Should one give to ‘meta-charity’ such as charitable fundraising or cost-effectiveness research?
This is an area that satisfies at least conditions (1)-(3). The average return on investment from charitable fundraising is 5:1 – far higher than the typical market return on investment – and the highest-return fundraising can significantly outstrip that. This suggests that ‘investment’ in fundraising could be extremely cost-effective.
As well as funding fundraising, one could fund additional research. The current state of knowledge regarding cost-effectiveness is poor, and, in general, where knowledge is poor, the value of gaining new information becomes very high. In the case of cost-effectiveness evaluation, it also has great potential to affect large future donation streams.
What are the best sources of information out there to donors?
This is an area that satisfies conditions (1) and (2). There is a proliferation of charity evaluators out there – which should one trust, and why?
What are the most promising cause-areas for further research?
This is an area that satisfies (1) and (3); it’s also an area that we feel it’s important to explore in order to explain to givers why we focus on the areas that we do. Our primary question is: are there any cause areas that could plausibly contain interventions with expected cost-effectiveness considerably greater than developing world health? Examples of such cause areas include climate change, medical and other research, political lobbying and economic empowerment.
In the near future, we don’t plan to do further in-depth evaluation of developing world health charities, as we’ve found GiveWell’s research in this area to be of high quality, and we don’t think that marginal research hours from us would provide much additional benefit. This kind of work does not appear to be our ‘comparative advantage’ in the charity effectiveness community. However, we will remain as an independent group in this area, and will check GiveWell’s research before relying on it, as we would do with other sources of information.
Many of these questions will be tackled in a new format for us—research papers.
A periodically updated list of possible research topics for new volunteers is available here.
Who’s on the research team?
Below are some of the people involved in our research team.
Rob Wiblin (Staff). As of late September, Rob has been our Director of Research. He graduated top of his year in Applied Economics from the Australia National University, and has worked as a research economist in several Australian Government agencies including the Productivity Commission.
Sally Murray (volunteer). Sally is a graduate student in Development Studies (Research) at the London School of Economics. Prior to this, she read Philosophy, Politics, and Economics at Oxford, and spent a year working for NGOs in India, Cambodia, Malawi and the UK. She currently manages several volunteers in London, with a primary focus on micronutrients.
Owen Cotton-Barratt (volunteer). Owen is a College Lecturer in pure mathematics at Oxford.
Nic Dunkley (volunteer). Nic is a PhD student in machine learning at Oxford University. He graduated with First Class Honours from Cambridge, and received in 96% in his Master’s degree in Applied Statistics at Oxford.
Bastian Stern (intern). Bastian is a postgraduate student in philosophy at Oxford University. He recently graduated top in his year with a double starred first from Cambridge, with one of the best undergraduate performances on record.
Michael Peyton Jones (intern and volunteer). Michael has background in computer programming, and recently graduated with First Class Honours from Oxford in Physics and Philosophy.
Rossa O’Keeffe-O’Donovan (volunteer). Rossa is doing a PhD in development economics at the University of Pennsylvania, on a Fulbright Scholarship. He graduated with First Class Honours in economics from Oxford, where he won a prize for the best undergraduate thesis. He is currently taking time out from Giving What We Can’s research to focus on his studies.
Eliz Kilich (volunteer). Eliz is currently reading Clinical Medicine at Oxford University with an interest in tropical and infectious disease. During her preclinical studies she was awarded a Casberd Scholarship by St. John’s College. She is currently investigating biomedical research and contagious diseases.
The research team also receives input from the Trustees and the Executive Director of Giving What We Can and external sources.
(1) GiveWell also places significant weight on other issues such as transparency and accountability, which they believe to be good proxies for cost effectiveness
(2) Ignoring any effects from changing the amount of currency in circulation!
Researching what we should
Our methodology
As you may have noticed, change is afoot in GWWC. In the past, we feel that our research has been lacking in a few ways, including: a) We haven’t made it clear, publicly, what our methodology has been for assessing charities; b) We haven’t made it clear, publicly, what role GWWC has, relative to other charity evaluators; c) The website has not been updated regularly, so there’s been a disconnect between our research—what we’ve known—and what’s been broadcast publicly; d) We have spent fewer man-hours on research than was desirable. To a large extent, this has been because, until very recently, GWWC was run on an entirely voluntary basis. Now that we have full-time staff, we hope to be able to rectify these issues over time.
As a first step, this blog post and the next will explain our methodology in a little more detail.What metric?
The crucial question for any charity evaluator is: what metric to use? GWWC is solely focused on finding the most cost-effective charities: that is, the charities that have the highest ratio of good produced per dollar given to the charity. Along with GiveWell, we are the only charity evaluators that are solely concerned with cost-effectiveness so understood. (1)
Explanation of what ‘producing good’ consists of
Isn’t the idea of ‘good’ subjective?The grain of truth in that question is that different people’s conceptions of the good vary. Some people think that scientific knowledge is of value in itself; other people think that scientific knowledge is only valuable insofar as it improves the lives of others.
However, the sum total of human wellbeing is a component of any plausible moral view, and, on most moral views, increasing human wellbeing is the main way to do good in the world. For these reasons, it’s this aspect that we focus on. If this is not a component of your moral view, then our research will not be of use to you (but then, if you aren’t concerned by increasing human wellbeing, it’s unlikely that giving to global poverty alleviation is your top priority). In future posts, we’ll explain how to measure increases in human wellbeing.
What does ‘producing’ good mean?
Good question. A mistake to avoid is to think of producing good as to mean causing good. To illustrate, consider two charities that one could give to. Charity A will save 10 lives with every $10 000 given to it; Charity B will save 5 lives with every $10 000 given to it. However, if you don’t give $10 000 to Charity A, it will fill up its room for more funding from other donors (who wouldn’t have given at all otherwise). Whereas if you don’t give the $10 000 to Charity B, it will not fill up its room for more funding from additional donors. In this case, you should donate to Charity B rather than Charity A: though you directly cause 10 lives to be saved by donating to Charity A, you don’t really do any additional good, because those lives would have been saved anyway. In contrast, you do make a difference (of 5 lives saved) by donating to Charity B.
Making this more precise: what producing good means is as follows. First, you consider two possibilities: (i) where you give $n to Charity X; (ii) where you burn the money. Now think: how good is the world in which (i) you give the money? And how good is the world (ii) in which you burn the money? The amount of good you produce by giving $n to charity X is the difference in the goodness of the world in which you do give, minus the goodness of the world in which you burn the money.(2)
Importantly, the above method means that all consequences are considered. So ‘good produced’ does not just mean “impact provable in a few years”; long-run consequences are just as relevant to cost-effectiveness as short-run consequences. Nor does ‘good produced’ mean ‘the effect on the direct beneficiaries of an action’; effects on third parties are just as relevant as effects on direct beneficiaries.
But surely you can never know what organisation does the most good per dollar?That’s exactly right – we’ll never know what the best charity is. However, that doesn’t prevent charity recommendations from being possible. It just means that we need to think in terms of expected good. To see this concept in action, consider two donation opportunities. Donating $10 000 to charity A will certainly save one life. Donating $10 000 to charity B will probably (80% likelihood) save two lives; but has some chance (20%) of producing no benefit.
In general, to work out the expected value of an action X, you look at each possible outcome that could result from X; for each possible outcome, you multiply the probability of that outcome occurring by the goodness of that outcome, if it does occur; and you then take the sum of this product across all possible outcomes. So the expected value of donating $10 000 to charity A is 1 life x 100% = 1; whereas the expected value of donating $10 000 to charity B is 2 lives x 80% = 1.6.
The concept of expected value becomes especially important in situations of great uncertainty, such as charity evaluation. Further posts will discuss in more detail how to correctly apply this concept.
What evidence do you use?
In principle, we’re open to evidence from any source, and if certain sources prove to be particularly reliable or unreliable, we will alter which sources we rely on. In the past, we’ve used the latest peer-reviewed research from health and development economics, information from larger cost-effectiveness-focused academic projects such as the Disease Control Priorities in Developing Countries (2nd edition) and the Abdul Latif Jameel Poverty Action Lab, and on research from other charity evaluators, in particular from GiveWell. Like a growing number of organisations, we place particular weight on randomised controlled trials where they are available.
In the near future we will also begin publishing draft reports on topics and passing them around in order to to solicit additional information from experts and the public.Sometimes, we have to rely more on ‘back-of-the-envelope’ calculations, and first-order research. This is especially true for some of the more conceptual, but potentially game-changing considerations.
What do you plan to work on, going forward?
We’ll work on whatever we think will generate the most benefit with our additional research hours. This can include areas where:
Particular questions we will address in the future include:There has been little useful work done
We think that there are large gains from small amounts of work
We think there might be a game-changing or ‘crucial’ considerations, that could increase the cost-effectiveness of one’s donations by several times.
Where we think we have particular skills that are lacking elsewhere.
Should one give now, or invest and give later?
This is an issue that satisfies (1), (3) and (4). It’s potentially game-changing because it’s at least plausible that one can have significantly greater impact with one’s donations by investing one’s money, and donating when we have better evidence regarding charity cost-effectiveness.
Should one give to ‘meta-charity’ such as charitable fundraising or cost-effectiveness research?This is an area that satisfies at least conditions (1)-(3). The average return on investment from charitable fundraising is 5:1 – far higher than the typical market return on investment – and the highest-return fundraising can significantly outstrip that. This suggests that ‘investment’ in fundraising could be extremely cost-effective.
As well as funding fundraising, one could fund additional research. The current state of knowledge regarding cost-effectiveness is poor, and, in general, where knowledge is poor, the value of gaining new information becomes very high. In the case of cost-effectiveness evaluation, it also has great potential to affect large future donation streams.
What are the best sources of information out there to donors?This is an area that satisfies conditions (1) and (2). There is a proliferation of charity evaluators out there – which should one trust, and why?
What are the most promising cause-areas for further research?This is an area that satisfies (1) and (3); it’s also an area that we feel it’s important to explore in order to explain to givers why we focus on the areas that we do. Our primary question is: are there any cause areas that could plausibly contain interventions with expected cost-effectiveness considerably greater than developing world health? Examples of such cause areas include climate change, medical and other research, political lobbying and economic empowerment.
In the near future, we don’t plan to do further in-depth evaluation of developing world health charities, as we’ve found GiveWell’s research in this area to be of high quality, and we don’t think that marginal research hours from us would provide much additional benefit. This kind of work does not appear to be our ‘comparative advantage’ in the charity effectiveness community. However, we will remain as an independent group in this area, and will check GiveWell’s research before relying on it, as we would do with other sources of information.
Many of these questions will be tackled in a new format for us—research papers.
A periodically updated list of possible research topics for new volunteers is available here.
Who’s on the research team?
Below are some of the people involved in our research team.Rob Wiblin (Staff). As of late September, Rob has been our Director of Research. He graduated top of his year in Applied Economics from the Australia National University, and has worked as a research economist in several Australian Government agencies including the Productivity Commission.
Sally Murray (volunteer). Sally is a graduate student in Development Studies (Research) at the London School of Economics. Prior to this, she read Philosophy, Politics, and Economics at Oxford, and spent a year working for NGOs in India, Cambodia, Malawi and the UK. She currently manages several volunteers in London, with a primary focus on micronutrients.
Owen Cotton-Barratt (volunteer). Owen is a College Lecturer in pure mathematics at Oxford.
Nic Dunkley (volunteer). Nic is a PhD student in machine learning at Oxford University. He graduated with First Class Honours from Cambridge, and received in 96% in his Master’s degree in Applied Statistics at Oxford.
Bastian Stern (intern). Bastian is a postgraduate student in philosophy at Oxford University. He recently graduated top in his year with a double starred first from Cambridge, with one of the best undergraduate performances on record.
Michael Peyton Jones (intern and volunteer). Michael has background in computer programming, and recently graduated with First Class Honours from Oxford in Physics and Philosophy.
Rossa O’Keeffe-O’Donovan (volunteer). Rossa is doing a PhD in development economics at the University of Pennsylvania, on a Fulbright Scholarship. He graduated with First Class Honours in economics from Oxford, where he won a prize for the best undergraduate thesis. He is currently taking time out from Giving What We Can’s research to focus on his studies.
Eliz Kilich (volunteer). Eliz is currently reading Clinical Medicine at Oxford University with an interest in tropical and infectious disease. During her preclinical studies she was awarded a Casberd Scholarship by St. John’s College. She is currently investigating biomedical research and contagious diseases.
The research team also receives input from the Trustees and the Executive Director of Giving What We Can and external sources.