Thanks for the post! I wanted to reply to a couple ideas you raised for GiveWell:
(1) Assess outcomes.
Many of the points you raised, such as making empirical after-the-fact estimates, relate to the question of why GiveWell isn’t putting more effort into collecting and examining post-hoc data demonstrating the impacts of our top charities.
We provide an estimate of the impact of a donation to each of our top charities, in humanitarian terms, here: http://www.givewell.org/charities/top-charities/impact. As you note, this is based on the expected impact of donations made to the charity today, rather than a look back at the impact of past work by the charity. We don’t currently collect outcomes data of the type you’re describing (we have generally focused on collecting data, for example, to show that children received deworming treatments, rather than that recipients’ income later improved; we rely on previously conducted studies to estimate the connection between the two).
Our impression is that post-hoc outcomes data of the type you’re describing—on lives saved, health outcomes, future test scores, or income—is largely not available from charities, even our top charities, such that we’d need to fund or collect it ourselves. We are taking some steps to do this in our work with IDinsight, an organization that supports and conducts rigorous evaluations of development interventions. As part of GiveWell Incubation Grants, Good Ventures supported the creation of an IDinsight embedded team at GiveWell to improve monitoring of GiveWell top charities and support the development of potential future GiveWell recommendations. Our work with IDinsight has so far furthered our impression that this information is quite challenging to collect, even with additional funding and effort, such that we’d most likely want to take a judicious approach and complete such work in conjunction with specialized, third-party organizations. In general, we expect significant time and resources are needed to gather data of the type you’re referring to. We’re willing to put these resources in when they seem likely to further our understanding, as demonstrated by our work with IDinsight, but expect a high ratio of resource-input to better-information output.
I’m unsure whether the GiveDirectly study remains underfunded; the footnote you cited referenced a conversation from 2014, and it appears from looking at our subsequent communications that the study has gone forward. But to your point about whether we should fund studies like these: We agree that we should strongly consider doing so, and decisions to do so will depend on room for more funding, as well as our expectation that the study will have sufficient power to be informative for our recommendations.
(Continued in next comment)
**Comment was edited to clarify impact of deworming treatment.
We agree that not everyone has an accurate view of GiveWell’s work, and that we should continue to improve our communications around the kinds of opportunities we recommend. Publishing information about our reasoning and goals on our website and blog is one way we aim to do this, as is speaking with the media and donors who use our research, but we agree there is room for improvement. In my experience working on GiveWell’s outreach, it has been particularly challenging to effectively communicate around the following:
We think we can continue to improve in our written and verbal communications around these topics. A goal on our website and in our own communications is to provide the most accurate picture we can at any given level of detail, within reasonable bounds of staff capacity and time. Someone who only reads a headline on our website should have the most accurate picture it’s possible to have after reading only a headline; someone who only reads our page listing top charities should have the most accurate picture associated with that level of detail, and so on.
Going forward, we think we can improve by more proactively reaching out when we become aware of a misimpression of GiveWell. We’ve had internal discussions about this, prompted by this post, and plan to more proactively communicate about mistakes or misunderstandings of our work when we become aware of them, including emailing the media with clarifications. We did not do this in the case of the quote you cite from The Atlantic article, and on reflection think this is something we should do in the future.
On the name of “top charities”: We’d be interested in whether there is a term that you feel would more succinctly and accurately convey our views on our recommendations than “top charities.” We want to avoid projecting overconfidence, but we also don’t want to suggest we’re less confident or think these are less good options for most donors than we believe they are. A concern with applying a restricted category, such as “top charities within global health and development,” would be suggesting that we hadn’t considered opportunities that fall outside of this category or that we chose this category arbitrarily, neither of which is true: GiveWell focuses on global health and development because our initial research led us to believe that the charities most likely to succeed by GiveWell’s criteria of cost-effectiveness and a strong, generalizable evidence base work in global health and development. (More on this here and here.) Similarly, a concern with referring to GiveWell’s top charities as something like “reasonably good options” could lead donors to be confused about whether we were recommending them—which, in the case of most low-time, low-trust donors (more below; this is the group we believe makes up most of our donor base), we do think they likely represent the best options.
GiveWell and the Open Philanthropy Project, the “last dollar,” and whether GiveWell’s top charities are better giving opportunities
Here, it’s helpful to distinguish between GiveWell and the Open Philanthropy Project, which are currently part of the same organization, but which we plan to legally separate this year. GiveWell—referring to our longtime work to find and recommend top charities, as described on www.givewell.org—does not have an organizational position on the “last dollar” question for Good Ventures, because GiveWell’s mission is to recommend and move money to the charities that meet its criteria; it does not have an organizational mission to consider Good Ventures’ overall or long-term budget. However, staff members serving the Open Philanthropy Project do, since theirs is a long-term partnership with Good Ventures and they are actively weighing tradeoffs between grantmaking opportunities that Good Ventures will have resources to fund in the short- and long-run. The Open Philanthropy Project views the “last dollar” discussion you refer to above (http://www.openphilanthropy.org/blog/good-ventures-and-giving-now-vs-later-2016-update) as unstable, and thinks it would be a mistake to view its recent writeup as high confidence that Open Philanthropy Project opportunities are better—in some sort of absolute sense—than GiveWell top charities.
GiveWell believes the following is true:
We do think GiveWell’s top charities represent the best giving opportunities we’re aware of for donors who have limited time to spend on their decision. That’s due to the strength of the evidence base, cost-effectiveness, and their transparency and ability to be vetted and spot-checked by donors with even a low degree of trust in GiveWell. As noted on GiveWell’s top charities page, “They represent the best opportunities we’re aware of to help low-income people with relatively high confidence and relatively short time horizons.”
We also think that it’s possible that more cost-effective or otherwise ‘better’ giving opportunities exist, but that we a) haven’t found them yet, and/or b) don’t consider them as potential top charities because they fail to meet our criteria (e.g., not having a strong evidence base or not needing additional funding), which were designed to serve low-time donors and produce the types of recommendations described above.
Some donors—who have a high degree of trust in a particular person or organization and want to outsource their thinking about giving to that person or organization, or who have a large amount of time with which to spend identifying and assessing giving opportunities—might identify other opportunities they feel offer the best bang for their buck, such as those identified by the Open Philanthropy Project.
We hope that the separation of the Open Philanthropy Project from GiveWell this year clarifies the difference in our approaches, which we believe is also a source of confusion.
Thanks again for your thoughtful post on our work.
Hi Ben,
Thanks for the post! I wanted to reply to a couple ideas you raised for GiveWell:
(1) Assess outcomes.
Many of the points you raised, such as making empirical after-the-fact estimates, relate to the question of why GiveWell isn’t putting more effort into collecting and examining post-hoc data demonstrating the impacts of our top charities.
We provide an estimate of the impact of a donation to each of our top charities, in humanitarian terms, here: http://www.givewell.org/charities/top-charities/impact. As you note, this is based on the expected impact of donations made to the charity today, rather than a look back at the impact of past work by the charity. We don’t currently collect outcomes data of the type you’re describing (we have generally focused on collecting data, for example, to show that children received deworming treatments, rather than that recipients’ income later improved; we rely on previously conducted studies to estimate the connection between the two).
Our impression is that post-hoc outcomes data of the type you’re describing—on lives saved, health outcomes, future test scores, or income—is largely not available from charities, even our top charities, such that we’d need to fund or collect it ourselves. We are taking some steps to do this in our work with IDinsight, an organization that supports and conducts rigorous evaluations of development interventions. As part of GiveWell Incubation Grants, Good Ventures supported the creation of an IDinsight embedded team at GiveWell to improve monitoring of GiveWell top charities and support the development of potential future GiveWell recommendations. Our work with IDinsight has so far furthered our impression that this information is quite challenging to collect, even with additional funding and effort, such that we’d most likely want to take a judicious approach and complete such work in conjunction with specialized, third-party organizations. In general, we expect significant time and resources are needed to gather data of the type you’re referring to. We’re willing to put these resources in when they seem likely to further our understanding, as demonstrated by our work with IDinsight, but expect a high ratio of resource-input to better-information output.
I’m unsure whether the GiveDirectly study remains underfunded; the footnote you cited referenced a conversation from 2014, and it appears from looking at our subsequent communications that the study has gone forward. But to your point about whether we should fund studies like these: We agree that we should strongly consider doing so, and decisions to do so will depend on room for more funding, as well as our expectation that the study will have sufficient power to be informative for our recommendations.
(Continued in next comment)
**Comment was edited to clarify impact of deworming treatment.
(Continued from previous comment)
(2) Market humbly.
We agree that not everyone has an accurate view of GiveWell’s work, and that we should continue to improve our communications around the kinds of opportunities we recommend. Publishing information about our reasoning and goals on our website and blog is one way we aim to do this, as is speaking with the media and donors who use our research, but we agree there is room for improvement. In my experience working on GiveWell’s outreach, it has been particularly challenging to effectively communicate around the following:
a) The uncertainty associated with deworming research. b) The limitations of our cost-effectiveness analyses. c) The type of opportunities GiveWell considers as potential top charities, and why.
We think we can continue to improve in our written and verbal communications around these topics. A goal on our website and in our own communications is to provide the most accurate picture we can at any given level of detail, within reasonable bounds of staff capacity and time. Someone who only reads a headline on our website should have the most accurate picture it’s possible to have after reading only a headline; someone who only reads our page listing top charities should have the most accurate picture associated with that level of detail, and so on.
Going forward, we think we can improve by more proactively reaching out when we become aware of a misimpression of GiveWell. We’ve had internal discussions about this, prompted by this post, and plan to more proactively communicate about mistakes or misunderstandings of our work when we become aware of them, including emailing the media with clarifications. We did not do this in the case of the quote you cite from The Atlantic article, and on reflection think this is something we should do in the future.
On the name of “top charities”: We’d be interested in whether there is a term that you feel would more succinctly and accurately convey our views on our recommendations than “top charities.” We want to avoid projecting overconfidence, but we also don’t want to suggest we’re less confident or think these are less good options for most donors than we believe they are. A concern with applying a restricted category, such as “top charities within global health and development,” would be suggesting that we hadn’t considered opportunities that fall outside of this category or that we chose this category arbitrarily, neither of which is true: GiveWell focuses on global health and development because our initial research led us to believe that the charities most likely to succeed by GiveWell’s criteria of cost-effectiveness and a strong, generalizable evidence base work in global health and development. (More on this here and here.) Similarly, a concern with referring to GiveWell’s top charities as something like “reasonably good options” could lead donors to be confused about whether we were recommending them—which, in the case of most low-time, low-trust donors (more below; this is the group we believe makes up most of our donor base), we do think they likely represent the best options.
GiveWell and the Open Philanthropy Project, the “last dollar,” and whether GiveWell’s top charities are better giving opportunities
Here, it’s helpful to distinguish between GiveWell and the Open Philanthropy Project, which are currently part of the same organization, but which we plan to legally separate this year. GiveWell—referring to our longtime work to find and recommend top charities, as described on www.givewell.org—does not have an organizational position on the “last dollar” question for Good Ventures, because GiveWell’s mission is to recommend and move money to the charities that meet its criteria; it does not have an organizational mission to consider Good Ventures’ overall or long-term budget. However, staff members serving the Open Philanthropy Project do, since theirs is a long-term partnership with Good Ventures and they are actively weighing tradeoffs between grantmaking opportunities that Good Ventures will have resources to fund in the short- and long-run. The Open Philanthropy Project views the “last dollar” discussion you refer to above (http://www.openphilanthropy.org/blog/good-ventures-and-giving-now-vs-later-2016-update) as unstable, and thinks it would be a mistake to view its recent writeup as high confidence that Open Philanthropy Project opportunities are better—in some sort of absolute sense—than GiveWell top charities.
GiveWell believes the following is true:
We do think GiveWell’s top charities represent the best giving opportunities we’re aware of for donors who have limited time to spend on their decision. That’s due to the strength of the evidence base, cost-effectiveness, and their transparency and ability to be vetted and spot-checked by donors with even a low degree of trust in GiveWell. As noted on GiveWell’s top charities page, “They represent the best opportunities we’re aware of to help low-income people with relatively high confidence and relatively short time horizons.”
We also think that it’s possible that more cost-effective or otherwise ‘better’ giving opportunities exist, but that we a) haven’t found them yet, and/or b) don’t consider them as potential top charities because they fail to meet our criteria (e.g., not having a strong evidence base or not needing additional funding), which were designed to serve low-time donors and produce the types of recommendations described above.
Some donors—who have a high degree of trust in a particular person or organization and want to outsource their thinking about giving to that person or organization, or who have a large amount of time with which to spend identifying and assessing giving opportunities—might identify other opportunities they feel offer the best bang for their buck, such as those identified by the Open Philanthropy Project.
This is discussed on GiveWell’s top charities page: www.givewell.org/charities/top-charities#Proscons.
We hope that the separation of the Open Philanthropy Project from GiveWell this year clarifies the difference in our approaches, which we believe is also a source of confusion.
Thanks again for your thoughtful post on our work.