(The actual question is in bold; the rest is background before and a potential recommendation on how to communicate your project’s scope after).
Regarding “evaluating the evaluators”: it seems to me that there are two main types of charity evaluators out there. Some seek to identify the single best use of donor funds (“best-charity evaluators”). The funds are the pure example of this, but GiveWell fits well into this camp after it removed GiveDirectly from being a top charity and eliminated the standout charities. I think GW would probably say that a donation to any of the four top charities could plausibly be the highest-impact use of one’s donations, depending on circumstances and imprecision in models.
Others organizations seek to present donors with a wider range of high-effectiveness options, without implying that each could plausibly be the best possible use of donors’ money (“great-charity evaluators”). Donors will need to consult their values and do more of their own research. These organizations often serve the important person of making more donations tax-deductible and saving effective charities the hassle of incorporating and applying in their country of operation. The Life You Can Save is the most obvious example, although it’s not clear to me how much their listings are based on their own evaluations vs. deference to trusted evaluators.
In my view, both types of recommenders play an important role in the effective giving ecosystem—but I appreciate why focusing on best-charity evaluators is consistent with GWWC’s goals for this project.
A footnote in your HLI report makes it sound like you are mainly evaluating against the standards of a best-charity evaluator. Is that an accurate characterization? The quote is:
At the beginning of our evaluating the evaluators project, we considered the relevant bar as to whether the evaluator “reliably recommended and/or granted to the most cost-effective funding opportunities based on a sufficiently plausible worldview.” We received pushback from external reviewers that, taken literally, this was an overly onerous bar that no evaluator could claim to meet. The issue was that we were using “most cost-effective” in a confusing way. We intended it to mean (roughly): “compared to other available recommendations or funding opportunities that we have evaluated, there is no clear superior alternative that seems more cost-effective, ex-ante” Given this is a bit of a mouthful, we decided to explain our approach as we’ve done here: selecting evaluators that we think are best able to help donors maximise their impact.
I have no concerns with that approach, but would recommend making it clear in posts and webpages that you are evaluating best-charity evaluators under standards appropriate for best-charity evaluators. You might also explicitly state that you don’t intend to evaluate great-charity recommenders at least at this time. I think one of the potential pitfalls of an evaluating-the-evaluators project is that people might draw inaccurate inferences from the absence of a major organization from your list. So you might say something like: “Note that we do not evaluate organizations (such as TLYCS) that recommend a broad range of charities without a conclusion that the recommended charity is plausibly the most effective option for donors.”
I think it’s fair to characterise our evaluations as looking for the “best” charity recommendations, rather than the best charity evaluators, or recommendations that reach a particular standard but that are not the best. Though we’re looking to recommend the best charities, we don’t think this means that there’s no value in looking into “great-charity evaluators” as you called them. We don’t have an all-or-nothing approach when looking into an evaluators’ work and recommendation and can choose to only include the recommendations from that evaluator that meet our potentially higher standard. This means, so long as it’s possible some of the recommendations of a “great-charity evaluator” are the best by a particular worldview, we’d see value in looking into them.
In one sense, this increases the bar for our evaluations, but in another it also means an evaluator’s recommendations might be the best even if we weren’t particularly impressed by the quality of the work. For example, suppose there was a cause area for which there was only one evaluator, the threshold for this evaluating being the best may well be: they are doing a sufficiently good job that there is a sufficiently plausible worldview by which donating via their recommendations is still their best option (i.e., compared to donating to the best evaluator in another area).
It’s too early to commit to how we will approach future evaluations, however, we currently lean towards sticking with the core idea of focusing on helping donors “maximise” expected cost-effectiveness, rather than “maximising” the number of donors giving cost-effectively / providing a variety of “great-but-not-best” options.
You might also explicitly state that you don’t intend to evaluate great-charity recommenders at least at this time.
As above, we would see value in looking at charity evaluators who take an approach of recommending everything above a minimum standard, but we would only look to follow the recommendations we thought were the best (...by some sufficiently plausible worldview).
but would recommend making it clear in posts and webpages that you are evaluating best-charity evaluators under standards appropriate for best-charity evaluators
I’d be interested in where you think we could improve our communications here. Part of the challenge we’ve faced is that we want to be careful not to overstate our work. For example, “we only provide recommendations from the best evaluators we know of and have looked into”, is accurate, but “we only provide recommendations from the best evaluators” is not (because there are evaluators we haven’t looked into yet). Another challenge is to not overly qualify everything we say, to the point of being confusing and inaccessible to regular donors. Still, after scrolling through some of our content, I think we could find a way to thread this needle better as it is an important distinction to emphasise — we also don’t want to understate our work!
(The actual question is in bold; the rest is background before and a potential recommendation on how to communicate your project’s scope after).
Regarding “evaluating the evaluators”: it seems to me that there are two main types of charity evaluators out there. Some seek to identify the single best use of donor funds (“best-charity evaluators”). The funds are the pure example of this, but GiveWell fits well into this camp after it removed GiveDirectly from being a top charity and eliminated the standout charities. I think GW would probably say that a donation to any of the four top charities could plausibly be the highest-impact use of one’s donations, depending on circumstances and imprecision in models.
Others organizations seek to present donors with a wider range of high-effectiveness options, without implying that each could plausibly be the best possible use of donors’ money (“great-charity evaluators”). Donors will need to consult their values and do more of their own research. These organizations often serve the important person of making more donations tax-deductible and saving effective charities the hassle of incorporating and applying in their country of operation. The Life You Can Save is the most obvious example, although it’s not clear to me how much their listings are based on their own evaluations vs. deference to trusted evaluators.
In my view, both types of recommenders play an important role in the effective giving ecosystem—but I appreciate why focusing on best-charity evaluators is consistent with GWWC’s goals for this project.
A footnote in your HLI report makes it sound like you are mainly evaluating against the standards of a best-charity evaluator. Is that an accurate characterization? The quote is:
I have no concerns with that approach, but would recommend making it clear in posts and webpages that you are evaluating best-charity evaluators under standards appropriate for best-charity evaluators. You might also explicitly state that you don’t intend to evaluate great-charity recommenders at least at this time. I think one of the potential pitfalls of an evaluating-the-evaluators project is that people might draw inaccurate inferences from the absence of a major organization from your list. So you might say something like: “Note that we do not evaluate organizations (such as TLYCS) that recommend a broad range of charities without a conclusion that the recommended charity is plausibly the most effective option for donors.”
This is a really insightful question!
I think it’s fair to characterise our evaluations as looking for the “best” charity recommendations, rather than the best charity evaluators, or recommendations that reach a particular standard but that are not the best. Though we’re looking to recommend the best charities, we don’t think this means that there’s no value in looking into “great-charity evaluators” as you called them. We don’t have an all-or-nothing approach when looking into an evaluators’ work and recommendation and can choose to only include the recommendations from that evaluator that meet our potentially higher standard. This means, so long as it’s possible some of the recommendations of a “great-charity evaluator” are the best by a particular worldview, we’d see value in looking into them.
In one sense, this increases the bar for our evaluations, but in another it also means an evaluator’s recommendations might be the best even if we weren’t particularly impressed by the quality of the work. For example, suppose there was a cause area for which there was only one evaluator, the threshold for this evaluating being the best may well be: they are doing a sufficiently good job that there is a sufficiently plausible worldview by which donating via their recommendations is still their best option (i.e., compared to donating to the best evaluator in another area).
It’s too early to commit to how we will approach future evaluations, however, we currently lean towards sticking with the core idea of focusing on helping donors “maximise” expected cost-effectiveness, rather than “maximising” the number of donors giving cost-effectively / providing a variety of “great-but-not-best” options.
As above, we would see value in looking at charity evaluators who take an approach of recommending everything above a minimum standard, but we would only look to follow the recommendations we thought were the best (...by some sufficiently plausible worldview).
I’d be interested in where you think we could improve our communications here. Part of the challenge we’ve faced is that we want to be careful not to overstate our work. For example, “we only provide recommendations from the best evaluators we know of and have looked into”, is accurate, but “we only provide recommendations from the best evaluators” is not (because there are evaluators we haven’t looked into yet). Another challenge is to not overly qualify everything we say, to the point of being confusing and inaccessible to regular donors. Still, after scrolling through some of our content, I think we could find a way to thread this needle better as it is an important distinction to emphasise — we also don’t want to understate our work!