Hello Katie, thanks for writing this up. I’m very concerned, however, about the evaluation criteria:
The primary metric used to assess grants at the end of the first year is the number of group members who apply for internships or graduate programs in priority areas and reach at least the interview stage [...]
The 80k’s priority paths are basically X-risk, work at an EA org (does ‘cause prioritisation’ happen anywhere else?) and earn to give. As such, on your stated primary metric, it seems anyone who switched their career from doing nothing to working on any of animals, global poverty, mental health or climate change would not count as a valuable shift. In other words, CEA only counts a career change as worthwhile if it focuses on the far future.
Three points on this. First, if this is the case—and please tell me if I’m mistaken—I consider it to be deeply regrettable. It moves the EA community from being a morally inclusive one (I’ve written on this previously), which brings together individuals who do—according to their own lights—the most good they can, and seek to help each other, to a morally exclusive one, where there is a right answer, we need to fight over what it is, and CEA will financially reward you if you accept their answer. It seems the sort of thing that would lead to the split of the EA movement (i.e “if EA is just effective futurism, those of us who aren’t effective futurists should go and do our own thing”).
Second, I note that, according to the latest EA survey, 54% of those who identify as EA consider those four areas—animals welfare/rights, global poverty, mental health and climate change—to be the ‘top cause’. Hence the evaluation metric is markedly not representative of the EA community (in the sense of matching the community’s priorities). You might think the EA has just got it wrong, but there is still an oddness here, because you’ve stated
We hope to avoid these negative effects by providing after-the-fact assignment of credit for outcomes outside the scope of the primary success criteria, and by emphasising the criterion of ‘being a good representative of EA’
I submit that your evaluation criteria—using 80k’s list—fails your own representativeness test, because it judges impact by a standard that would not be widely accepted among self-identifying effective altruists.
Third, given the evaluation criteria is quite sharply divergent from what people would expect, I would have appreciated if this had been flagged more clearly.
It seems like the nature of your concern originates from this paragraph:
“We used the 80,000 Hours list of priority paths as the basis for our list of accredited roles, but expanded it to be somewhat broader. The areas and roles that we intend to accredit are still being decided upon, and we expect the number of accredited roles and areas to increase in the future. We’ve chosen a relatively restricted set of criteria for the time being, as we think the costs to later restricting the criteria will be significantly higher than the costs of expanding them.”
I think some additional color on how we expanded the criteria to be broader than 80K priority paths and how we expect to expand it to be broader in the future will mitigate some (but not all) of your concerns.
80K priority paths are the initial basis of our list of accredited roles because we have high credence that these are high-impact roles and because 80K can provide infrastructure to assist in helping members of local groups connect with some of these roles. We don’t expect that these are the only roles that are important and it is not our intent to accredit only these roles.
Interestingly, our initial list of accredited roles included any Open Phil or GiveWell grantee in order to make the list more inclusive. We now feel that this expansion might have been too inclusive because of the significant expansion in the organizations Open Phil has provided with grants. (For example, we probably wouldn’t be happy to accredit someone working at VasoRx without further information.)
Our current best guess for how to proceed is to accredit anyone working at an organization on the 80K Job Board (which includes global poverty and animal welfare organization) but continue to review individual positive outcomes on a case-by-case basis and accredit some career-related outcomes that are neither on the 80K job board nor Open Phil/GiveWell grantees.
Unfortunately, any list of accredited outcomes that we distribute to grantees will be imperfect. As we mentioned in the original post, our goals in designing an evaluation process for EA community grants were to:
Give groups clear guidance on what would cause us to evaluate their activities favourably.
Make it easy for the EA community and potential funders to understand and evaluate the success of EA Community Grants.
Provide sufficient evidence of the value produced to enable CEA to make well-informed decisions on whether to renew funding for given groups, and whether to scale the EA Community Grants process.
Avoid incentivising groups to optimise for our metrics rather than for what is actually highest-impact.
Minimise the time cost to CEA and to groups in evaluating their results.
Our current best guess for how to do this is to use the cause prioritization research done by others to create an initial list of clear, easy-to-understand positive outcomes that we can communicate to CEA’s donors and grantees and then to accredit additional positive outcomes on a case-by-case basis. We think this is superior to accrediting everything on a case-by-case basis because it reduces uncertainty for grantees and makes the project easier for funders to evaluate. We also think that 80K represents the best research on what careers have an impact, so it would be surprising if we don’t take advantage of that resource.
If you think we should accredit entirely different outcomes, think this is the wrong approach for accrediting career-related outcomes, or think that we should use a different list, we’d be very open to suggestions. We plan to make the next iterations to our accredited criteria before the next application round in January so feedback now would be particularly timely.
Hello Katie, thanks for writing this up. I’m very concerned, however, about the evaluation criteria:
The 80k’s priority paths are basically X-risk, work at an EA org (does ‘cause prioritisation’ happen anywhere else?) and earn to give. As such, on your stated primary metric, it seems anyone who switched their career from doing nothing to working on any of animals, global poverty, mental health or climate change would not count as a valuable shift. In other words, CEA only counts a career change as worthwhile if it focuses on the far future.
Three points on this. First, if this is the case—and please tell me if I’m mistaken—I consider it to be deeply regrettable. It moves the EA community from being a morally inclusive one (I’ve written on this previously), which brings together individuals who do—according to their own lights—the most good they can, and seek to help each other, to a morally exclusive one, where there is a right answer, we need to fight over what it is, and CEA will financially reward you if you accept their answer. It seems the sort of thing that would lead to the split of the EA movement (i.e “if EA is just effective futurism, those of us who aren’t effective futurists should go and do our own thing”).
Second, I note that, according to the latest EA survey, 54% of those who identify as EA consider those four areas—animals welfare/rights, global poverty, mental health and climate change—to be the ‘top cause’. Hence the evaluation metric is markedly not representative of the EA community (in the sense of matching the community’s priorities). You might think the EA has just got it wrong, but there is still an oddness here, because you’ve stated
I submit that your evaluation criteria—using 80k’s list—fails your own representativeness test, because it judges impact by a standard that would not be widely accepted among self-identifying effective altruists.
Third, given the evaluation criteria is quite sharply divergent from what people would expect, I would have appreciated if this had been flagged more clearly.
Hi Michael,
It seems like the nature of your concern originates from this paragraph:
“We used the 80,000 Hours list of priority paths as the basis for our list of accredited roles, but expanded it to be somewhat broader. The areas and roles that we intend to accredit are still being decided upon, and we expect the number of accredited roles and areas to increase in the future. We’ve chosen a relatively restricted set of criteria for the time being, as we think the costs to later restricting the criteria will be significantly higher than the costs of expanding them.”
I think some additional color on how we expanded the criteria to be broader than 80K priority paths and how we expect to expand it to be broader in the future will mitigate some (but not all) of your concerns.
80K priority paths are the initial basis of our list of accredited roles because we have high credence that these are high-impact roles and because 80K can provide infrastructure to assist in helping members of local groups connect with some of these roles. We don’t expect that these are the only roles that are important and it is not our intent to accredit only these roles.
Interestingly, our initial list of accredited roles included any Open Phil or GiveWell grantee in order to make the list more inclusive. We now feel that this expansion might have been too inclusive because of the significant expansion in the organizations Open Phil has provided with grants. (For example, we probably wouldn’t be happy to accredit someone working at VasoRx without further information.)
Our current best guess for how to proceed is to accredit anyone working at an organization on the 80K Job Board (which includes global poverty and animal welfare organization) but continue to review individual positive outcomes on a case-by-case basis and accredit some career-related outcomes that are neither on the 80K job board nor Open Phil/GiveWell grantees.
Unfortunately, any list of accredited outcomes that we distribute to grantees will be imperfect. As we mentioned in the original post, our goals in designing an evaluation process for EA community grants were to:
Give groups clear guidance on what would cause us to evaluate their activities favourably.
Make it easy for the EA community and potential funders to understand and evaluate the success of EA Community Grants.
Provide sufficient evidence of the value produced to enable CEA to make well-informed decisions on whether to renew funding for given groups, and whether to scale the EA Community Grants process.
Avoid incentivising groups to optimise for our metrics rather than for what is actually highest-impact.
Minimise the time cost to CEA and to groups in evaluating their results.
Our current best guess for how to do this is to use the cause prioritization research done by others to create an initial list of clear, easy-to-understand positive outcomes that we can communicate to CEA’s donors and grantees and then to accredit additional positive outcomes on a case-by-case basis. We think this is superior to accrediting everything on a case-by-case basis because it reduces uncertainty for grantees and makes the project easier for funders to evaluate. We also think that 80K represents the best research on what careers have an impact, so it would be surprising if we don’t take advantage of that resource.
If you think we should accredit entirely different outcomes, think this is the wrong approach for accrediting career-related outcomes, or think that we should use a different list, we’d be very open to suggestions. We plan to make the next iterations to our accredited criteria before the next application round in January so feedback now would be particularly timely.