Effective Altruism Grants project update

ricoh_aficio29 Sep 2017 20:05 UTC
28 points
35 commentsEA link

The Centre for Effective Altruism has distributed its first round of grants through its new Effective Altruism Grants program. The aim of the project is to solve funding constraints for high-impact projects. You can read more about our motivation and aims.

This post details (1) the grants we’ve made, (2) our assumptions, (3) the grant methodology, (4) cost-benefit considerations, (5) mistakes, (6) difficulties, (7) project changes, and (8) our plans for EA Grants going forward.

Grants

We are sharing information about our grant this round to give people a better sense of what kinds of projects we look for, should we run EA Grants rounds in the future. You can see the grants we made.

We have allocated £369,924 for distribution, withholding the remainder of the allotted £500,000 to further fund some of the current recipients, contingent on performance.

We also facilitated the funding of grants by the Open Philanthropy Project and a couple of private donors.

Assumptions

There were many implicit assumptions we made in deciding that and how to run EA Grants. A few of the major ones include:

Many good projects are hamstrung by small funding gaps.

We believe some high-value projects have unmet funding needs. The individuals and small organizations we decided to fund are generally too small to get on the radar of foundations like the Open Philanthropy Project, and small donors rarely have time or expertise to evaluate many small projects. But there are high returns to funding them.

Value alignment is useful for maintaining project relevance.

In order to be comfortable with this arrangement, we placed particular emphasis on evaluating value alignment, altruistic motivation, and judgment. Value alignment was particularly important, even more than ostensibly well-defined projects, since some autonomy is inevitable. All else equal we preferred projects by applicants who have a track record of doing this or other projects well and on a voluntary or selflessly motivated basis. (One exception to this rule is that we must stipulate that funding is not used for certain activities that don’t fit within our charitable objects.)

At this funding level, a hefty application process would be more costly than useful.

Many grantmaking processes require multi-page proposals. Since our grants were both smaller and more speculative than many of the grants foundations distribute, applications of that length felt unnecessarily costly, both for the applicants and for us as evaluators. This had costs: projects that are hard to describe briefly suffer from insufficient space to make their cases. We tried to get the best of both worlds by requesting additional information where we found them hard to assess with just what we had. We leave open the possibility of longer proposals in the future should we run subsequent rounds.

Grant methodology

The grants application process had three rounds, and is best described as a process-based approach.

First round

In the first round the three grants associates eliminated applications which we could clearly assess would not meet our selection criteria. We received 722 applications and desk rejected 413 of them, about 57% of the applicants.

Second round

The second round involved taking the remaining applications and assessing applicants based on their track record, values, and plans. This assessment adhered to a rubric, weighting each category in accordance with its predictive power for project success. The scores combined into one weighted score per applicant, which we used to rank the remaining applicants.

We then went through the list by rank and chose applicants to interview, discussing applicants about which there was large divergence in scores or general opinion. Given our £500,000 budget and most of three staff members’ time for two weeks, we decided to interview 63 candidates.

Third round

Most candidates had three, 10-minute interviews, which we used to further assess their achievements, values, and plans. Candidates we knew well received only one interview. For candidates with skillsets we couldn’t evaluate internally we arranged a fourth interview with a relevant technical expert. We then used the data from these interviews, as well as any additional information requested from references and/​or the applicants themselves, to adjust their written application scores. While each interviewer could make modifications to scores in all three categories, interviewers each had a category of focus, so their assessments in their respective area received the most weight.

Finally, we went through the new rank-ordered list and decided who to fund and how much. We initially assigned grants values to candidates in rank order until we’d exhausted the funding pool, then adjusted amounts to fit the particular circumstances of the grantees. Such considerations include our credence in the score given, the counterfactuals of funding each candidate, the potential risks associated with the candidate and/​or their proposal, and what candidates could do with money on the margin. We passed promising candidates who did not fit our charitable objects or who requested money out of scope of our funding capacity onto some private donors associated with CEA and/​or the relevant program officer at the Open Philanthropy Project.

Through this process we selected 22 candidates to fund, partially or in full, and passed another 11 on to the Open Philanthropy Project.

Cost-benefit considerations

An important consideration in our thinking is whether or not the costs of running EA Grants exceed its benefits. Since the counterfactual is likely a future grant made by the Open Philanthropy Project, one angle for evaluating EA Grants is to compare its costs and benefits relative to the distribution(s) Open Phil might have made otherwise. CEA distributed £600 per hour worked by the grants team, whereas we estimate Open Phil distributes ~£20,000 per hour. However, we think a comparison made in this way has limitations.

Costs

The costs are the £500,000 disseminated, plus ~740 CEA staff hours thus far. We expect to spend another 100 hours on activities related to this round of grantees, mostly arranging mentors and ensuring financial regulatory compliance. There have also been costs to other EA organizations — mostly the Open Philanthropy Project, who has decided to evaluate and fund some of the grantees who went through the application process.

An Open Phil staff member made a rough guess that it takes them 13-75 hours per grant distributed. Their average grant size is quite a bit larger, so it seems reasonable to assume it would take them about 25 hours to distribute a pot the size of EA Grants. This, of course, ignores the time costs of institution-building. Much of the time we spent in this funding round went to building the internal grants infrastructure and relationships with other funders. Should we run this project again, we expect to be able to run a similar grants process in a fraction of the time.

This ratio has limited meaning, most notably ignoring that Open Phil found this project compelling enough to fund. While they can distribute more funding per hour having achieved scale, we find it plausible that the additional costs for these smaller projects were still worth it on the margin. The cost-per-dollar distributed is less than that of other impact-focused foundations, likely on par if we were to factor in staff overhead time.

Benefits

The benefits are notably quite complicated to calculate. Any individual project is itself going to be challenging to evaluate, since most of the value is likely to come from hard-to-track, long-term changes to sentiments and behavior. Rather than try to compute the value of each grant with a common base metric, we have instead opted for projects that seem robustly positive should they work. This, again, is not unlike Open Phil’s strategy; the real question is how effective we think our distributions were compared to theirs.

It seems likely that we’ve picked up on value they would not have, given the scale of interventions they generally consider, which is more valuable per dollar than where they would have given. Reasons to believe this include:

  • Scaling potential. By funding early-stage projects, many with plans to grow, the returns to funding at this stage are higher variance but also higher potential upside.

  • Inexpensive salaries. Most people requested living wages at or below nonprofit employee salaries.

  • Funding individuals. Not only were salaries cheaper, but individuals are cheaper. Organizations often spend 1.7x an employee’s salary on overhead.

CEA’s counterfactuals are unclear. We are unsure if CEA would have received the additional money were EA Grants not in our plans. Assuming that to be the case, Open Phil might have later granted the money to some other community-building activity. Had CEA staff not worked on this program, we would have accelerated progress on writing collated EA content, built out the EA events infrastructure, and worked on plans for EA academic engagement. As for the projects we funded, we estimate that about one quarter of the projects wouldn’t have happened at all, and the rest would have received less time, since the grantees would have pursued other funding (from the Open Philanthropy Project, or elsewhere) or self-funded by working or going into personal debt.

Mistakes

Our communication was confusing. We initially announced the process with little advertisement. Then, we advertised to the EA Newsletter, but only shortly before the application deadline, and extended the deadline by two days.

We underestimated the number of applications we would receive, which gave us less time per candidate in the initial evaluation than we would have liked. It also caused delays, which we did not adequately communicate to applicants. We should have been less ambitious in setting our initial deadlines for replying, and should have communicated all changes in our timetable immediately and in writing to all applicants.

Our advertisement did not make sufficiently clear that we might not be able to fund educational expenses through CEA. Fortunately, the Open Philanthropy Project was receptive to considering some of the academic applicants.

Difficulties

Project evaluation

We found it hard to make decisions on first-round applications that looked potentially promising but were outside of our in-house expertise. Many applicants had proposals for studies and charities we felt under-qualified to assess. Most of those applicants we turned down; some we deferred to the relevant Open Phil program manager. We are in the process of establishing relationships with domain experts who can help us do this in the future.

Conflicts of interest

One difficulty in running this program is its susceptibility to conflicts of interest (COIs).

Many of the most promising applications came from people who are already deeply involved with the community. Involvement with the community gives us evidence of value-alignment, and the community also provides a context within which it is easier to come up with proposals that we think are important.

Unfortunately, since many applicants, and particularly many of the best, were deeply involved with the community, our assessing staff tended to have many COIs. This includes one of the team members, who was both an grant evaluator and an applicant.

Rather than avoid giving where COIs existed, we adopted a view much like that of the Open Philanthropy Project. You can see the details articulated in Holden Karnofsky’s post on hits-based giving. We recognized and tried to mitigate the effects of COIs by asking for expert input, expecting domain expertise to help correct for personal domain-irrelevant sentiments. However, given our process-based approach and comparatively limited internal capacity, it was both less necessary and less feasible for us to develop in-house expertise on areas about which we formerly knew little, another means of reducing the impact of COIs.

The measures we took include:

  • Blinding applications during the first and second rounds of the application process, such that all written applications received scores while anonymized.

  • Asking staff members to declare conflicts of interest with finalists, where they existed. The team then found replacement interviewers and asked the associated staff member to step out of decisionmaking for those candidates.

  • Deferring applications to staff of the Open Philanthropy Project when the project proposals were outside our domains of expertise.

  • Establishing scores in our rubric associated with observable measures, tying applicants’ scores to specific features of their abilities and plans rather than our general impression.

  • For the grant applicant who was also an assessor, removing him from all discussions about his application, obscuring his score and ranking, and subjecting him to the same evaluation process as all other grantees.

Project changes

We can’t fund educational expenses.

We adhered closely to our grant areas, funding nothing out of scope of what we described on the website. However, we have since determined that we cannot fund all of the projects for which we encouraged people to apply. Most notably, CEA’s charitable objects do not allow us to pay for education expenses, making it impossible for us to give grants for Masters or PhD programs. However, the Open Philanthropy Project is able to do so and has started to consider funding candidates pursuing research in their priority areas.

We are unlikely to make grants for longer than a year.

While we offered opportunities for grant renewal, we didn’t make any grants lasting more than a year. This was more a result of happenstance than an intentional decision. For the few finalists who requested more than a year of funding, we were sufficiently unsure of either their proposal or their future funding situation so as to not want to commit more than a year upfront. That being said, we’re still open to doing so in the future.

Plans going forward

It seems likely that we will run a similar program in the future. Kerry Vaughan has just taken over ownership of this project, and will be in charge of deciding on and implementing changes. That being said, the initial EA Grants team has many ideas of how to improve the scheme, and in particular how to solve the mistakes discussed above. We will coordinate with Kerry and post again when we have more information.

As I will no longer work on this project after October 6th, please direct questions and comments to eagrants@centreforeffectivealtruism.org.

Thanks to Ryan Carey, Rohin Shah, and Vipul Naik for corrections to this post.

ricoh_aficio29 Sep 2017 20:05 UTC
28 points
35 commentsEA link