Upvoted because this is an important topic I’ve seen little discussion of. Although you take pains to draw attention to the limitations of this data set, these caveats aren’t included in the conclusion, so I’d be wary of anyone acting on this verbatim. I’d be interested in seeing drop out rates in other social movements to give a better idea of the base rate.
AdamGleave
Karma: 1,482
There aren’t many people with PhD-level research experience in relevant fields who are focusing on AI safety, so I think it’s a bit early to conclude these skills are “extremely rare” amongst qualified individuals.
AI safety research spans a broad range of areas, but for the more ML-oriented research the skills are, unsurprisingly, not that different from other fields of ML research. There are two main differences I’ve noticed:
In AI safety you often have to turn ill-defined, messy intuitions into formal problem statements before you can start working on them. In other areas of AI, people are more likely to have already formalized the problem for you.
It’s important to be your own harshest critic. This is cultivated in some other fields, such as computer security and (in a different way) in mathematics. But ML tends to encourage a sloppy attitude here.
Both of these I think are fairly easily measurable from looking at someone’s past work and talking to them, though.
Identifying highly capable individuals is indeed hard, but I don’t think this is any more of a problem in AI safety research than in other fields. I’ve been involved in screening in two different industries (financial trading and, more recently, AI research). In both cases there’s always been a lot of guesswork involved, and I don’t get the impression it’s any better in other sectors. If anything I’ve found screening in AI easier: at least you can actually read the person’s work, rather than everything behind behind an NDA (common in many industries).
2017 Donor Lottery Report
Based on Toggl time tracking, I spent 45 hours on the process, including talking to the organisations and preparing the report.
All of the funding was unrestricted, although I had discussions with each organisation about their strategy and provided feedback. The organisations supported are all small, and I expect most of the upside to come from allowing them to demonstrate their worth and grow in the future, so I’d prefer not to constrain their plans.
I had anticipated my donations being somewhat more skewed towards a single organisation, but had always intended to make some grant to all organisations that I felt were promising after conducting an in-depth investigation. In particular, the organisations involved invested significant time (conversations with me, collating relevant data, feedback on this report), and I believe this should be rewarded, similar to the rationale behind GiveWell’s participation grants. I also think it’s likely that much of the money moved as a result of my research will be from third-parties influenced by this post, and I feel my recommendations are more credible if I put “my money where my mouth is”.
I’m sympathetic to the general thrust of the argument, that we should be reasonably optimistic about “business-as-usual” leading to successful narrow alignment. I put particular weight on the second argument, that the AI research community will identify and be successful at solving these problems.
However, you mostly lost me in the third argument. You suggest using whatever state-of-the-art general purpose learning technique exists to model human values, and then optimise them. I’m pessimistic about this since it involves an adversarial relationship between the optimiser (e.g. an RL algorithm) and the learned reward function. This will work if the optimiser is weak and the reward model is strong. But if we are hypothesising a far improved reward learning technique, we should also assume far more powerful RL algorithms than we have today.
Currently, it seems like RL is generally an easier problem than learning a reward function. For example, current IRL algorithms will overfit the reward function to demonstrations in a high-dimensional environment. If you later optimize the reward with an RL algorithm, you get a policy which does well under the learned reward function, but terribly (often worse than random) on the ground truth reward function. This is why you normally learn the policy jointly with the reward in a GAN-based approach. Regularizers to learn a good reward model (which can then generalize) is in active area of research, see e.g. the variational discriminator bottleneck. However, solving it in generality seems very hard. There’s been little success in adversarial defences, which is a related problem, and there are theoretical reasons to believe adversarial examples will be present for any model class in high-dimensional environments.
Overall, I’m optimistic about the research community solving these problems, but think that present techniques are far from adequate. Although improved general-purpose learning techniques will be important, I believe there will also need to be a concerted focus on solving alignment-related problems.
Long-Term Future Fund: Ask Us Anything!
As part of CEA’s due diligence process, all grantees must submit progress reports documenting how they’ve spent their money. If a grantee applies for renewal, we’ll perform a detailed evaluation of their past work. Additionally, we informally look back at past grants, focusing on grants that were controversial at the time, or seem to have been particularly good or bad.
I’d like us to be more systematic in our grant evaluation, and this is something we’re discussing. One problem is that many of the grants we make are quite small: so it just isn’t cost-effective for us to evaluate all our grants in detail. Because of this, any more detailed evaluation we perform would have to be on a subset of grants.
I view there being two main benefits of evaluation: 1) improving future grant decisions; 2) holding the fund accountable. Point 1) would suggest choosing grants we expect to be particularly informative: for example, those where fund managers disagreed internally, or those which we were particularly excited about and would like to replicate. Point 2) would suggest focusing on grants that were controversial amongst donors, or where there were potential conflicts of interest.
It’s important to note that other things help with these points, too. For 1) improving our grant making process, we are working on sharing best-practices between the different EA Funds. For 2) we are seeking to increase transparency about our internal processes, such as in this doc (which we will soon add as an FAQ entry). Since evaluation is time consuming in the short-term we are likely to only evaluate a small percentage of our grants, though we may scale this up as fund capacity grows.
The LTFF is happy to renew grants so long as the applicant has been making strong progress and we believe working independently continues to be the best option for them. Examples of renewals in this round include Robert Miles, who we first funded in April 2019, and Joe Collman, who we funded in November 2019. In particular, we’d be happy to be the #1 funding source of a new EA org for several years (subject to the budget constraints Oliver mentions in his reply).
Many of the grants we make to individuals are for career transitions, such as someone retraining from one research field to another, or for one-off projects. So I would expect most grants to not be renewals. That said, the bar for renewals does tend to be higher. This is because we pursue a hits-based giving approach, so are willing to fund projects that are likely not to work out—but of course will not want to renew the grant if it is clearly not working.
I think being a risk-tolerant funder is particularly valuable since most employers are, quite rightly, risk-averse. Firing people tends to be harmful to morale; internships or probation periods can help, but take a lot of supervisory time. This means people who might be a great hire but are high-variance often don’t get hired. Funding them for a period of time to do independent work can derisk the grantee, since they’ll have a more substantial portfolio to show.
The level of excitement about long-term independent work varies between fund managers. I tend to think it’s hard for people to do great work independently. I’m still open to funding it, but I want to see a compelling case that there’s not an organisation that would be a good home for the applicant. Some other fund managers are more concerned by perverse incentives in established organisations (especially academia), so are more willing to fund independent research.
I’d be interested to hear thoughts on how we could better support our grantees here. We do sometimes forward applications on to other funders (with the applicants permission), but don’t have any systematic program to secure further funding (beyond applying for renewals). We could try something like “demo days” popular in the VC world, but I’m not sure there’s a large enough ecosystem of potential funders for this to be worth it.
I agree with @Habryka that our current process is relatively lightweight which is good for small grants but doesn’t provide adequate accountability for large grants. I think I’m more optimistic about the LTFF being able to grow into this role. There’s a reasonable number of people who we might be excited about working as fund managers—the main thing that’s held us back from growing the team is the cost of coordination overhead as you add more individuals. But we could potentially split the fund into two sub-teams that specialize in smaller and larger grants (with different evaluation process), or even create a separate fund in EA Funds that focuses on more established organisations. Nothing certain yet, but it’s a problem we’re interested in addressing.
The LTFF chooses grants to make from our open application rounds. Because of this, our grant composition depends a lot on the composition of applications we receive. Although we may of course apply a different bar to applications in different areas, the proportion of grants we make certainly doesn’t represent what we think is the ideal split of total EA funding between cause-areas.
In particular, I tend to see more variance in our scores between applications in the same cause-area than I do between cause-areas. This is likely because most of our applications are for speculative or early-stage projects. Given this, if you’re reading this and are interested in applying to the LTFF but haven’t seen us fund projects in your area before—don’t let that put you off. We’re open to funding things in a very broad range of areas provided there’s a compelling long-termist case.
Because cause prioritization isn’t actually that decision relevant for most of our applications, I haven’t thought especially deeply about it. In general, I’d say the fund is comparably excited about marginal work in reducing long-term risks from AI, biosafety, and general longtermist macrostrategy and capacity building. I don’t currently see promising interventions in climate change, which already attracts significant funding from other sources, although we’d be open to funding something that seemed neglected, especially if it focused on mitigating or predicting extreme risks.
One area where there’s active debate is the degree to which we should support general governance improvements. For example, we made a $50,000 grant to the Center for Election Science (CES) in our September 2020 round. CES has significantly more room for funding, so the main thing holding us back was uncertainty regarding the long-termist case for impact compared to more targeted interventions.
From an internal perspective I’d view the fund as being fairly close to risk-neutral. We hear around twice as many complaints that we’re too risk-tolerant than too risk-averse, although of course the people who reach out to us may not be representative of our donors as a whole.
We do explicitly try to be conservative around things with a chance of significant negative impact to avoid the unilateralist’s curse. I’d estimate this affects less than 10% of our grant decisions, although the proportion is higher in some areas, such as community building, biosecurity and policy.
It’s worth noting that, unless I see a clear case for a grant, I tend to predict a low expected value—not just a high-risk opportunity. This is because I think most projects aren’t going to positively influence the long-term future—otherwise the biggest risks to our civilization would already be taken care of. Based on that prior, it takes significant evidence to update me in favour of a grant having substantial positive expected value. This produces similar decisions to risk-aversion with a more optimistic prior.
Unfortunately, it’s hard to test this prior: we’d need to see how good the grants we didn’t make would have been. I’m not aware of any grants we passed on that turned out to be really good. But I haven’t evaluated this systematically, and we’d only know about those which someone else chose to fund.
An important case where donors may be better off making donations themselves rather than donating via us is when they have more information than we do about some promising donation opportunities. In particular, you likely hear disproportionately about grants we rejected from people already in your network. You may be in a much better position to evaluate these than we are, especially if the impact of the grant hinges on the individual’s abilities, or requires a lot of context to understand.
It’s unfortunate that individual donors can’t directly make grants to individuals in a tax efficient manner. You could consider donating to a donor lottery—these will allow you to donate the same amount of money (in expectation) in a tax efficient manner. While grants can only be made within CEA’s charitable objects, this should cover the majority of things donors would want to support, and in any case the LTFF also faces this restriction. (Jonas also mentioned to me that EA Funds is considering offering Donor-Advised Funds that could grant to individuals as long as there’s a clear charitable benefit. If implemented, this would also allow donors to provide tax-deductible support to individuals.)
I largely agree with Habryka’s comments above.
In terms of the contrast with the AWF in particular, I think the funding opportunities in the long-termist vs animal welfare spaces look quite different. One big difference is that interest in long-termist causes has exploded in the last decade. As a result, there’s a lot of talent interested in the area, but there’s limited organisational and mentorship capacity to absorb this talent. By contrast, the animal welfare space is more mature, so there’s less need to strike out in an independent direction. While I’m not sure on this, there might also be a cultural factor—if you’re trying to perform advocacy, it seems useful to have an organisation brand behind you (even if it’s just a one-person org). This seems much less important if you want to do research.
Tangentially, I see a lot of people debating whether EA is talent constrained, funding constrained, vetting constrained, etc. My view is that for most orgs, at least in the AI safety space, they can only grow by a relatively small (10-30%) rate per year while still providing adequate mentorship. This is talent constrained in the sense that having a larger applicant pool will help the orgs select even better people. But adding more talent won’t necessarily increase the number of hires.
While 10-30% is a relatively small growth rate, if it is sustained then I expect it to eventually outstrip growth in the longtermist talent pipeline: my median guess would be sometime in the next 3-7 years. I see the LTFF’s grants to individuals in part trying to bridge the gap while orgs scale up, giving talented people space to continue to develop, and perhaps even found an org. So I’d expect our proportion of individual grants to decline eventually. This is a personal take, though, and I think others on the fund are more excited about independent research on a more long-term basis.
Historically I think the LTFF’s biggest issue has been insufficiently clear messaging, especially for new donors. For example, we received feedback from numerous donors in our recent survey that they were disappointed we weren’t funding interventions on climate change. We’ve received similar feedback from donors surprised by the number of AI-related grants we make. Regardless of whether or not the fund should change the balance of cause areas we fund, it’s important that donors have clear expectations regarding how their money will be used.
We’ve edited the fund page to make our focus areas more explicit, and EA Funds also added Founders Pledge Climate Change Fund for donors who want to focus on that area (and Jonas emailed donors who made this complaint, encouraging to switch their donations to the climate change fund). I hope this will help clarify things, but we’ll have to be attentive to donor feedback both via things like this AMA and our donor survey, so that we can proactively correct any misconceptions.
Another issue I think we have is that we currently lack the capacity to be more proactively engaged with our grantees. I’d like us to do this for around 10% of our grant applications, particularly those where we are a large proportion of an organisation’s budget. In these cases it’s particularly important that we hold the organisation accountable, and provide strategic advice. In around a third of these cases, we’ve chosen not to make the grant because we feel unexcited about the organisation’s current direction, even though we think it could be a good donation opportunity for a more proactive philanthropist. We’re looking to grow our capacity, so we can hopefully pursue more active philanthropy in the future.
Do the LTFF fund managers make forecasts about potential outcomes of grants?
To add to Habryka’s response: we do give each grant a quantitative score (on −5 to +5, where 0 is zero impact). This obviously isn’t as helpful as a detailed probabilistic forecast, but I think it does give a lot of the value. For example, one question I’d like to answer from retrospective evaluation is whether we should be more consensus driven or fund anything that at least one manager is excited about. We could address this by scrutinizing past grants that had a high variance in scores between managers.
I think it might make sense to start doing forecasting for some of our larger grants (where we’re willing to invest more time), and when the key uncertainties are easy to operationalize.
The cop-out answer of course is to say we’d grow the fund team or, if that isn’t an option, we’d all start working full-time on the LTFF and spend a lot more time thinking about it.
If there’s some eccentric billionaire who will only give away their money right now to whatever I personally recommend, then off the top of my head:
-
For any long-termist org who (a) I’d usually want to fund at a small scale; and (b) whose leadership’s judgement I’d trust, I’d give them as much money as they can plausibly make use of in the next 10 years. I expect that even organisations that are not usually considered funding constrained could probably produce 10-20% extra impact if they invested twice as much in their staff (let them rent really close to the office, pay for PAs or other assistants to save time, etc).
I also think there can be value in having an endowment: it lets the organisation make longer-term plans, can raise the organisation’s prestige, and some things (like creating a professorship) often require endowments.
However, I do think there are some cases it can be negative: some organisations benefit a lot from the accountability of donors, and being too well-funded can attract the wrong people, like with the resource curse. So I’d be selective here, but more in terms of “do I trust the board and leadership to a blank cheque” than “at a detailed level, do I think this org is doing the most valuable work?”
-
I’d also be tempted to throw a lot of money at interventions that seem shovel-ready and robustly positive, even if they wouldn’t normally be something I’d be excited about. For example, I’d feel reasonably good about funding the CES for $10-20m, and probably similar sized grants to Nuclear Threat Initiative, etc.
-
This is more speculative, but I’d be tempted to try and become the go-to angel investor or VC fund for AI startups. I think I’m in a reasonably good position for this now, being an AI researcher and also having a finance background, and having a billion dollars would help out here.
The goal wouldn’t be to make money (which is good since most VC’s don’t seem to do that well!) But being an early investor gives a lot of leverage over a company’s direction. Industry is a huge player in fundamental AI research, and in particular I would 85% predict the first transformative AI is developed by an industry lab, not academia. Having a board seat and early insight into a start-up that is about to develop the first transformative AI seems hugely valuable. Of course, there’s no guarantee I’d manage this—perhaps I miss that startup, or a national lab or pre-existing industrial lab (Google/Facebook/Huawei/etc) develops the technologies first. But start-ups are responsible for a big fraction of disruptive technology, so it’s a reasonable bet.
-
Yes, I think we’re definitely limited by our application pool, and it’s something I’d like to change.
I’m pretty excited about the possibility of getting more applications. We’ve started advertising the fund more, and in the latest round we got the highest number of applications we rated as good (score >= 2.0, where 2.5 is the funding threshold). This is about 20-50% more than the long-term trend, though it’s a bit hard to interpret (our scores are not directly comparable across time). Unfortunately the percentage of good applications also dropped this round, so we do need to avoid too indiscriminate outreach to avoid too high a review burden.
I’m most excited about more active grant-making. For example, we could post proposals we’d like to see people work on, or reach out to people in particular areas to encourage them to apply for funding. Currently we’re bottlenecked on fund manager time, but we’re working on scaling that.
I’d be hesitant about funding individuals or organisations that haven’t applied—our application process is lightweight, so if someone chooses not to apply even after we prompt them, that seems like a bad sign. A possible exception would be larger organisations that already make the information we need available for assessment. Right now I’m not excited about funding more large organisations, since I think the marginal impact there is lower, but if the LTFF had a lot more money to distribute then I’d want to scale up our organisation grants.
Of course there’s lots of things we would not want to (or cannot) fund, so I’ll focus on things which I would not want to fund, but which someone reading this might have been interested in supporting or applying for.
-
Organisations or individuals seeking influence, unless they have a clear plan for how to use that influence to improve the long-term future, or I have an exceptionally high level of trust in them
This comes up surprisingly often. A lot of think-tanks and academic centers fall into this trap by default. A major way in which non-profits sustain themselves is by dealing in prestige: universities selling naming rights being a canonical example. It’s also pretty easy to justify to oneself: of course you have to make this one sacrifice of your principles, so you can do more good later, etc.
I’m torn on this because gaining leverage can be a good strategy, and indeed it seems hard to see how we’ll solve some major problems without individuals or organisations pursuing this. So I wouldn’t necessarily discourage people from pursuing this path, though you might want to think hard about whether you’ll be able to avoid value drift. But there’s a big information asymmetry as a donor: if someone is seeking support for something that isn’t directly useful now, with the promise of doing something useful later, it’s hard to know if they’ll follow through on that.
-
Movement building that increases quantity but reduces quality or diversity. The initial composition of a community has a big effect on its long-term composition: people tend to recruit people like themselves. The long-termist community is still relatively small, so we can have a substantial effect on the current (and therefore long-term) composition now.
So when I look for whether to fund a movement building intervention, I don’t just ask if it’ll attract enough good people to be worth the cost, but also whether the intervention is sufficiently targeted. This is a bit counterintuitive, and certainly in the past (e.g. when I was running student groups) I tended to assume that bigger was always better.
That said, the details really matter here. For example, AI risk is already in the public conscience, but most people have only been exposed to terrible low-quality articles about it. So I like Robert Miles YouTube channel since it’s a vastly better explanation of AI risk than most people will have come across. I still think most of the value will come from a small percentage of people who seriously engage with it, but I expect it to be positive or at least neutral for the vast majority of viewers.
-
These are very much a personal take, I’m not sure if others on the fund would agree.
-
Buying extra time for people already doing great work. A lot of high-impact careers pay pretty badly: many academic roles (especially outside the US), some non-profit and think-tank work, etc. There’s certainly diminishing returns to money, and I don’t want the long-termist community to engage in zero-sum consumption of Veblen goods. But there’s also plenty of things that are solid investments in your productivity, like having a comfortable home office, a modern computer, ordering takeaway or having cleaners, enough runway to not have financial insecurity, etc.
Financial needs also vary a fair bit from person to person. I know some people who are productive and happy living off Soylent and working on a laptop on their bed, whereas I’d quickly burn out doing that. Others might have higher needs than me, e.g. if they have financial dependents.
As a general rule, if I’d be happy to fund someone for $Y/year if they were doing this work by themselves, and they’re getting paid $X/year by their employer to do this work, I think I should be happy to pay the difference $(Y-X)/year provided the applicant has a good plan for what to do with the money. If you think you might benefit from more money, I’d encourage you to apply. Even if you don’t think you’ll get it: a lot of people underestimate how much their time is worth.
-
Biosecurity. At the margins I’m about equally excited by biosecurity as I am about mitigating AI risks, largely because biosecurity currently seems much more neglected from a long-termist perspective. Yet the fund makes many more grants in the AI risk space.
We have received a reasonable number of biosecurity applications in recent rounds (though we still receive substantially more for AI), but our acceptance rate has been relatively low. I’d be particularly excited about seeing applications with a relatively clear path to impact. Many of our applications have been for generally trying to raise awareness, and I think getting the details right is really crucial here: targeting the right community, having enough context and experience to understand what that community would benefit from hearing, etc.
- 's comment on Long-Term Future Fund: Ask Us Anything! by (10 Dec 2020 9:54 UTC; 28 points)
- 's comment on Long-Term Future Fund: Ask Us Anything! by (6 Dec 2020 12:05 UTC; 13 points)
- 's comment on Long-Term Future Fund: Ask Us Anything! by (10 Dec 2020 15:30 UTC; 7 points)
- 's comment on Long-Term Future Fund: Ask Us Anything! by (10 Dec 2020 8:09 UTC; 3 points)
-
I’ve already covered in this answer areas where we don’t make many grants but I would be excited about us making more grants. So in this answer I’ll focus on areas where we already commonly make grants, but would still like to scale this up further.
I’m generally excited to fund researchers when they have a good track record, are focusing on important problems and when the research problem is likely to slip through the cracks of other funders or research groups. For example, distillation style research, or work that is speculative or doesn’t neatly fit into an existing discipline.
Another category which is a bit harder to define are grants where we have a comparative advantage at evaluating. This could be that one of the fund managers happens to already be an expert in the area and has a lot of context. Or maybe the application is time-sensitive and we’re just about to start evaluating a grant round. In these cases the counterfactual impact is higher: these grants are less likely to be made by other donors.
Thanks for running this! It’s unfortunate this is at the same time as ICML/IJCAI/AAMAS, I’d have been interested in attending otherwise. Not sure what proportion of your target audience go to the major ML conferences, but might be worth trying to schedule around them for next year.