I am wondering how the fund managers are thinking more long-term about encouraging more independent researchers and projects to come into existence and stay in existence. So far as I can tell, there hasn’t been much renewed granting to independent individuals and projects (i.e. granting for a second or third time to grantees who have previously already received an LTFF grant). Do most grantees have a solid plan for securing funding after their LTFF grant money runs out, and if so what do they tend to do?
I think LTFF is doing something valuable by giving people the freedom to not “sell out” to more traditional or mass-appeal funding sources (e.g. academia, established orgs, Patreon). I’m worried about a situation where receiving a grant from LTFF isn’t enough to be sustainable, so that people go back to doing more “safe” things like working in academia or at an established org.
The LTFF is happy to renew grants so long as the applicant has been making strong progress and we believe working independently continues to be the best option for them. Examples of renewals in this round include Robert Miles, who we first funded in April 2019, and Joe Collman, who we funded in November 2019. In particular, we’d be happy to be the #1 funding source of a new EA org for several years (subject to the budget constraints Oliver mentions in his reply).
Many of the grants we make to individuals are for career transitions, such as someone retraining from one research field to another, or for one-off projects. So I would expect most grants to not be renewals. That said, the bar for renewals does tend to be higher. This is because we pursue a hits-based giving approach, so are willing to fund projects that are likely not to work out—but of course will not want to renew the grant if it is clearly not working.
I think being a risk-tolerant funder is particularly valuable since most employers are, quite rightly, risk-averse. Firing people tends to be harmful to morale; internships or probation periods can help, but take a lot of supervisory time. This means people who might be a great hire but are high-variance often don’t get hired. Funding them for a period of time to do independent work can derisk the grantee, since they’ll have a more substantial portfolio to show.
The level of excitement about long-term independent work varies between fund managers. I tend to think it’s hard for people to do great work independently. I’m still open to funding it, but I want to see a compelling case that there’s not an organisation that would be a good home for the applicant. Some other fund managers are more concerned by perverse incentives in established organisations (especially academia), so are more willing to fund independent research.
I’d be interested to hear thoughts on how we could better support our grantees here. We do sometimes forward applications on to other funders (with the applicants permission), but don’t have any systematic program to secure further funding (beyond applying for renewals). We could try something like “demo days” popular in the VC world, but I’m not sure there’s a large enough ecosystem of potential funders for this to be worth it.
I want to see a compelling case that there’s not an organisation that would be a good home for the applicant.
My impression is that it is not possible for everyone who want to help with the long term ti get hired by an org, for the simple reason that there are not enough openings at those orgs. At least in AI Safety, all entry level jobs are very competitive, meaning that not getting in is not a strong signal that one could not have done well there.
I can’t respond for Adam, but just wanted to say that I personally agree with you, which is one of the reasons I’m currently excited about funding independent work.
Thanks for picking up the thread here Asya! I think I largely agree with this, especially about the competitiveness in this space. For example, with AI PhD applications, I often see extremely talented people get rejected who I’m sure would have got an offer a few years ago.
I’m pretty happy to see the LTFF offering effectively “bridge” funding for people who don’t quite meet the hiring bar yet, but I think are likely to in the next few years. However, I’d be hesitant about heading towards a large fraction of people working independently long-term. I think there’s huge advantages from the structure and mentorship an org can provide. If orgs aren’t scaling up fast enough, then I’d prefer to focus on trying to help speed that up.
The main way I could see myself getting more excited about long-term independent research is if we saw flourishing communities forming amongst independent researchers. Efforts like LessWrong and the Alignment Forum help in terms of providing infrastructure. But right now it still seems much worse than working for an org, especially if you want to go down any of the more traditional career paths later. But I’d love to be proven wrong here.
I claim we have proof of concept. The people who started the existing AI Safety research orgs did not have AI Safety mentors. Current independent researcher have more support than they had. In a way an org is just a crystalized collaboration of previously independent researchers.
I think that there are some PR reasons why it would be good if most AI Safety researchers where part of academia or other respectable orgs (e.g. DeepMind). But I also think it is good to have a minority of researchers who are disconnected from the particular pressures of that environment.
However, being part of academia is not the same as being part of an AI Safety org. MIRI people are not part of academia, and someone doing AI Safety research as part of their PhD in a “normal” (not AI Safety focused) PhD program, is sorta an independent researcher.
The main way I could see myself getting more excited about long-term independent research is if we saw flourishing communities forming amongst independent researchers.
We are working on that. I’m not optimistic about current orgs keeping up with the growth of the field, and I don’t think it is healthy for the career to be too competitive, since this will lead to goodhearted on career intensives. But I do think a looser structure, built on personal connections rather than formal org employment, can grow in a much more flexible way, and we are experimenting with various methods to make this happen.
Yeah, I am also pretty worried about this. I don’t think we’ve figured out a great solution to this yet.
I think we don’t really have sufficient capacity to evaluate organizations on an ongoing basis and provide good accountability. Like, if a new organization were to be funded by us and then grow to a budget of $1M a year, I don’t feel like we have the capacity to evaluate their output and impact sufficiently well to justify giving them $1M each year (or even just $500k).
Our current evaluation process routes feels pretty good for smaller projects, and granting to established organizations that have other active evaluators looking into them that we can talk to, but doesn’t feel very well-suited to larger organizations that don’t have existing evaluations done on them (there is a lot of due diligence work to be done on that I think requires higher staff capacity than we have).
I also think the general process of the LTFF specializing into more something like venture funding, with other funders stepping in for more established organizations feels pretty good to me. I do think the current process has a lot of unnecessary uncertainty and risk in it, and I would like to work on that. So one thing I’ve been trying to get better at is predicting which projects could get long-term funding from other funders, and to try to help projects get to a place where they can receive long-term funding from more than just the LTFF.
Capital wise, I also think that we don’t really have the funding to support organizations over longer periods of time. I.e. supporting 3 organizations at $500k a year would take up almost all of our budget, and I think it’s not worth trading that off against the other smaller grants we’ve historically been making. But it is one of the most promising ways I would want to use additional funds we could get.
I agree with @Habryka that our current process is relatively lightweight which is good for small grants but doesn’t provide adequate accountability for large grants. I think I’m more optimistic about the LTFF being able to grow into this role. There’s a reasonable number of people who we might be excited about working as fund managers—the main thing that’s held us back from growing the team is the cost of coordination overhead as you add more individuals. But we could potentially split the fund into two sub-teams that specialize in smaller and larger grants (with different evaluation process), or even create a separate fund in EA Funds that focuses on more established organisations. Nothing certain yet, but it’s a problem we’re interested in addressing.
Ah yeah, I also think that if the opportunity presents itself we could grow into this role a good amount. Though I think on the margin I think it’s more likely we are going to invest even more into more early-stage expertise and maybe do more active early-stage grantmaking.
Just to add a comment with regards to sustainable funding for independent researchers. There haven’t previously been many options available for this, however, there are a growing number of virtual research institutes through which affiliated researchers can apply to academic funding agencies. The virtual institute can then administer the grant for a researcher (usually for much lower overheads than a traditional institution), while they effectively still do independent work. The Ronin Institute administers funding from US granters, and I am a Board member at IGDORE which can receive funding from some European granters. That said, it may still be quite difficult for individuals to secure academic funding without having some traditional academic credentials (PhD, publications, etc.).
What do you mean by “There haven’t previously been many options available”? What is stopping you from just giving people money? Why do you need an institute as middle hand?
My understanding is that (1) to deal with the paperwork etc. for grants from governments or government-like bureaucratic institutions, you need to be part of an institution that’s done it before; (2) if the grantor is a nonprofit, they have regulations about how they can use their money while maintaining nonprofit status, and it’s very easy for them to forward the money to a different nonprofit institution, but may be difficult or impossible for them to forward the money to an individual. If it is possible to just get a check as an individual, I imagine that that’s the best option. Unless there are other considerations I don’t know about.
Btw Theiss is another US organization in this space.
If it is possible to just get a check as an individual, I imagine that that’s the best option.
One other benefit of a virtual research institute is that they can act as formal employers for independent researchers, which may be desirable for things like receiving healthcare coverage or welfare benefits.
Thanks for mentioning Theiss, I didn’t know of them before. Their website doesn’t look so active now, but it’s good to know about the history of the independent research scene.
Theiss was very much active as of December 2020. They’ve just been recruiting so successfully through word-of-mouth that they haven’t gotten around to updating the website.
I don’t think healthcare and taxes undermine what I said, at least not for me personally. For healthcare, individuals can buy health insurance too. For taxes, self-employed people need to pay self-employment tax, but employees and employers both have to pay payroll tax which adds up to a similar amount, and then you lose the QBI deduction (this is all USA-specific), so I think you come out behind even before you account for institutional overhead, and certainly after. Or at least that’s what I found when I ran the numbers for me personally. It may be dependent on income bracket or country so I don’t want to over-generalize...
That’s all assuming that the goal is to minimize the amount of grant money you’re asking for, while holding fixed after-tax take-home pay. If your goal is to minimize hassle, for example, and you can just apply for a bit more money to compensate, then by all means join an institution, and avoid the hassle of having to research health care plans and self-employment tax deductions and so on.
I could be wrong or misunderstanding things, to be clear. I recently tried to figure this out for my own project but might have messed up, and as I mentioned, different income brackets and regions may differ. Happy to talk more. :-)
I am wondering how the fund managers are thinking more long-term about encouraging more independent researchers and projects to come into existence and stay in existence. So far as I can tell, there hasn’t been much renewed granting to independent individuals and projects (i.e. granting for a second or third time to grantees who have previously already received an LTFF grant). Do most grantees have a solid plan for securing funding after their LTFF grant money runs out, and if so what do they tend to do?
I think LTFF is doing something valuable by giving people the freedom to not “sell out” to more traditional or mass-appeal funding sources (e.g. academia, established orgs, Patreon). I’m worried about a situation where receiving a grant from LTFF isn’t enough to be sustainable, so that people go back to doing more “safe” things like working in academia or at an established org.
Any thoughts on this topic?
The LTFF is happy to renew grants so long as the applicant has been making strong progress and we believe working independently continues to be the best option for them. Examples of renewals in this round include Robert Miles, who we first funded in April 2019, and Joe Collman, who we funded in November 2019. In particular, we’d be happy to be the #1 funding source of a new EA org for several years (subject to the budget constraints Oliver mentions in his reply).
Many of the grants we make to individuals are for career transitions, such as someone retraining from one research field to another, or for one-off projects. So I would expect most grants to not be renewals. That said, the bar for renewals does tend to be higher. This is because we pursue a hits-based giving approach, so are willing to fund projects that are likely not to work out—but of course will not want to renew the grant if it is clearly not working.
I think being a risk-tolerant funder is particularly valuable since most employers are, quite rightly, risk-averse. Firing people tends to be harmful to morale; internships or probation periods can help, but take a lot of supervisory time. This means people who might be a great hire but are high-variance often don’t get hired. Funding them for a period of time to do independent work can derisk the grantee, since they’ll have a more substantial portfolio to show.
The level of excitement about long-term independent work varies between fund managers. I tend to think it’s hard for people to do great work independently. I’m still open to funding it, but I want to see a compelling case that there’s not an organisation that would be a good home for the applicant. Some other fund managers are more concerned by perverse incentives in established organisations (especially academia), so are more willing to fund independent research.
I’d be interested to hear thoughts on how we could better support our grantees here. We do sometimes forward applications on to other funders (with the applicants permission), but don’t have any systematic program to secure further funding (beyond applying for renewals). We could try something like “demo days” popular in the VC world, but I’m not sure there’s a large enough ecosystem of potential funders for this to be worth it.
My impression is that it is not possible for everyone who want to help with the long term ti get hired by an org, for the simple reason that there are not enough openings at those orgs. At least in AI Safety, all entry level jobs are very competitive, meaning that not getting in is not a strong signal that one could not have done well there.
Do you disagree with this?
I can’t respond for Adam, but just wanted to say that I personally agree with you, which is one of the reasons I’m currently excited about funding independent work.
Thanks for picking up the thread here Asya! I think I largely agree with this, especially about the competitiveness in this space. For example, with AI PhD applications, I often see extremely talented people get rejected who I’m sure would have got an offer a few years ago.
I’m pretty happy to see the LTFF offering effectively “bridge” funding for people who don’t quite meet the hiring bar yet, but I think are likely to in the next few years. However, I’d be hesitant about heading towards a large fraction of people working independently long-term. I think there’s huge advantages from the structure and mentorship an org can provide. If orgs aren’t scaling up fast enough, then I’d prefer to focus on trying to help speed that up.
The main way I could see myself getting more excited about long-term independent research is if we saw flourishing communities forming amongst independent researchers. Efforts like LessWrong and the Alignment Forum help in terms of providing infrastructure. But right now it still seems much worse than working for an org, especially if you want to go down any of the more traditional career paths later. But I’d love to be proven wrong here.
I claim we have proof of concept. The people who started the existing AI Safety research orgs did not have AI Safety mentors. Current independent researcher have more support than they had. In a way an org is just a crystalized collaboration of previously independent researchers.
I think that there are some PR reasons why it would be good if most AI Safety researchers where part of academia or other respectable orgs (e.g. DeepMind). But I also think it is good to have a minority of researchers who are disconnected from the particular pressures of that environment.
However, being part of academia is not the same as being part of an AI Safety org. MIRI people are not part of academia, and someone doing AI Safety research as part of their PhD in a “normal” (not AI Safety focused) PhD program, is sorta an independent researcher.
We are working on that. I’m not optimistic about current orgs keeping up with the growth of the field, and I don’t think it is healthy for the career to be too competitive, since this will lead to goodhearted on career intensives. But I do think a looser structure, built on personal connections rather than formal org employment, can grow in a much more flexible way, and we are experimenting with various methods to make this happen.
Yeah, I am also pretty worried about this. I don’t think we’ve figured out a great solution to this yet.
I think we don’t really have sufficient capacity to evaluate organizations on an ongoing basis and provide good accountability. Like, if a new organization were to be funded by us and then grow to a budget of $1M a year, I don’t feel like we have the capacity to evaluate their output and impact sufficiently well to justify giving them $1M each year (or even just $500k).
Our current evaluation process routes feels pretty good for smaller projects, and granting to established organizations that have other active evaluators looking into them that we can talk to, but doesn’t feel very well-suited to larger organizations that don’t have existing evaluations done on them (there is a lot of due diligence work to be done on that I think requires higher staff capacity than we have).
I also think the general process of the LTFF specializing into more something like venture funding, with other funders stepping in for more established organizations feels pretty good to me. I do think the current process has a lot of unnecessary uncertainty and risk in it, and I would like to work on that. So one thing I’ve been trying to get better at is predicting which projects could get long-term funding from other funders, and to try to help projects get to a place where they can receive long-term funding from more than just the LTFF.
Capital wise, I also think that we don’t really have the funding to support organizations over longer periods of time. I.e. supporting 3 organizations at $500k a year would take up almost all of our budget, and I think it’s not worth trading that off against the other smaller grants we’ve historically been making. But it is one of the most promising ways I would want to use additional funds we could get.
I agree with @Habryka that our current process is relatively lightweight which is good for small grants but doesn’t provide adequate accountability for large grants. I think I’m more optimistic about the LTFF being able to grow into this role. There’s a reasonable number of people who we might be excited about working as fund managers—the main thing that’s held us back from growing the team is the cost of coordination overhead as you add more individuals. But we could potentially split the fund into two sub-teams that specialize in smaller and larger grants (with different evaluation process), or even create a separate fund in EA Funds that focuses on more established organisations. Nothing certain yet, but it’s a problem we’re interested in addressing.
Ah yeah, I also think that if the opportunity presents itself we could grow into this role a good amount. Though I think on the margin I think it’s more likely we are going to invest even more into more early-stage expertise and maybe do more active early-stage grantmaking.
Just to add a comment with regards to sustainable funding for independent researchers. There haven’t previously been many options available for this, however, there are a growing number of virtual research institutes through which affiliated researchers can apply to academic funding agencies. The virtual institute can then administer the grant for a researcher (usually for much lower overheads than a traditional institution), while they effectively still do independent work. The Ronin Institute administers funding from US granters, and I am a Board member at IGDORE which can receive funding from some European granters. That said, it may still be quite difficult for individuals to secure academic funding without having some traditional academic credentials (PhD, publications, etc.).
What do you mean by “There haven’t previously been many options available”? What is stopping you from just giving people money? Why do you need an institute as middle hand?
My understanding is that (1) to deal with the paperwork etc. for grants from governments or government-like bureaucratic institutions, you need to be part of an institution that’s done it before; (2) if the grantor is a nonprofit, they have regulations about how they can use their money while maintaining nonprofit status, and it’s very easy for them to forward the money to a different nonprofit institution, but may be difficult or impossible for them to forward the money to an individual. If it is possible to just get a check as an individual, I imagine that that’s the best option. Unless there are other considerations I don’t know about.
Btw Theiss is another US organization in this space.
One other benefit of a virtual research institute is that they can act as formal employers for independent researchers, which may be desirable for things like receiving healthcare coverage or welfare benefits.
Thanks for mentioning Theiss, I didn’t know of them before. Their website doesn’t look so active now, but it’s good to know about the history of the independent research scene.
Theiss was very much active as of December 2020. They’ve just been recruiting so successfully through word-of-mouth that they haven’t gotten around to updating the website.
I don’t think healthcare and taxes undermine what I said, at least not for me personally. For healthcare, individuals can buy health insurance too. For taxes, self-employed people need to pay self-employment tax, but employees and employers both have to pay payroll tax which adds up to a similar amount, and then you lose the QBI deduction (this is all USA-specific), so I think you come out behind even before you account for institutional overhead, and certainly after. Or at least that’s what I found when I ran the numbers for me personally. It may be dependent on income bracket or country so I don’t want to over-generalize...
That’s all assuming that the goal is to minimize the amount of grant money you’re asking for, while holding fixed after-tax take-home pay. If your goal is to minimize hassle, for example, and you can just apply for a bit more money to compensate, then by all means join an institution, and avoid the hassle of having to research health care plans and self-employment tax deductions and so on.
I could be wrong or misunderstanding things, to be clear. I recently tried to figure this out for my own project but might have messed up, and as I mentioned, different income brackets and regions may differ. Happy to talk more. :-)
+1.