Program Associate at Open Philanthropy and chair of the Long-Term Future Fund. I spend half my time on AI and half my time on EA community-building. Any views I express on the forum are my own, not the views of my employer.
abergal
Karma: 2,011
Great post! I most agree with that we should be more clear that things are still very, very uncertain. I think there are several factors that push against this:
The EA community and discourse doesn’t have any formal structure for propagating ideas, unlike academia. You are likely to hear about something if it’s already popular. Critical or new posts and ideas are unpopular by definition to begin with, so they fall by the wayside.
The story for impact for many existing EA organizations often relies on a somewhat narrow worldview. It does seem correct to me that we should both be trying to figure out the truth and taking bets on worlds where we have a lot of important things to do right now. But it’s easy to mentally conflate “taking an important bet” and “being confident that this is what the world looks like”, both from inside and outside an organization. I personally try to pursue a mixed strategy, where I take some actions assuming a particular worldview where I have a lot of leverage now, and some actions trying to get at the truth. But it’s kind of a weird mental state to hold, and I assume most EAs don’t have enough career flexibility to do this.
I do think that the closer you get to people doing direct work, the more people are skeptical and consider alternative views. I think the kind of deference you talk about in this post is much more common among people who are less involved with the movement.
That being said, it’s not great that the ideas that newcomers and people who aren’t in the innermost circles see are not the best representatives of the truth or of the amount of uncertainty involved. I’m interested in trying to think of ways to fix that—like I said, I think it’s hard because there are lots of different channels and no formal mechanism for what ideas “the movement” is exposed to. Without formal mechanisms, it seems hard to leave an equilibrium where a small number of reputable people or old but popular works of literature have disproportionate influence.
That being said, I really appreciate a lot of recent attempts by people to express uncertainty more publically—see e.g. Ben’s podcast, Will’s talk, 80K’s recent posts, my talk and interviews. For better or for worse, it does seem like a small number of individuals have disproportionate influence over the discourse, and as such I think they do have some responsibility to convey uncertainty in a thoughtful way.
Hi—great post! I was pointed to this because I’ve been working on a variety of hardware-related projects at FHI and AI Impacts, including generating better hardware forecasts. (I wrote a lot here, but would also be excited to talk to you directly and have even more to say—I contacted you through Facebook.)
At first glance, it seemed to me that the existence of Ajeya’s report demonstrates that the EA community already has enough people with sufficient knowledge and access to expert opinion that, on the margin, adding one expert in hardware to the EA community wouldn’t improve these forecasts much.
I think this isn’t true.
For one, I think while the forecasts in that report are the best publicly available thing we have, there’s significant room to do better, e.g.
The forecasts rely on data for the sale price of hardware along with their reported FLOPS performance. But the sale price is only one component of the costs to run hardware and doesn’t include power, data center costs, storage and networking etc. Arguably, we also care about the price performance for large hardware producers (e.g. Google) more than hardware consumers, and the sale price won’t necessarily be reflective of that since it includes a significant mark-up over the cost of manufacture.
The forecasts don’t consider existing forecasts from e.g. the IRDS that you mention, which are actually very pessimistic about the scaling of energy costs for CMOS chips over the next 15 years. (Of course, this doesn’t preclude better scaling through switching to other technology).
If I recall correctly, the report partially justifies its estimate by guessing that even if chip design improvements bottom out, improvements in manufacturing cost and chip lifetime might still create a relatively steady rate of progress. I think this requires some assumptions about the cost model that may not be true, though I haven’t done enough investigation yet to be sure.
(This isn’t to disparage the report—I think it’s an awesome report and the current estimate is a great starting point, and Ajeya very explicitly disclaims that these are the forecasts most likely to be knowably mistaken.)
As a side note, I think EAs tend to misuse and misunderstand Moore’s Law in general. As you say, Moore’s Law says that the number of transistors on a chip doubles every two years. This has remained true historically, but is only dubiously correlated with ‘price performance Moore’s Law’—a doubling of price performance every two years. As I note above, I think the data publicly collected on price performance is poor, partially because the ‘price’ and ‘performance’ of hardware is trickier to define than it looks. But e.g. this recent paper estimates that the price performance of at least universal processors has slowed considerably in recent years (the paper estimates 8% improvement in performance-per-dollar annually from 2008 − 2013, see section 4.3.2 ‘Current state of performance-per-dollar of universal processors’). Even if price performance Moore’s Law ever held true, it’s really not clear that it holds now.
For two, I think it’s not the case that we have access to enough people with sufficient knowledge and expert opinion. I’ve been really interested in talking to hardware experts, and I think I would selfishly benefit substantially from experts who had thought more about “the big picture” or more speculative hardware possibilities (most people I talk to have domain expertise in something very specific and near-term). I’ve also found it difficult to get a lot of people’s time, and would selfishly benefit from having access more hardware experts that were explicitly longtermist-aligned and excited to give me more of it. :) Basically, I’d be very in favor of having more people in industry available as advisors, as you suggest.
You also touch on this some, but I will say that I do think now is actually a particularly impactful time to influence policy on the company-level (in addition to in government, which seems to be implementing a slew of new semiconductor legislation and seems increasingly interested in regulating hardware companies.) A recent report estimates that ASICs are poised to take over 50% of the hardware market in the coming years, and most ASIC companies now are small start-ups—I think there’s a case that influencing the policy and ethics of these small companies is much more tractable than their larger counterparts, and it would be worth someone thinking carefully about how to do that. Working as an early employee seems like a good potential way.
Lastly, I will say that I think there might be valuable work to be done at the intersection of hardware and economics—for an example, see again this paper. I think things like understanding models of hardware costs, the overall hardware market, cloud computing, etc. are not well-encapsulated by the kind of understanding technical experts tend to have and is valuable for the longtermist community to have access to. (This is also some of what I’ve been working on locally.)
Sadly, I think those changes would in fact be fairly large and would take up a lot of fund manager time. I think small modifications to original proposals wouldn’t be enough, and it would require suggesting new projects or assessing applicants holistically and seeing if a career change made sense.
In my mind, this relates to ways in which mentorship is a bottleneck in longtermist work right now-- there are probably lots of people who could be doing useful direct work, but they would require resources and direction that we as a community don’t have the capacity for. I don’t think the LTFF is well-placed to provide this kind of mentorship, though we do offer to give people one-off feedback on their applications.
This round, we switched from a system where we had all the grant discussion in a single spreadsheet to one where we discuss each grant in a separate Google doc, linked from a single spreadsheet. One fund manager has commented that they feel less on-top of this grant round than before as a result. (We’re going to rethink this system again for next grant round.) We also changed the fund composition a bunch—Helen and Matt left, I became chair, and three new guest managers joined. A priori, this could cause a shift in standards, though I have no particular reason to think it would shift them downward.
I personally don’t think the standards have fallen because I’ve been keeping close track of all the grants and feel like I have a good model of the old fund team (and in some cases, have asked them directly for advice). I think the old team would have made similar decisions to the ones we’re making on this set of applications. It’s possible there would have been a few differences, but not enough to explain a big change in spending.
[I work at Open Philanthropy] Hi Linda–-- thanks for flagging this. After checking internally, I’m not sure what project you’re referring to here; generally speaking, I agree with you/others in this thread that it’s not good to fully funge against incoming funds from other grantmakers in the space after agreeing to fund something, but I’d want to have more context on the specifics of the situation.
It totally makes sense that you don’t want to name the source or project, but if you or your source would feel comfortable sharing more information, feel free to DM me or ask your source to DM me (or use Open Phil’s anonymous feedback form). (And just to flag explicitly, we would/do really appreciate this kind of feedback.)
Filtering for obvious misfits, I think the majority reason is that I don’t think the project proposal will be sufficiently valuable for the long-term future, even if executed well. The minority reason is that there isn’t strong enough evidence that the project will be executed well.
Sorry if this is an unsatisfying answer—I think our applications are different enough that it’s hard to think of common reasons for rejection that are more granular. Also, often the bottom line is “this seems like it could be good, but isn’t as good as other things we want to fund”. Here are some more concrete kinds of reasons that I think have come up at least more than once:
Project seems good for the medium-term future, but not for the long-term future
Applicant wants to learn the answer to X, but X doesn’t seem like an important question to me
Applicant wants to learn about X via doing Y, but I think Y is not a promising approach for learning about X
Applicant proposes a solution to some problem, but I think the real bottleneck in the problem lies elsewhere
Applicant wants to write something for a particular audience, but I don’t think that writing will be received well by that audience
Project would be good if executed exceptionally well, but applicant doesn’t have a track record in this area, and there are no references that I trust to be calibrated to vouch for their ability
Applicant wants to do research on some topic, but their previous research on similar topics doesn’t seem very good
Applicant wants money to do movement-building, but several people have reported negative interactions with them
I’m commenting here to say that while I don’t plan to participate in public discussion of the FTX situation imminently (for similar reasons to the ones Holden gives above, though I don’t totally agree with some of Holden’s explanations here, and personally put more weight on some considerations here than others), I am planning to do so within the next several months. I’m sorry for how frustrating that is, though I endorse my choice.
We got feedback from several people that they weren’t applying to the funds because they didn’t want to have a public report. There are lots of reasons that I sympathize with for not wanting a public report, especially as an individual (e.g. you’re worried about it affecting future job prospects, you’re asking for money for mental health support and don’t want that to be widely known, etc.). My vision (at least for the Long-Term Future Fund) is to become a good default funding source for individuals and new organizations, and I think that vision is compromised if some people don’t want to apply for publicity reasons.
Broadly, I think the benefits to funding more people outweigh the costs to transparency.
Really good question!
We currently have ~$315K in the fund balance.* My personal median guess is that we could use $2M over the next year while maintaining this year’s bar for funding. This would be:
$1.7M more than our current balance
$500K more per year than we’ve spent in previous years
$800K more than the total amount of donations received in 2020 so far
$400K more than a naive guess for what the total amount of donations received will be in all of 2020. (That is, if we wanted a year of donations to pay for a year of funding, we would need $400K more in donations next year than what we got this year.)
Reasoning below:
Generally, we fund anything above a certain bar, without accounting explicitly for the amount of money we have. According to this policy, for the last two years, the fund has given out ~$1.5M per year, or ~$500K per grant round, and has not accumulated a significant buffer.
This round had an unusually large number of high-quality applicants. We spent $500K, but we pushed two large grant decisions to our next payout round, and several of our applicants happened to receive money from another source just before we communicated our funding decision. This makes me think that if this increase in high-quality applicants persists, it would be reasonable to have $600K - $700K per grant round, for a total of ~$2M over the next year.
My personal guess is that the increase in high-quality applications will persist, and I’m somewhat hopeful that we will get even more high-quality applications, via a combination of outreach and potentially some active grantmaking. This makes me think that $2M over the next year would be reasonable for not going below the ROI on the last marginal dollar of the grants we made this year, though I’m not certain. (Of the two other fund managers who have made quantitative guesses on this so far, one fund manager also had $2M as their median guess, while another thought slightly above $1.5M was more likely.)
I also think there’s a reasonable case for having slightly more than our median guess available in the fund. This would both act as a buffer in case we end up with more grants above our current bar than expected, and would let us proactively encourage potential grantees to apply for funding without being worried that we’ll run out of money.
If we got much more money than applications that meet our current bar, we would let donors know. I think we would also consider lowering our bar for funding, though this would only happen after checking in with the largest donors.
* This is less than the amount displayed in our fund page, which is still being updated with our latest payouts.
FWIW, I think this kind of questioning is fairly Habryka-specific and not really standard for our policy applicants; I think in many cases I wouldn’t expect that it would lead to productive discussions (and in fact could be counterproductive, in that it might put off potential allies who we might want to work with later).
I make the calls on who is the primary evaluator for which grants; as Habryka said, I think he is probably most skeptical of policy work among people on the LTFF, and hasn’t been the primary evaluator for almost any (maybe none?) of the policy-related grants we’ve had. In your case, I thought it was unusually likely that a discussion between you and Habryka would be productive and helpful for my evaluation of the grant (though I was interested primarily in different but related questions, not “whether policy work as a whole is competitive with other grants”), because I generally expect people more embedded in the community (and in the case above, you (Sam) in particular, which I really appreciate), to be more open to pretty frank discussions about the effectiveness of particular plans, lines of work, etc.
(These are more additional considerations, not intended to be counterarguments given that your post itself was mostly pointing at additional considerations.)
Objection.
The longtermism community can enjoy above-average growth for only a finite window of time. (It can at most control all resources, after which its growth equals average growth.)
Thus, spending money on growing the longtermism community now rather than later merely moves a transient window of additional resource growth to an earlier point in time.
We should be indifferent about the timing of benefits, so this effect doesn’t matter. On the other hand, by waiting one year we can earn positive expected returns by (e.g.) investing into the stock market.
To sum up, giving later rather than now has two effects: (1) moving a fixed window of additional growth around in time and (2) leaving us more time to earn positive financial returns. The first effect is neutral, the second is positive. Thus, overall, giving later is always better.
Given that longtermists are generally concerned with trajectory changes, controlling all resources seems like we’ve largely ‘won’, and having more financial returns on top of this seems fairly negligible by comparison. In many cases I’d gladly trade absolute financial returns for controlling a greater fraction of the world’s resources sooner.Second, the target you need to hit is arguably pretty narrow. The objection only applies conclusively to things that basically create cause-agnostic, transferable resources that are allocated at least as well as if allocated by your future self. If resources are tied to a particular cause area, are not transferable, or are more poorly allocated, they count less.
Speaking to movement-building as an alternative to financial investment in particular:
It feels to me like quality-adjusted longtermists are more readily transferable and cause-agnostic than money on short time-scales, in the sense that they can either earn money or do direct work, and at least right now, we seem to be having trouble effectively turning money into direct work.
It’s definitely a lot less clear whether there’s a compounding effect to longtermists and how readily they can be transferred into the longer-term future. For what it’s worth, I’d guess there is such a compounding effect, and they can be transferred, especially given historical evidence of transfer of values between generations. Whether this is true / consistent / competitive with stock market returns is definitely debateable and a matter of ‘messy empirics’, though.
Fund managers can now opt to be compensated as contractors, at a rate of $40 / hour.
I’d overall like to see more work that has a solid longtermist justification but isn’t as close to existing longtermist work. It seems like the LTFF might be well-placed to encourage this, since we provide funding outside of established orgs. This round, we received many applications from people who weren’t very engaged with the existing longtermist community. While these didn’t end up meeting our bar, some of the projects were fairly novel and good enough to make me excited about funding people like this in general.
There are also lots of particular less-established directions where I’d personally be interested in seeing more work, e.g.:
Work on structured transparency tools for detecting risks from rogue actors
Work on information security’s effect on AI development
Work on the offense—defense balance in a world with many advanced AI systems
Work on the likelihood and moral value of extraterrestrial life
Work on increasing institutional competence, particularly around existential risk mitigation
Work on effectively spreading longtermist values outside of traditional movement-building
These are largely a reflection of what I happen to have been thinking about recently and definitely not my fully-endorsed answer to this question—I’d like to spend time talking to others and coming to more stable conclusions about specific work the LTFF should encourage more of.
There’s no strict ‘minimum number’—sometimes the grant is clearly above or below our bar and we don’t consult anyone, and sometimes we’re really uncertain or in disagreement, and we end up consulting lots of people (I think some grants have had 5+).
I will also say that each fund is somewhat intentionally composed of fund managers with somewhat varying viewpoints who trust different sets of experts, and the voting structure is such that if any individual fund manager is really excited about an application, it generally gets funded. As a result, I think in practice, there’s more diversity in what gets funded than you might expect from a single grantmaking body, and there’s less risk that you won’t get funded just because a particular person dislikes you.
Like Adam, I’ll focus on things that someone reading this might be interested in supporting or applying for. I want to emphasize that this is my personal take, not representing the whole fund, and I would be sad if this response stopped anyone from applying—there’s a lot of healthy disagreement within the fund, and we fund lots of things where at least one person thinks it’s below our bar. I also think a well-justified application could definitely change my mind.
Improving science or technology, unless there’s a strong case that the improvement would differentially benefit existential risk mitigation (or some other aspect of our long-term trajectory). As Ben Todd explains here, I think this is unlikely to be as highly-leveraged for improving the long-term future as trajectory changing efforts. I don’t think there’s a strong case that generally speeding up economic growth is an effective existential risk intervention.
Climate change mitigation. From the evidence I’ve seen, I think climate change is unlikely to be either directly existentially threatening or a particularly highly-leveraged existential risk factor. (It’s also not very neglected.) But I could be excited about funding research work that changed my mind about this.
Most self-improvement / community-member-improvement type work, e.g. “I want to create materials to help longtermists think better about their personal problems.” I’m not universally unexcited about funding this, and there are people who I think do good work like this, but my overall prior is that proposals here won’t be very good.
I am also unexcited about the things Adam wrote.
Rebecca Kagan is currently working as a fund manager for us (sorry for the not-up-to-date webpage).
I think we probably will seek out funding from larger institutional funders if our funding gap persists. We actually just applied for a ~$1M grant from the Survival and Flourishing Fund.
FWIW I had a similar initial reaction to Sophia, though reading more carefully I totally agree that it’s more reasonable to interpret your comment as a reaction to the newsletter rather than to the proposal. I’d maybe add an edit to your high-level comment just to make sure people don’t get confused?
Hey! I definitely don’t expect people starting AI safety research to have a track record doing AI safety work—in fact, I think some of our most valuable grants are paying for smart people to transition into AI safety from other fields. I don’t know the details of your situation, but in general I don’t think “former physics student starting AI safety work” fits into the category of “project would be good if executed exceptionally well”. In that case, I think most of the value would come from supporting the transition of someone who could potentially be really good, rather than from the object-level work itself.
In the case of other technical Ph.D.s, I generally check whether their work is impressive in the context of their field, whether their academic credentials are impressive, what their references have to say. I also place a lot of weight on whether their proposal makes sense and shows an understanding of the topic, and on my own impressions of the person after talking to them.
I do want to emphasize that “paying a smart person to test their fit for AI safety” is a really good use of money from my perspective—if the person turns out to be good, I’ve in some sense paid for a whole lifetime of high-quality AI safety research. So I think my bar is not as high as it is when evaluating grant proposals for object-level work from people I already know.
Hey, Sam – first, thanks for taking the time to write this post, and running it by us. I’m a big fan of public criticism, and I think people are often extra-wary of criticizing funders publicly, relative to other actors of the space.
Some clarifications on what we have and haven’t funded:
I want to make a distinction between “grants that work on policy research” and “grants that interact with policymakers”.
I think our bar for projects that involve the latter is much higher than for projects that are just doing the former.
I think we regularly fund “grants that work on policy research” – e.g., we’ve funded the Centre for Governance of AI, and regularly fund individuals who are doing PhDs or otherwise working on AI governance research.
I think we’ve funded a very small number of grants that involve interactions with policymakers – I can think of three such grants in the last year, two of which were for new projects. (In one case, the grantee has requested that we not report the grant publicly).
Responding to the rest of the post:
I think it’s roughly correct that I have a pretty high bar for funding projects that interact with policymakers, and I endorse this policy. (I don’t want to speak for the Long-Term Future Fund as a whole, because it acts more like a collection of fund managers than a single entity, but I suspect many others on the fund also have a high bar, and that my opinion in particular has had a big influence on our past decisions.)
Some other things in your post that I think are roughly true:
Previous experience in policy has been an important factor in my evaluations of these grants, and all else equal I think I am much more likely to fund applicants who are more senior (though I think the “20 years experience” bar is too high).
There have been cases where we haven’t funded projects (more broadly than in policy) because an individual has given us information about or impressions of them that led us to think the project would be riskier or less impactful than we initially believed, and we haven’t shared the identity or information with the applicant to preserve the privacy of the individual.
We have a higher bar for funding organizations than other projects, because they are more likely to stick around even if we decide they’re not worth funding in the future.
When evaluating the more borderline grants in this space, I often ask and rely heavily on the advice of others working in the policy space, weighted by how much I trust their judgment. I think this is basically a reasonable algorithm to follow, given that (a) they have a lot of context that I don’t, and (b) I think the downside risks of poorly-executed policy projects have spillover effects to other policy projects, which means that others in policy are genuine stakeholders in these decisions.
That being said, I think there’s a surprising amount of disagreement in what projects others in policy think are good, so I think the particular choice of advisors here makes a big difference.
I do think projects interacting with policymakers have substantial room for downside, including:
Pushing policies that are harmful
Making key issues partisan
Creating an impression (among policymakers or the broader world) that people who care about the long-term future are offputting, unrealistic, incompetent, or otherwise undesirable to work with
“Taking up the space” such that future actors who want to make long-term future-focused asks are encouraged or expected to work through or coordinate with the existing project
I suspect we also differ in our views of the upsides of some of this work– a lot of the projects we’ve rejected have wanted to do AI-focused policy work, and I tend to think that we don’t have very good concrete asks for policymakers in this space.