Program Associate at Open Philanthropy and chair of the Long-Term Future Fund. I spend half my time on AI and half my time on EA community-building. Any views I express on the forum are my own, not the views of my employer.
abergal
Reflections on my time on the Long-Term Future Fund
Open Philanthropy is seeking proposals for outreach projects
The Long-Term Future Fund has room for more funding, right now
Long-Term Future Fund: May 2021 grant recommendations
[AMA] Announcing Open Phil’s University Group Organizer and Century Fellowships
Long-Term Future Fund: April 2023 grant recommendations
Long-Term Future Fund: July 2021 grant recommendations
Public reports are now optional for EA Funds grantees
Long-Term Future Fund: December 2021 grant recommendations
Great post! I most agree with that we should be more clear that things are still very, very uncertain. I think there are several factors that push against this:
The EA community and discourse doesn’t have any formal structure for propagating ideas, unlike academia. You are likely to hear about something if it’s already popular. Critical or new posts and ideas are unpopular by definition to begin with, so they fall by the wayside.
The story for impact for many existing EA organizations often relies on a somewhat narrow worldview. It does seem correct to me that we should both be trying to figure out the truth and taking bets on worlds where we have a lot of important things to do right now. But it’s easy to mentally conflate “taking an important bet” and “being confident that this is what the world looks like”, both from inside and outside an organization. I personally try to pursue a mixed strategy, where I take some actions assuming a particular worldview where I have a lot of leverage now, and some actions trying to get at the truth. But it’s kind of a weird mental state to hold, and I assume most EAs don’t have enough career flexibility to do this.
I do think that the closer you get to people doing direct work, the more people are skeptical and consider alternative views. I think the kind of deference you talk about in this post is much more common among people who are less involved with the movement.
That being said, it’s not great that the ideas that newcomers and people who aren’t in the innermost circles see are not the best representatives of the truth or of the amount of uncertainty involved. I’m interested in trying to think of ways to fix that—like I said, I think it’s hard because there are lots of different channels and no formal mechanism for what ideas “the movement” is exposed to. Without formal mechanisms, it seems hard to leave an equilibrium where a small number of reputable people or old but popular works of literature have disproportionate influence.
That being said, I really appreciate a lot of recent attempts by people to express uncertainty more publically—see e.g. Ben’s podcast, Will’s talk, 80K’s recent posts, my talk and interviews. For better or for worse, it does seem like a small number of individuals have disproportionate influence over the discourse, and as such I think they do have some responsibility to convey uncertainty in a thoughtful way.
Movement building and investing to give later
Hi—great post! I was pointed to this because I’ve been working on a variety of hardware-related projects at FHI and AI Impacts, including generating better hardware forecasts. (I wrote a lot here, but would also be excited to talk to you directly and have even more to say—I contacted you through Facebook.)
At first glance, it seemed to me that the existence of Ajeya’s report demonstrates that the EA community already has enough people with sufficient knowledge and access to expert opinion that, on the margin, adding one expert in hardware to the EA community wouldn’t improve these forecasts much.
I think this isn’t true.
For one, I think while the forecasts in that report are the best publicly available thing we have, there’s significant room to do better, e.g.
The forecasts rely on data for the sale price of hardware along with their reported FLOPS performance. But the sale price is only one component of the costs to run hardware and doesn’t include power, data center costs, storage and networking etc. Arguably, we also care about the price performance for large hardware producers (e.g. Google) more than hardware consumers, and the sale price won’t necessarily be reflective of that since it includes a significant mark-up over the cost of manufacture.
The forecasts don’t consider existing forecasts from e.g. the IRDS that you mention, which are actually very pessimistic about the scaling of energy costs for CMOS chips over the next 15 years. (Of course, this doesn’t preclude better scaling through switching to other technology).
If I recall correctly, the report partially justifies its estimate by guessing that even if chip design improvements bottom out, improvements in manufacturing cost and chip lifetime might still create a relatively steady rate of progress. I think this requires some assumptions about the cost model that may not be true, though I haven’t done enough investigation yet to be sure.
(This isn’t to disparage the report—I think it’s an awesome report and the current estimate is a great starting point, and Ajeya very explicitly disclaims that these are the forecasts most likely to be knowably mistaken.)
As a side note, I think EAs tend to misuse and misunderstand Moore’s Law in general. As you say, Moore’s Law says that the number of transistors on a chip doubles every two years. This has remained true historically, but is only dubiously correlated with ‘price performance Moore’s Law’—a doubling of price performance every two years. As I note above, I think the data publicly collected on price performance is poor, partially because the ‘price’ and ‘performance’ of hardware is trickier to define than it looks. But e.g. this recent paper estimates that the price performance of at least universal processors has slowed considerably in recent years (the paper estimates 8% improvement in performance-per-dollar annually from 2008 − 2013, see section 4.3.2 ‘Current state of performance-per-dollar of universal processors’). Even if price performance Moore’s Law ever held true, it’s really not clear that it holds now.
For two, I think it’s not the case that we have access to enough people with sufficient knowledge and expert opinion. I’ve been really interested in talking to hardware experts, and I think I would selfishly benefit substantially from experts who had thought more about “the big picture” or more speculative hardware possibilities (most people I talk to have domain expertise in something very specific and near-term). I’ve also found it difficult to get a lot of people’s time, and would selfishly benefit from having access more hardware experts that were explicitly longtermist-aligned and excited to give me more of it. :) Basically, I’d be very in favor of having more people in industry available as advisors, as you suggest.
You also touch on this some, but I will say that I do think now is actually a particularly impactful time to influence policy on the company-level (in addition to in government, which seems to be implementing a slew of new semiconductor legislation and seems increasingly interested in regulating hardware companies.) A recent report estimates that ASICs are poised to take over 50% of the hardware market in the coming years, and most ASIC companies now are small start-ups—I think there’s a case that influencing the policy and ethics of these small companies is much more tractable than their larger counterparts, and it would be worth someone thinking carefully about how to do that. Working as an early employee seems like a good potential way.
Lastly, I will say that I think there might be valuable work to be done at the intersection of hardware and economics—for an example, see again this paper. I think things like understanding models of hardware costs, the overall hardware market, cloud computing, etc. are not well-encapsulated by the kind of understanding technical experts tend to have and is valuable for the longtermist community to have access to. (This is also some of what I’ve been working on locally.)
Sadly, I think those changes would in fact be fairly large and would take up a lot of fund manager time. I think small modifications to original proposals wouldn’t be enough, and it would require suggesting new projects or assessing applicants holistically and seeing if a career change made sense.
In my mind, this relates to ways in which mentorship is a bottleneck in longtermist work right now-- there are probably lots of people who could be doing useful direct work, but they would require resources and direction that we as a community don’t have the capacity for. I don’t think the LTFF is well-placed to provide this kind of mentorship, though we do offer to give people one-off feedback on their applications.
This round, we switched from a system where we had all the grant discussion in a single spreadsheet to one where we discuss each grant in a separate Google doc, linked from a single spreadsheet. One fund manager has commented that they feel less on-top of this grant round than before as a result. (We’re going to rethink this system again for next grant round.) We also changed the fund composition a bunch—Helen and Matt left, I became chair, and three new guest managers joined. A priori, this could cause a shift in standards, though I have no particular reason to think it would shift them downward.
I personally don’t think the standards have fallen because I’ve been keeping close track of all the grants and feel like I have a good model of the old fund team (and in some cases, have asked them directly for advice). I think the old team would have made similar decisions to the ones we’re making on this set of applications. It’s possible there would have been a few differences, but not enough to explain a big change in spending.
How to change minds
[I work at Open Philanthropy] Hi Linda–-- thanks for flagging this. After checking internally, I’m not sure what project you’re referring to here; generally speaking, I agree with you/others in this thread that it’s not good to fully funge against incoming funds from other grantmakers in the space after agreeing to fund something, but I’d want to have more context on the specifics of the situation.
It totally makes sense that you don’t want to name the source or project, but if you or your source would feel comfortable sharing more information, feel free to DM me or ask your source to DM me (or use Open Phil’s anonymous feedback form). (And just to flag explicitly, we would/do really appreciate this kind of feedback.)
Filtering for obvious misfits, I think the majority reason is that I don’t think the project proposal will be sufficiently valuable for the long-term future, even if executed well. The minority reason is that there isn’t strong enough evidence that the project will be executed well.
Sorry if this is an unsatisfying answer—I think our applications are different enough that it’s hard to think of common reasons for rejection that are more granular. Also, often the bottom line is “this seems like it could be good, but isn’t as good as other things we want to fund”. Here are some more concrete kinds of reasons that I think have come up at least more than once:
Project seems good for the medium-term future, but not for the long-term future
Applicant wants to learn the answer to X, but X doesn’t seem like an important question to me
Applicant wants to learn about X via doing Y, but I think Y is not a promising approach for learning about X
Applicant proposes a solution to some problem, but I think the real bottleneck in the problem lies elsewhere
Applicant wants to write something for a particular audience, but I don’t think that writing will be received well by that audience
Project would be good if executed exceptionally well, but applicant doesn’t have a track record in this area, and there are no references that I trust to be calibrated to vouch for their ability
Applicant wants to do research on some topic, but their previous research on similar topics doesn’t seem very good
Applicant wants money to do movement-building, but several people have reported negative interactions with them
I’m commenting here to say that while I don’t plan to participate in public discussion of the FTX situation imminently (for similar reasons to the ones Holden gives above, though I don’t totally agree with some of Holden’s explanations here, and personally put more weight on some considerations here than others), I am planning to do so within the next several months. I’m sorry for how frustrating that is, though I endorse my choice.
We got feedback from several people that they weren’t applying to the funds because they didn’t want to have a public report. There are lots of reasons that I sympathize with for not wanting a public report, especially as an individual (e.g. you’re worried about it affecting future job prospects, you’re asking for money for mental health support and don’t want that to be widely known, etc.). My vision (at least for the Long-Term Future Fund) is to become a good default funding source for individuals and new organizations, and I think that vision is compromised if some people don’t want to apply for publicity reasons.
Broadly, I think the benefits to funding more people outweigh the costs to transparency.
Hey, Sam – first, thanks for taking the time to write this post, and running it by us. I’m a big fan of public criticism, and I think people are often extra-wary of criticizing funders publicly, relative to other actors of the space.
Some clarifications on what we have and haven’t funded:
I want to make a distinction between “grants that work on policy research” and “grants that interact with policymakers”.
I think our bar for projects that involve the latter is much higher than for projects that are just doing the former.
I think we regularly fund “grants that work on policy research” – e.g., we’ve funded the Centre for Governance of AI, and regularly fund individuals who are doing PhDs or otherwise working on AI governance research.
I think we’ve funded a very small number of grants that involve interactions with policymakers – I can think of three such grants in the last year, two of which were for new projects. (In one case, the grantee has requested that we not report the grant publicly).
Responding to the rest of the post:
I think it’s roughly correct that I have a pretty high bar for funding projects that interact with policymakers, and I endorse this policy. (I don’t want to speak for the Long-Term Future Fund as a whole, because it acts more like a collection of fund managers than a single entity, but I suspect many others on the fund also have a high bar, and that my opinion in particular has had a big influence on our past decisions.)
Some other things in your post that I think are roughly true:
Previous experience in policy has been an important factor in my evaluations of these grants, and all else equal I think I am much more likely to fund applicants who are more senior (though I think the “20 years experience” bar is too high).
There have been cases where we haven’t funded projects (more broadly than in policy) because an individual has given us information about or impressions of them that led us to think the project would be riskier or less impactful than we initially believed, and we haven’t shared the identity or information with the applicant to preserve the privacy of the individual.
We have a higher bar for funding organizations than other projects, because they are more likely to stick around even if we decide they’re not worth funding in the future.
When evaluating the more borderline grants in this space, I often ask and rely heavily on the advice of others working in the policy space, weighted by how much I trust their judgment. I think this is basically a reasonable algorithm to follow, given that (a) they have a lot of context that I don’t, and (b) I think the downside risks of poorly-executed policy projects have spillover effects to other policy projects, which means that others in policy are genuine stakeholders in these decisions.
That being said, I think there’s a surprising amount of disagreement in what projects others in policy think are good, so I think the particular choice of advisors here makes a big difference.
I do think projects interacting with policymakers have substantial room for downside, including:
Pushing policies that are harmful
Making key issues partisan
Creating an impression (among policymakers or the broader world) that people who care about the long-term future are offputting, unrealistic, incompetent, or otherwise undesirable to work with
“Taking up the space” such that future actors who want to make long-term future-focused asks are encouraged or expected to work through or coordinate with the existing project
I suspect we also differ in our views of the upsides of some of this work– a lot of the projects we’ve rejected have wanted to do AI-focused policy work, and I tend to think that we don’t have very good concrete asks for policymakers in this space.