I’m somewhat loose with definitions here, defining it effectively the same way as “Transformative AI” (TAI) but intentionally not being too prescriptive for the most part (except in cases where definitions become important)
Tom Barnes
Karma: 1,671
Navigating Risks from Advanced Artificial Intelligence: A Guide for Philanthropists [Founders Pledge]
I think some form of AI-assited governance have great potential.
However, it seems like several of these ideas are (in theory) possible in some format today—yet in practice don’t get adopted. E.g.
Enhancing epistemics and decision-making processes at the top levels of organizations, leading to more informed and rational strategies.
I think it’s very hard to get even the most basic forms of good epistemic practices (e.g. putting probabilities on helpful, easy-to-forecast statements) embedded at the top levels of organizations (for standard moral maze-type reasons).
As such I think the role of AI here is pretty limited—the main bottleneck to adoption is political / bureacratic, rather than technological.
I’d guess the way to make progress here is in aligning [implementation of AI-assisted governance] with [incentives of influential people in the organization] - i.e. you first have to get the organization to actually care about good goverance (perhaps by joining it, or using external levers).
[Of course, if we go through crazy explosive AI-driven growth then maybe the existing model of large organizations being slow will no longer be true—and hence there would be more scope for AI-assisted governance]
Hi Arden, thanks for the comment
I think this was something that got lost-in-translation during the grant writeup process. In the grant evaluation doc this was written as:
I think [Richard’s research] clearly fits into the kind of project that we want the EA community to be - [that output] feels pretty closely aligned to our “principles-first EA” vision
This a fairly fuzzy view, but my impression is Richard’s outputs will align with the takes in this post both by “fighting for EA to thrive long term” (increasing the quality of discussion around EA in the public domain), and also by increasing the number of “thoughtful, sincere, selfless” individuals in the community (via his substack which has a decently sized readership), who may become more deeply involved in EA as a result.
--
On the broader question about “principles first” vs “cause specific” EA work:
I think EAIF will ceteris paribus fund more “principles-first” projects than cause specific meta projects compared to previously.
However, I think this counterbalances other grantmaking changes which focus on cause-specific meta directly (e.g. OP GCR capacity building / GHW funding).
I’d guess this nets out such that the fraction of funding towards “principles-first” EA decreases, rather than increases (due to OP’s significantly larger assets).
As such, the decision to focus on “principles-first” is more of a comp advantage / focus for EAIF specifically, rather than a belief about what the community should do more broadly
(That said, on the margin I think a push in this direction is probably helpful / healthy for the community more broadly, but this is pretty lightly held and other fund managers might disagree)
Effective Altruism Infrastructure Fund: March 2024 recommendations
I thought it might be helpful for me to add my own thoughts, as a fund manger at EAIF (Note I’m speaking in a personal capacity, not on behalf of EA Funds or EV).
Firstly, I’d like to apologise for my role in these mistakes. I was the Primary Investigator (PI) for Igor’s application, and thus I share some responsibility here. Specifically as the PI, I should have (a) evaluated the application sooner, (b) reached a final recommendation sooner, and (c) been more responsive to communications after making a decision
I did not make an initial decision until November 20. This was too short a timeframe to provide Igor a final decision by November 24.
I did not reach a final recommendation until November 30th. This was due to the final recommendation we made being somewhat more complex than the original proposal.[1]
In February, I did not provide with a full response for Igor’s request for an update on his application.
Second, I’d like to apologise to any other applicants to EAIF who have faced similar unreasonably long delays. Whilst we get back to the most applicatants on a reasonable timeframe (see other comments), there are a few cases I am well aware of where we deadlines have been missed for too long. I’m aware of a couple of instances where this has caused significant stress—again, I would like to express my deepest regret for this.
As broader context, I think it’s worth emphasising that EAIF is highly under-resourced at the moment. It’s fairly common for orgs to say they’re “capacity constrained”—but I think this is more true for EAIF in the last ~3 months than any other period:
In summer 2023, EAIF had five part-time fund managers. With OP’s distancing from EAIF, we dropped down the three. In late 2023, we then dropped to just myself (part-time), and Caleb as acting EAIF chair.
Given these changes, I would be suprised if EAIF has run on more than ~0.25 FTE over the past three months.
As such, it has been a challenge for EAIF to fulfil all of it’s key responsibilites—as well as developoing a coherent strategy and fundraising given constraints.
We are now recruiting / onboarding new fund managers, so this pressure should be alleviated soon.
- ^
I’m happy to go into details as to the details about changes we proposed and why, although I don’t think they are especially relevant to this situation
My boring answer would be to see details on our website. In terms of submission style, we say:
We recommend that applicants take about 1–2 hours to write their applications. This does not include the time spent developing the plan and strategy for the project – we recommend thinking about those carefully prior to applying.
Please keep your answers brief and ensure the total length of your responses does not exceed 10,000 characters. We recommend a total length of 2,000–5,000 characters.
We recommend focusing on the substantive arguments in favour of your project rather than polishing your submission.
We recommend honestly communicating the strengths and weaknesses of your project rather than trying to “sell” your proposal.
You can find details on the scope of grants that EAIF will consider funding for here (although this is subject to change—details here).
For non-obvious mistakes, some examples that come to mind are:
Unclear theory of change—I think good applications often have a clear sense of what they’re trying to acheive, and how they plan to acheive it. This may seem relatively obvious, but I think still often goes underestimated. Put another way: it’s very rare for me to think “this applicant has thought about their path to impact too much”
Providing too little information—whilst we do recommend that applicants don’t take too long to write applications, it can be hard to make well evidenced decisions without having much information to go on. For projects that are clearly great / terrible this is less of an issue, but projects close to the bar do benefit from some (at least basic) info.
Providing too much (irrelevant) information—On the flip side, a large amount of (irrelevant) information can distract from the core case for the project. E.g. if an applicant does not have track record in an area they’re looking to move towards, I much prefer that they directly state this rather than include highly irrelevant info to fill the page.
Not providing any references—We often reach out to references, who can give a more detailed opinion on the applicant and/or their project plan. Without any 3rd party to contact, it can be difficult to verify claims made in an application.
Optimising for p(receive grant) rather than Impact—this is a tricky one, since people apply for projects which they believe are highly impactful, and an obvious instrumental goal to that happening is to get funding. But ultimately, it’s worth being upfront and honest about weakenesses, since ultimately our common goal is to do the most good, and perusasion / deception undermine that (even if this increases p(receive grant))
Intepreting rejection (or success) too strongly- The grant appplication process (like job applications) is extremely noisy, in which a single decision gives limited evidence about an application. Of course, this advise goes both ways—it is not literally 0 evidence, and some projects shouldn’t be funded—but I do worry if people over-update on a rejection from EAIF, especially when they are pretty close to the bar
Currently we don’t have a process for retroactively evaluating EAIF grants. However, there are a couple of informal channels which can help to improve decision-making:
We request that grantees fill out a short form detailing the impact of their grant after six months. These reports are both directly helpful for evaluating a future application from the grantee, and indirectly helpful at calibrating the “bang-for-your-buck” we should expect from different grant sizes for different projects
When evaluting the renewal of a grant, we can compare the initial application’s plans with the track record they list in a later application, to see if the grant was a success on their own terms.
One technique I’ve picked up is evaluating grants in reverse—reading the details of the project, and then giving a rough estimate of a willingness to pay for a project of that nature. Looking at the actual cost of the project can then help quickly determine if it meets a bar for funding that I’ver pre-registered
I think a lack of a proper M&E function is a problem, and one that I would be keen to address longer term
Hey—I think it’s important to clarify that EAIF is optimising for something fairly different from GiveWell (although we share the same broad aim):
Specifically, GiveWell is optimising for lives saved in the next few years, under the constraint of health projects in LMICs, with a high probability of impact and fairly immediate / verifable results.
Meanwhile, EAIF is focused on a hits-based, low-certainty area, where the evidence base is weaker, grants have longer paths to impact, and the overarching goal is often unclear.
As such, a direct/equivalent comparison is fairly challenging, with our “bar for funding” fairly different to GiveWell’s. The other caveat is that we don’t have a systematic process for retroactively classifying grants as “wins” or “losses”—our current M&E process is much more fuzzy.
Given this, any answer about the cost-effectiveness of GiveWell vs EAIF will be pretty subjective and prone to error.
Nonetheless, my personal opinion is that the mean EAIF grant is likely more impactful than the typical GiveWell grant. Very briefly, this is becuase:
I think many of our grants have / would have a >1x multiplier on donations to GiveWell top charities, if we evaluated them under this framework (as outlined here)
Further, I think there are more impactful ways to save / improve the lives of current people than donating to GiveWell’s top charities; and I think there are even greater opportunities for impact (via improving animal welfare, or the long-term future). Many of EAIF’s grantees cover more than just fundraising for effective global health charities, and thus I expect they will (on average) have a higher impact
But this is just my personal view, contingent on a very large number of assumptions, which people very reasonably disagree on.
I think the premise of your question is roughly correct: I do think it’s pretty hard to “help EA notice what it is important to work on”, for a bunch of reasons:
It could lead to new, unexpected directions which might be counterintuive / controversial.
it requires the community to have the psychological, financial and intellectual safety to identify / work on causes which may not be promising
It needs a non-trivial number of people to engage with the result of exploration, and act upon it (including people who can direct substantial resources)
It has a very long feedback loop, which can be a) demoralising, and b) difficult to predict if it ever has an impact.
Given those challenges, it’s not suprising to me if we struggle to find many projects in this area. To overcome that I think we would need to take a more active approach (e.g. RFPs, etc). But we are still in the early days of thinking about these kinds of questions
Good Question! We have discussed running RFP(s) to more directly support projects we’d like to see. First, though, I think we want to do some more strategic thinking about the direction we want EAIF to go in, and hence at this stage I think we are fairly unsure about which project types we’d like to see more of.
Caveats aside, I personally[1] would be pretty interested in:
Macrostrategy / cause prioritization research. I think a substantional amount of intellectual progress was made in the 2000s / early 2010s from a constellation of different places (e.g. early FHI, the rationality community, Randomistas, GiveWell, etc) which led to the EA community developing some crucial ideas. Sadly, I think we have seen less of that “raw idea generation process” in recent times. I’d be pretty excited if there was a project that was able to revive this spirit, although I think it would be (very) difficult to pull off.
High quality communications of EA Principles. Many core EA ideas are hard to communicate, especially in low bandwith formats. In practice I think this means that discourse around EA (e.g. on twitter) is pretty poor (and worsening). Whilst there’s been work to remedy this in specific cause areas (like AISCC), there don’t seem to be many public communications champions of EA as an intellectual project, nor as a community of people earnestly aiming to improve the world. Again, I think this is hard to remedy, and easy to get wrong, but I would be pretty excited for someone to try.
Fundraising. Promising projects across all cause areas are going unfunded due to funding constraints (EAIF included). I’m additionally worried that there are several fundraising organisations - who’s principle goal is “fund EA/EA-ish projects”—are distancing themselves from the EA label, leaving projects (especially in the EA community) without a source of funding.
- ^
Not speaking for EAIF / EA Funds / EV
Hey, good question!
Here’s a crude rationale:
Suppose that by donating $1k to an EAIF project, they get 1 new person to consider donating more effectively.
This 1 new person pledges to give 1% of their salary to GiveWell’s top charities, and they do this for the next 10 years.
If they make (say) $50k / year, then over 10 years they will donate $5k to GiveWell charities.
The net result is that a $1k donation to EAIF led to $5k donated to top GiveWell charities—or a dollar donated to EAIF goes 5x further than a GiveWell Top charity
Of course, there are a bunch of important considerations and nuance that have been ignored in this hypothetical—indeed, I think it’s pretty important to be cautious / suspicious about calculations like the above, so we should often discount the “multiplier” factor signficantly. Nonetheless, I think (some version of) the above argument goes through for a number of projects EAIF supports.
EA Infrastructure Fund Ask Us Anything (January 2024)
I agree there’s no single unified resource. Having said that, I found Richard Ngo’s “five alignment clusters” pretty helpful for bucketing different groups & arguments together. Reposting below:
MIRI cluster. Think that P(doom) is very high, based on intuitions about instrumental convergence, deceptive alignment, etc. Does work that’s very different from mainstream ML. Central members: Eliezer Yudkowsky, Nate Soares.
Structural risk cluster. Think that doom is more likely than not, but not for the same reasons as the MIRI cluster. Instead, this cluster focuses on systemic risks, multi-agent alignment, selective forces outside gradient descent, etc. Often work that’s fairly continuous with mainstream ML, but willing to be unusually speculative by the standards of the field. Central members: Dan Hendrycks, David Krueger, Andrew Critch.
Constellation cluster. More optimistic than either of the previous two clusters. Focuses more on risk from power-seeking AI than the structural risk cluster, but does work that is more speculative or conceptually-oriented than mainstream ML. Central members: Paul Christiano, Buck Shlegeris, Holden Karnofsky. (Named after Constellation coworking space.)
Prosaic cluster. Focuses on empirical ML work and the scaling hypothesis, is typically skeptical of theoretical or conceptual arguments. Short timelines in general. Central members: Dario Amodei, Jan Leike, Ilya Sutskever.
Mainstream cluster. Alignment researchers who are closest to mainstream ML. Focuses much less on backchaining from specific threat models and more on promoting robustly valuable research. Typically more concerned about misuse than misalignment, although worried about both. Central members: Scott Aaronson, David Bau.
To return to the question “what is the current best single article (or set of articles) that provide a well-reasoned and comprehensive case for believing that there is a substantial (>10%) probability of an AI catastrophe this century?”, my guess is that these different groups would respond as follows:[1]
MIRI cluster: List of Lethalities, Sharp Left Turn, Superintelligence
Structural Risk cluster: Natural selection favours AIs, RAAP
Constellation cluster: Is Power-seeking AI an x-risk, some Cold Takes posts, Scheming AIs
Prosaic cluster: Concrete problems in AI safety, [perhaps something more recent?]
Mainstream cluster: Reform AI Alignment, [not sure—perhaps nothing arguing for >10%?]
- ^
But I could easily be misrepresenting these different groups’ “core” argument, and I haven’t read all of these, so could be misunderstanding
A couple of weeks ago I blocked all mentions of “Effective Altruism”, “AI Safety”, “OpenAI”, etc from my twitter feed. Since then I’ve noticed it become much less of a time sink, and much better for mental health. Would strongly recommend!
I wrote the following on a draft of this post. For context, I currently do (very) part-time work at EAIF
Overall, I‘m pretty excited to see EAIF orient to a principles-first EA. Despite recent challenges, I continue to believe that the EA community is doing something special and important, and is fundamentally worth fighting for. With this reorientation of EAIF, I hope we can get the EA community back to a strong position. I share many of the uncertainties listed—about whether this is a viable project, how EAIF will practically evaluate grants under this worldview, or if it’s even philosophically coherent. Nonetheless, I’m excited to see what can be done.
EA Infrastructure Fund’s Plan to Focus on Principles-First EA
Yeah that’s fair. I wrote this somewhat off the cuff, but because it got more engagement than I thought I’d make it a full post if I wrote again
Is your claim “Impartial altruists with ~no credence on longtermism would have more impact donating to AI/GCRs over animals / global health”?
To my mind, this is the crux, because:
If Yes, then I agree that it totally makes sense for non-longtermist EAs to donate to AI/GCRs
If No, then I’m confused why one wouldn’t donate to animals / global health instead?
[I use “donate” rather than “work on” because donations aren’t sensitive to individual circumstances, e.g. personal fit. I’m also assuming impartiality because this seems core to EA to me, but of course one could donate / work on a topic for non-impartial/ non-EA reasons]
I’m pretty confident (~80-90%?) this is true, for reasons well summarized here.
I’m interested in thoughts on the OOM difference between animal welfare vs GHD (i.e. would $100m to animal welfare be 2x better than GHD, or 2000x?)