Iâm somewhat loose with definitions here, defining it effectively the same way as âTransformative AIâ (TAI) but intentionally not being too prescriptive for the most part (except in cases where definitions become important)
Tom Barnesđ¸
Karma: 1,672
I think some form of AI-assited governance have great potential.
However, it seems like several of these ideas are (in theory) possible in some format todayâyet in practice donât get adopted. E.g.
Enhancing epistemics and decision-making processes at the top levels of organizations, leading to more informed and rational strategies.
I think itâs very hard to get even the most basic forms of good epistemic practices (e.g. putting probabilities on helpful, easy-to-forecast statements) embedded at the top levels of organizations (for standard moral maze-type reasons).
As such I think the role of AI here is pretty limitedâthe main bottleneck to adoption is political /â bureacratic, rather than technological.
Iâd guess the way to make progress here is in aligning [implementation of AI-assisted governance] with [incentives of influential people in the organization] - i.e. you first have to get the organization to actually care about good goverance (perhaps by joining it, or using external levers).
[Of course, if we go through crazy explosive AI-driven growth then maybe the existing model of large organizations being slow will no longer be trueâand hence there would be more scope for AI-assisted governance]
Hi Arden, thanks for the comment
I think this was something that got lost-in-translation during the grant writeup process. In the grant evaluation doc this was written as:
I think [Richardâs research] clearly fits into the kind of project that we want the EA community to be - [that output] feels pretty closely aligned to our âprinciples-first EAâ vision
This a fairly fuzzy view, but my impression is Richardâs outputs will align with the takes in this post both by âfighting for EA to thrive long termâ (increasing the quality of discussion around EA in the public domain), and also by increasing the number of âthoughtful, sincere, selflessâ individuals in the community (via his substack which has a decently sized readership), who may become more deeply involved in EA as a result.
--
On the broader question about âprinciples firstâ vs âcause specificâ EA work:
I think EAIF will ceteris paribus fund more âprinciples-firstâ projects than cause specific meta projects compared to previously.
However, I think this counterbalances other grantmaking changes which focus on cause-specific meta directly (e.g. OP GCR capacity building /â GHW funding).
Iâd guess this nets out such that the fraction of funding towards âprinciples-firstâ EA decreases, rather than increases (due to OPâs significantly larger assets).
As such, the decision to focus on âprinciples-firstâ is more of a comp advantage /â focus for EAIF specifically, rather than a belief about what the community should do more broadly
(That said, on the margin I think a push in this direction is probably helpful /â healthy for the community more broadly, but this is pretty lightly held and other fund managers might disagree)
I thought it might be helpful for me to add my own thoughts, as a fund manger at EAIF (Note Iâm speaking in a personal capacity, not on behalf of EA Funds or EV).
Firstly, Iâd like to apologise for my role in these mistakes. I was the Primary Investigator (PI) for Igorâs application, and thus I share some responsibility here. Specifically as the PI, I should have (a) evaluated the application sooner, (b) reached a final recommendation sooner, and (c) been more responsive to communications after making a decision
I did not make an initial decision until November 20. This was too short a timeframe to provide Igor a final decision by November 24.
I did not reach a final recommendation until November 30th. This was due to the final recommendation we made being somewhat more complex than the original proposal.[1]
In February, I did not provide with a full response for Igorâs request for an update on his application.
Second, Iâd like to apologise to any other applicants to EAIF who have faced similar unreasonably long delays. Whilst we get back to the most applicatants on a reasonable timeframe (see other comments), there are a few cases I am well aware of where we deadlines have been missed for too long. Iâm aware of a couple of instances where this has caused significant stressâagain, I would like to express my deepest regret for this.
As broader context, I think itâs worth emphasising that EAIF is highly under-resourced at the moment. Itâs fairly common for orgs to say theyâre âcapacity constrainedââbut I think this is more true for EAIF in the last ~3 months than any other period:
In summer 2023, EAIF had five part-time fund managers. With OPâs distancing from EAIF, we dropped down the three. In late 2023, we then dropped to just myself (part-time), and Caleb as acting EAIF chair.
Given these changes, I would be suprised if EAIF has run on more than ~0.25 FTE over the past three months.
As such, it has been a challenge for EAIF to fulfil all of itâs key responsibilitesâas well as developoing a coherent strategy and fundraising given constraints.
We are now recruiting /â onboarding new fund managers, so this pressure should be alleviated soon.
- ^
Iâm happy to go into details as to the details about changes we proposed and why, although I donât think they are especially relevant to this situation
My boring answer would be to see details on our website. In terms of submission style, we say:
We recommend that applicants take about 1â2 hours to write their applications. This does not include the time spent developing the plan and strategy for the project â we recommend thinking about those carefully prior to applying.
Please keep your answers brief and ensure the total length of your responses does not exceed 10,000 characters. We recommend a total length of 2,000â5,000 characters.
We recommend focusing on the substantive arguments in favour of your project rather than polishing your submission.
We recommend honestly communicating the strengths and weaknesses of your project rather than trying to âsellâ your proposal.
You can find details on the scope of grants that EAIF will consider funding for here (although this is subject to changeâdetails here).
For non-obvious mistakes, some examples that come to mind are:
Unclear theory of changeâI think good applications often have a clear sense of what theyâre trying to acheive, and how they plan to acheive it. This may seem relatively obvious, but I think still often goes underestimated. Put another way: itâs very rare for me to think âthis applicant has thought about their path to impact too muchâ
Providing too little informationâwhilst we do recommend that applicants donât take too long to write applications, it can be hard to make well evidenced decisions without having much information to go on. For projects that are clearly great /â terrible this is less of an issue, but projects close to the bar do benefit from some (at least basic) info.
Providing too much (irrelevant) informationâOn the flip side, a large amount of (irrelevant) information can distract from the core case for the project. E.g. if an applicant does not have track record in an area theyâre looking to move towards, I much prefer that they directly state this rather than include highly irrelevant info to fill the page.
Not providing any referencesâWe often reach out to references, who can give a more detailed opinion on the applicant and/âor their project plan. Without any 3rd party to contact, it can be difficult to verify claims made in an application.
Optimising for p(receive grant) rather than Impactâthis is a tricky one, since people apply for projects which they believe are highly impactful, and an obvious instrumental goal to that happening is to get funding. But ultimately, itâs worth being upfront and honest about weakenesses, since ultimately our common goal is to do the most good, and perusasion /â deception undermine that (even if this increases p(receive grant))
Intepreting rejection (or success) too strongly- The grant appplication process (like job applications) is extremely noisy, in which a single decision gives limited evidence about an application. Of course, this advise goes both waysâit is not literally 0 evidence, and some projects shouldnât be fundedâbut I do worry if people over-update on a rejection from EAIF, especially when they are pretty close to the bar
Currently we donât have a process for retroactively evaluating EAIF grants. However, there are a couple of informal channels which can help to improve decision-making:
We request that grantees fill out a short form detailing the impact of their grant after six months. These reports are both directly helpful for evaluating a future application from the grantee, and indirectly helpful at calibrating the âbang-for-your-buckâ we should expect from different grant sizes for different projects
When evaluting the renewal of a grant, we can compare the initial applicationâs plans with the track record they list in a later application, to see if the grant was a success on their own terms.
One technique Iâve picked up is evaluating grants in reverseâreading the details of the project, and then giving a rough estimate of a willingness to pay for a project of that nature. Looking at the actual cost of the project can then help quickly determine if it meets a bar for funding that Iâver pre-registered
I think a lack of a proper M&E function is a problem, and one that I would be keen to address longer term
HeyâI think itâs important to clarify that EAIF is optimising for something fairly different from GiveWell (although we share the same broad aim):
Specifically, GiveWell is optimising for lives saved in the next few years, under the constraint of health projects in LMICs, with a high probability of impact and fairly immediate /â verifable results.
Meanwhile, EAIF is focused on a hits-based, low-certainty area, where the evidence base is weaker, grants have longer paths to impact, and the overarching goal is often unclear.
As such, a direct/âequivalent comparison is fairly challenging, with our âbar for fundingâ fairly different to GiveWellâs. The other caveat is that we donât have a systematic process for retroactively classifying grants as âwinsâ or âlossesââour current M&E process is much more fuzzy.
Given this, any answer about the cost-effectiveness of GiveWell vs EAIF will be pretty subjective and prone to error.
Nonetheless, my personal opinion is that the mean EAIF grant is likely more impactful than the typical GiveWell grant. Very briefly, this is becuase:
I think many of our grants have /â would have a >1x multiplier on donations to GiveWell top charities, if we evaluated them under this framework (as outlined here)
Further, I think there are more impactful ways to save /â improve the lives of current people than donating to GiveWellâs top charities; and I think there are even greater opportunities for impact (via improving animal welfare, or the long-term future). Many of EAIFâs grantees cover more than just fundraising for effective global health charities, and thus I expect they will (on average) have a higher impact
But this is just my personal view, contingent on a very large number of assumptions, which people very reasonably disagree on.
I think the premise of your question is roughly correct: I do think itâs pretty hard to âhelp EA notice what it is important to work onâ, for a bunch of reasons:
It could lead to new, unexpected directions which might be counterintuive /â controversial.
it requires the community to have the psychological, financial and intellectual safety to identify /â work on causes which may not be promising
It needs a non-trivial number of people to engage with the result of exploration, and act upon it (including people who can direct substantial resources)
It has a very long feedback loop, which can be a) demoralising, and b) difficult to predict if it ever has an impact.
Given those challenges, itâs not suprising to me if we struggle to find many projects in this area. To overcome that I think we would need to take a more active approach (e.g. RFPs, etc). But we are still in the early days of thinking about these kinds of questions
Good Question! We have discussed running RFP(s) to more directly support projects weâd like to see. First, though, I think we want to do some more strategic thinking about the direction we want EAIF to go in, and hence at this stage I think we are fairly unsure about which project types weâd like to see more of.
Caveats aside, I personally[1] would be pretty interested in:
Macrostrategy /â cause prioritization research. I think a substantional amount of intellectual progress was made in the 2000s /â early 2010s from a constellation of different places (e.g. early FHI, the rationality community, Randomistas, GiveWell, etc) which led to the EA community developing some crucial ideas. Sadly, I think we have seen less of that âraw idea generation processâ in recent times. Iâd be pretty excited if there was a project that was able to revive this spirit, although I think it would be (very) difficult to pull off.
High quality communications of EA Principles. Many core EA ideas are hard to communicate, especially in low bandwith formats. In practice I think this means that discourse around EA (e.g. on twitter) is pretty poor (and worsening). Whilst thereâs been work to remedy this in specific cause areas (like AISCC), there donât seem to be many public communications champions of EA as an intellectual project, nor as a community of people earnestly aiming to improve the world. Again, I think this is hard to remedy, and easy to get wrong, but I would be pretty excited for someone to try.
Fundraising. Promising projects across all cause areas are going unfunded due to funding constraints (EAIF included). Iâm additionally worried that there are several fundraising organisations - whoâs principle goal is âfund EA/âEA-ish projectsââare distancing themselves from the EA label, leaving projects (especially in the EA community) without a source of funding.
- ^
Not speaking for EAIF /â EA Funds /â EV
Hey, good question!
Hereâs a crude rationale:
Suppose that by donating $1k to an EAIF project, they get 1 new person to consider donating more effectively.
This 1 new person pledges to give 1% of their salary to GiveWellâs top charities, and they do this for the next 10 years.
If they make (say) $50k /â year, then over 10 years they will donate $5k to GiveWell charities.
The net result is that a $1k donation to EAIF led to $5k donated to top GiveWell charitiesâor a dollar donated to EAIF goes 5x further than a GiveWell Top charity
Of course, there are a bunch of important considerations and nuance that have been ignored in this hypotheticalâindeed, I think itâs pretty important to be cautious /â suspicious about calculations like the above, so we should often discount the âmultiplierâ factor signficantly. Nonetheless, I think (some version of) the above argument goes through for a number of projects EAIF supports.
I agree thereâs no single unified resource. Having said that, I found Richard Ngoâs âfive alignment clustersâ pretty helpful for bucketing different groups & arguments together. Reposting below:
MIRI cluster. Think that P(doom) is very high, based on intuitions about instrumental convergence, deceptive alignment, etc. Does work thatâs very different from mainstream ML. Central members: Eliezer Yudkowsky, Nate Soares.
Structural risk cluster. Think that doom is more likely than not, but not for the same reasons as the MIRI cluster. Instead, this cluster focuses on systemic risks, multi-agent alignment, selective forces outside gradient descent, etc. Often work thatâs fairly continuous with mainstream ML, but willing to be unusually speculative by the standards of the field. Central members: Dan Hendrycks, David Krueger, Andrew Critch.
Constellation cluster. More optimistic than either of the previous two clusters. Focuses more on risk from power-seeking AI than the structural risk cluster, but does work that is more speculative or conceptually-oriented than mainstream ML. Central members: Paul Christiano, Buck Shlegeris, Holden Karnofsky. (Named after Constellation coworking space.)
Prosaic cluster. Focuses on empirical ML work and the scaling hypothesis, is typically skeptical of theoretical or conceptual arguments. Short timelines in general. Central members: Dario Amodei, Jan Leike, Ilya Sutskever.
Mainstream cluster. Alignment researchers who are closest to mainstream ML. Focuses much less on backchaining from specific threat models and more on promoting robustly valuable research. Typically more concerned about misuse than misalignment, although worried about both. Central members: Scott Aaronson, David Bau.
To return to the question âwhat is the current best single article (or set of articles) that provide a well-reasoned and comprehensive case for believing that there is a substantial (>10%) probability of an AI catastrophe this century?â, my guess is that these different groups would respond as follows:[1]
MIRI cluster: List of Lethalities, Sharp Left Turn, Superintelligence
Structural Risk cluster: Natural selection favours AIs, RAAP
Constellation cluster: Is Power-seeking AI an x-risk, some Cold Takes posts, Scheming AIs
Prosaic cluster: Concrete problems in AI safety, [perhaps something more recent?]
Mainstream cluster: Reform AI Alignment, [not sureâperhaps nothing arguing for >10%?]
- ^
But I could easily be misrepresenting these different groupsâ âcoreâ argument, and I havenât read all of these, so could be misunderstanding
A couple of weeks ago I blocked all mentions of âEffective Altruismâ, âAI Safetyâ, âOpenAIâ, etc from my twitter feed. Since then Iâve noticed it become much less of a time sink, and much better for mental health. Would strongly recommend!
I wrote the following on a draft of this post. For context, I currently do (very) part-time work at EAIF
Overall, Iâm pretty excited to see EAIF orient to a principles-first EA. Despite recent challenges, I continue to believe that the EA community is doing something special and important, and is fundamentally worth fighting for. With this reorientation of EAIF, I hope we can get the EA community back to a strong position. I share many of the uncertainties listedâabout whether this is a viable project, how EAIF will practically evaluate grants under this worldview, or if itâs even philosophically coherent. Nonetheless, Iâm excited to see what can be done.
Yeah thatâs fair. I wrote this somewhat off the cuff, but because it got more engagement than I thought Iâd make it a full post if I wrote again
Is your claim âImpartial altruists with ~no credence on longtermism would have more impact donating to AI/âGCRs over animals /â global healthâ?
To my mind, this is the crux, because:
If Yes, then I agree that it totally makes sense for non-longtermist EAs to donate to AI/âGCRs
If No, then Iâm confused why one wouldnât donate to animals /â global health instead?
[I use âdonateâ rather than âwork onâ because donations arenât sensitive to individual circumstances, e.g. personal fit. Iâm also assuming impartiality because this seems core to EA to me, but of course one could donate /â work on a topic for non-impartial/â non-EA reasons]
Mildly against the Longtermism --> GCR shift
Epistemic status: Pretty uncertain, somewhat ramblyTL;DR replacing longtermism with GCRs might get more resources to longtermist causes, but at the expense of non-GCR longtermist interventions and broader community epistemics
Over the last ~6 months Iâve noticed a general shift amongst EA orgs to focus less on reducing risks from AI, Bio, nukes, etc based on the logic of longtermism, and more based on Global Catastrophic Risks (GCRs) directly. Some data points on this:
Open Phil renaming itâs EA Community Growth (Longtermism) Team to GCR Capacity Building
This post from Claire Zabel (OP)
Giving What We Canâs new Cause Area Fund being named âRisk and Resilience,â with the goal of âReducing Global Catastrophic Risksâ
Longview-GWWCâs Longtermism Fund being renamed the âEmerging Challenges Fundâ
Anecdotal data from conversations with people working on GCRs /â X-risk /â Longtermist causes
My guess is these changes are (almost entirely) driven by PR concerns about longtermism. I would also guess these changes increase the number of people donation /â working on GCRs, which is (by longtermist lights) a positive thing. After all, no-one wants a GCR, even if only thinking about people alive today.
Yet, I canât help but feel something is off about this framing. Some concerns (no particular ordering):
From a longtermist (~totalist classical utilitarian) perspective, thereâs a huge difference between ~99% and 100% of the population dying, if humanity recovers in the former case, but not the latter. Just looking at GCRs on their own mostly misses this nuance.
(see Parfit Reasons and Persons for the full thought experiment)
From a longtermist (~totalist classical utilitarian) perspective, preventing a GCR doesnât differentiate between âhumanity prevents GCRs and realises 1% of itâs potentialâ and âhumanity prevents GCRs realises 99% of its potentialâ
Preventing an extinction-level GCR might move us from 0% to 1% of future potential, but thereâs 99x more value in focusing on going from the âokay (1%)â to âgreat (100%)â future.
See Aird 2020 for more nuances on this point
From a longtermist (~suffering focused) perspective, reducing GCRs might be net-negative if the future is (in expectation) net-negative
E.g. if factory farming continues indefinitely, or due to increasing the chance of an S-Risk
See Melchin 2021 or DiGiovanni 2021 for more
(Note this isnât just a concern for suffering-focused ethics people)
From a longtermist perspective, a focus on GCRs neglects non-GCR longtermist interventions (e.g. trajectory changes, broad longtermism, patient altruism/âphilanthropy, global priorities research, institutional reform, )
From a âcurrent generationsâ perspective, reducing GCRs is probably not more cost-effective than directly improving the welfare of people /â animals alive today
Iâm pretty uncertain about this, but my guess is that alleviating farmed animal suffering is more welfare-increasing than e.g. working to prevent an AI catastrophe, given the latter is pretty intractable (But I havenât done the numbers)
See discussion here
If GCRs actually are more cost-effective under a âcurrent generationsâ worldview, then I question why EAs would donate to global health /â animal charities (since this is no longer a question of âworldview diversificationâ, just raw cost-effectiveness)
More meta points
From a community-building perspective, pushing people straight into GCR-oriented careers might work short-term to get resources to GCRs, but could lose the long-run benefits of EA /â Longtermist ideas. I worry this might worsen community epistemics about the motivation behind working on GCRs:
If GCRs only go through on longtermist grounds, but longtermism is false, then impartial altruists should rationally switch towards current-generations opportunities. Without a grounding in cause impartiality, however, people wonât actually make that switch
From a general virtue ethics /â integrity perspective, making this change on PR /â marketing reasons aloneâwithout an underlying change in longtermist motivationâfeels somewhat deceptive.
As a general rule about integrity, I think itâs probably bad to sell people on doing something for reason X, when actually you want them to do it for Y, and youâre not transparent about that
Thereâs something fairly disorienting about the community switching so quickly from [quite aggressive] âyay longtermism!â (e.g. much hype around launch of WWOTF) to essentially disowning the word longtermism, with very little mention /â admission that this happened or why
I havenât yet decided, but itâs likely that a majority of my donations will go to this yearâs donor lottery. Iâm fairly convinced by the arguments in favour of donor lotteries [1, 2], and would encourage others to consider them if theyâre unsure where to give.
Having said that, lotteries have less fuzzies than donating directly, so I may separately give to some effective charities which Iâm personally excited about.
ThanksâI just saw RP put out this post, which makes much the same point. Good to be cautious about interpreting these results!
YouGov Poll on SBF and EA
I recently came across this article from YouGov (published last week), summarizing a survey of US citizens for their opinions on Sam Bankman-Fried, Cryptocurrency and Effective Altruism.
I half-expected the survey responses to be pretty negative about EA, given press coverage and potential priming effects associating SBF to EA. So I was positively surprised that:
(itâs worth noting that there were only ~1000 participants, and the survey was online only)
Iâm pretty confident (~80-90%?) this is true, for reasons well summarized here.
Iâm interested in thoughts on the OOM difference between animal welfare vs GHD (i.e. would $100m to animal welfare be 2x better than GHD, or 2000x?)