Whatâs the reason for the change from Longtermism to GCRs? How has/âwill this change strategy going forward?
Tom Barnesđ¸
Karma: 1,672
It seems that OPâs AI safety & gov teams have both been historically capacity-constrained. Why the decision to hire for these roles now (rather than earlier)?
Ten Project Ideas for AI X-Risk Prioritization
I made a list of 10 ideas Iâd be excited for someone to tackle, within the broad problem of âhow to prioritize resources within AI X-risk?â I wonât claim these projects are more /â less valuable than other things people could be doing. However, Iâd be excited if someone gave a stab at them
10 Ideas:
Threat Model Prioritization
Country Prioritization
An Inside-view timelines model
Modelling AI Lab deployment decisions
How are resources currently allocated between risks /â interventions /â countries
How to allocate between âAI Safetyâ vs âAI Governanceâ
Timing of resources to reduce risks
Correlation between Safety and Capabilities
How should we reason about prioritization?
How should we model prioritization?
I wrote up a longer (but still scrappy) doc here
Thanks Ajeya, this is very helpful and clarifying!
I am the only person who is primarily focused on funding technical research projects ⌠I began making grants in November 2022
Does this mean that prior to November 2022 there were ~no full-time technical AI safety grantmakers at Open Philanthropy?
OP (prev. GiveWell labs) has been evaluating grants in the AI safety space for over 10 years. In that time the AI safety field and Open Philanthropy have both grown, with OP granting over $300m on AI risk. Open Phil has also done a lot of research on the problem. So, from someone on the outside, it seems surprising that the number of people making grants has been consistently low
Following the episode with Mustafa, it would be great to interview the founders of leading AI labsâperhaps Dario (Anthropic) [again], Sam (OpenAI), or Demis (DeepMind). Or alternatively, the companies that invest /â support themâSundar (Google) or Satya (Microsoft).
It seems valuable to elicit their honest opinions[1] about âp(doom)â, timelines, whether they believe theyâve been net-positive for the world, etc.
- ^
I think one risk here is either:
a) not challenging them firmly enoughâlending them undue credibility /â legitimacy in the minds of listeners
b) challenging them too stronglyâreducing willingness to engage, less goodwill, etc
- ^
For deception (not deceptive alignment) - AI Deception: A Survey of Examples, Risks, and Potential Solutions (section 2)
This looks very exciting, thanks for posting!
Iâll quickly mention a couple of things that stuck out to me that might make the CEA potentially overoptimistic:
IQ points lost per Îźg/âdl of leadâthis is likely a logarithmic relationship (as suggested by Bernard and Schukraft). For a BLL of 2.4 â 10 Îźg/âdl, IQ loss from 1 Îźg/âdl increase may be close to 0.5, but above 10, itâs closer to 0.2 per 1 Îźg/âdl increase, and above 20, closer to 0.1. Average BLL in Bangladesh seem to be around 6.8 Îźg/âdl, though amongst residents living near turmeric sources of lead, it could plausibly be (much) higher, and thus a lower IQ gain will be had from a 1 Îźg/âdl reduction in lead.
Income loss per IQ points lossâThe CEA assumes that 1 IQ point loss leads to a 2.1% reduction in income. However, some work by GiveWell (here) suggests this might be closer to 0.67% (and there might be some reasons to discount this further, e.g. due to replicability concerns)
Replicability of interventionâas noted in the text, itâs hard to estimate how much the Bangladesh program reduced lead exposure by. If Bangladeshâs average BLL level is around 6.8 Îźg/âdl, then a 1.64 reduction from the intervention implies this intervention cut BLL by 25% for half of children in Bangladesh. This is great, but I can see several reasons why this may not be informative of future programsâ cost-effectiveness
Maybe Turmeric is much more prevalent in rural Bangladesh than other regions
Maybe it was unusually easy to get regulators to agree to introduce /â enforce standards
Maybe it was unusually easy to get producers to switch away from lead chromate
Each of these reasons on their own is fairly weak, but the likelihood of at least one being true gives us reason to discount future cost-effectiveness analyses. More generally, we might expect some regression to the mean w.r.t reducing exposure from tulmericâmaybe everything went right for this particular program, but this is unlikely to be true in future programs. To be clear, there are likely also reasons that this analysis is too pessimistic, and thus on net it may be the case that cost-effectiveness remains at $1/â DALY (or even better). Nonetheless, I think itâs good to be cautious, since $1 /â DALY implies this program was >800x better than cash transfers and >80x better than GiveWellâs top charitiesâa strong claim to make (though still possible!)
My bad, thanks so much!
It would be great to have some way to filter for multiple topics.
Example: Suppose I want to find posts related to the cost-effectiveness of AI safety. Instead of just filtering for âAI safetyâ, or for just âForecasting and estimationâ, I might want to find posts only at the intersection of those two. I attempted to do this by customizing my frontpage feed, but this doesnât really work (since it heavily biases to new/âupvoted posts)
it relies primarily on heuristics like organiser track record and higher-level reasoning about plans.
I think this is mostly correct, with the caveat that we donât exclusively rely on qualitative factors and subjective judgement alone. The way Iâd frame it is more as a spectrum between
[Heuristics] <------> [GiveWell-style cost-effectiveness modelling]
I think Iâd place FPâs longtermist evaluation methodology somewhere between those two poles, with flexibility based on whatâs feasible in each cause
Iâll +1 everything Johannes has already said, and add that several people (including myself) have been chewing over the âhow to rate longtermist projectsâ question for quite some time. Iâm unsure when we will post something publicly, but I hope it wonât be too far in the future.
If anyone is curious for details feel free to reach out!
Quick take: renaming shortforms to Quick takes is a mistake
This looks super interesting, thanks for posting! I especially appreciate the âHow to applyâ section
One thing Iâm interested in is seeing how this actually looks in practiceâspecifying real exogenous uncertainties (e.g. about timelines, takeoff speeds, etc), policy levers (e.g. these ideas, different AI safety research agendas, etc), relations (e.g. between AI labs, governments, etc) and performance metrics (e.g âp(doom)â, plus many of the sub-goals you outline). What are the conclusions? What would this imply about prioritization decisions? etc
I appreciate this would be super challenging, but if you are aware of any attempts to do it (even if using just a very basic, simplifying model), Iâd be curious to hear how itâs gone
Should recent ai progress change the plans of people working on global health who are focused on economic outcomes?
I think so, see here or here for a bit more discussion on this
If you think that AI will go pretty well by default (which I think many neartermists do)
My guess/âimpression is that this just hasnât been discussed by neartermists very much (which I think is one sad side-effect from bucketing all AI stuff in a âlongtermistâ worldview)
Great question!
One can claim Gift Aid on a donation to the Patient Philanthropy Fund (PPF), e.g. if donating through Giving What We Can. So a basic rate taxpayer gets a 25% âreturnâ on the initial donation (via gift aid). The fund can then be expected to make a financial return equivalent to an index fund (~10% p.a for e.g. S&P 500).
So, if you buy the claim that your expected impact will be 9x larger in 10 years than today, then a ÂŁ1,000 donation today will have an expected (mean) impact of ÂŁ11,250, for longtermist causes (ÂŁ1,000 * 1.25 * 9)[1]
Therefore I think the question of:
âdonate now and claim gift aidâ OR âinvest then donate laterâ
...can be reframed as:
âdonate now and claim gift aidâ OR âdonate to (e.g. PPF) now and claim gift aid, for the PPF to invest and then donate laterâ
(I.e. I think gift aid considerations donât favour one option over the other)
Of course, one may reasonably disagree on giving now vs giving laterâthis is a much more messy question, and one that I wonât attempt to answer here.
Iâm not sure about paying into an organisationâs fund.
I think that conditional on giving later, the PPF is a better option than individually taking an âinvesting to giveâ approach (roughly for reasons described here)
(disclaimer: I work on the operations side of the PPF)
- ^
A ÂŁ1,000 donation becomes $1,250 for a basic rate taxpayer. Over 10 years, expected impact will increase by 9x (using the Investing to Give report modelâs mean estimate)
Using the same logic for global health or animal welfare, your expected (mean) impact from a ÂŁ1,000 donation in 10 years would be ÂŁ2,625 (ÂŁ1,000 * 1.25 * 2.1x) and ÂŁ5,250 (ÂŁ1,000 * 1.25 * 4.2x).
Note however that no âPPF equivalentâ for global health or animal welfare currently exists, AFAIK
- ^
I think this could be an interesting avenue to explore. One very basic way to (very roughly) do this is to model p(doom) effectively as a discount rate. This could be an additional user input on GiveWellâs spreadsheets.
So for example, if your p(doom) is 20% in 20 years, then you could increase the discount rate by roughly 1% per year
[Techinically this will be somewhat off since (Iâm guessing) most peopleâs p(doom) doesnât increase at a constant rate, in the way a fixed discount rate does.]
Rob Besinger of MIRI tweets:
...Iâm happy to say that MIRI leadership thinks âhumanity never builds AGIâ would be the worst catastrophe in history, would cost nearly all of the futureâs value, and is basically just unacceptably bad as an option.
Just to add that the Research Institute for Future Design (RIFD) is a Founders Pledge recommendation for longtermist institutional reform
(disclaimer: I am a researcher at Founders Pledge)
OpenPhil might be in a position to expand EAâs expected impact if it added a cause area that allowed for more speculative investments in Global Health & Development.
My impression is that Open Philanthropyâs Global Health and Development team already does this? For example, OP has focus areas on Global aid policy, Scientific research and South Asian air quality, areas which are inherently risky/âuncertain.
They have also take a hit based approach philosophically, and this is what distinguishes them from GiveWellâsee e.g.
Hits. We are explicitly pursuing a hits-based approach to philanthropy with much of this work, and accordingly might expect just one or two âhitsâ from our portfolio to carry the whole. In particular, if one or two of our large science grants ended up 10x more cost-effective than GiveWellâs top charities, our portfolio to date would cumulatively come out ahead. In fact, the dollar-weighted average of the 33 BOTECs we collected above is (modestly) above the 1,000x bar, reflecting our ex ante assessment of that possibility. But the concerns about the informational value of those BOTECs remain, and most of our grants seems noticeably less likely to deliver such âhitsâ.
[Reposting my comment here from previous version]
(FYI to othersâIâve just seen Ajeyaâs very helpful writeup, which has already partially answer this question!)