Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
I am very amenable to either of these. If someone is starting these, or if they are convinced that these could be super valuable, please do get in touch.
Also keen on this.
Specifically, I would be interested in someone carrying out an independent impact report for the APPG for Future Generations and could likely offer some funding for this.
Excellent list! On biodefence-relevant tech: One market screaming (actually, literally) for a solution is parents of small children. They get infected at a very high rate and I suspect this is via transmission routes in pre-schools that we would also be concerned about in a pandemic. While it is uncertain how a business addressing this market will actually shake out, I think it is generally in the direction of something likely to be biodefence-relevant. As a currently sick parent of small children myself, I would be very happy to start such a business after my current project comes to an end. And I think only ~40% reduction in sick days is enough to make this business viable and the current bar for hygiene is really low so I think it is feasible to achieve such reductions.
More generally, and in parallel to your proposed method of looking for markets that specific interventions could target, I would also consider looking for markets that roughly point in the direction of biodefence. The saying goes that it is more likely to have business success if focusing on a problem, rather than having a solution looking for a problem to solve. Another example from my personal history of markets roughly in the direction of biodefence is traveling abroad—many people have their long-planned and/or expensive holidays ruined due to stomach bugs, etc.
I think it’s probably true that teams inside of major labs are better placed to work on AI lab coordination broadly, and this post was published before news of the frontier models forum came out. Still, I think there is still room for coordination to promote AI safety outcomes between labs, e.g. something that brings together open-source actors. However, this project area is probably less tractable and neglected now than when we originally shared this idea.
The biodefense market opportunities dovetails nicely with the recently announced market shaping accelerator by UChicago—https://marketshaping.uchicago.edu/challenge/
The “Retrospective grant evaluations of longtermist projects” idea seems like something that would work really well in conjunction with an impact market, like Manifund. That — retroactive evaluations — must be done extremely well for impact markets to function.
Since this could potentially be a really difficult/expensive process, randomized conditional prediction markets could also help (full explanation here). Here’s an example scheme I cooked up:
Subsidize prediction markets on all of the following:
Conditional on Project A being retroactively evaluated by the Retroactive Evaluation Team (RET), how much impact will it have[1]?
Conditional on Project B being retroactively evaluated by the RET, how much impact will it have?
etc.
Then, randomly pick one project (say, Project G) to retroactively evaluate, and fund the retroactive evaluation of Project G.
For all the other projects’ markets, refund all of the investors and, to quote DYNOMIGHT, “use the SWEET PREDICTIVE KNOWLEDGE … for STAGGERING SCIENTIFIC PROGRESS and MAXIMAL STATUS ENHANCEMENT.”
Obviously, the amount of impact would need to be metricized in some way. Again obviously, this is an incredibly difficult problem that I’m handwaving away.
The one idea that comes to mind is evaluating n projects and ranking their relative impact, where n is a proper subset of the number of total projects greater than 1. Then, change the questions to “Conditional on Project A/B/C/etc being retroactively evaluated, will it be ranked highest?” That avoids actually putting a number on it, but it comes with its own host of problems
Yeah, we’d absolutely love to see (and fund!) retroactive evals of longtermist project—as Saul says, these are absolutely necessary for impact certs. For example, the ACX Minigrants impact certs round is going to need evals for distributing the $40k in retroactive funding. Scott is going to be the one to decide how the funding is divvied up, but I’d love to sponsor external evals as well.
1.
I really like this list. Lost of the ideas look very sensible.
I also really really value that you are doing prioritisation exercises across ideas and not just throwing out ideas that you feel sound nice without any evidence of background research (like FTX, and others, did). Great work!
– –
2.
Quick question about the research: Does the process consider cost-effectiveness as a key factor? For each of the ideas do you feel like you have a sense of why this thing has not happened already?
– –
3.
Some feedback on the idea here I know most about: policy field building (although admittedly from a UK not US perspective). I found the idea strong and was happy to see it on the list but I found reading the description of it unconvincing. I am not sure there is much point getting people to take jobs in government without giving them direction, strategic clarity, things to do to or leavers to pull to drive change. Policy success needs an ecosystem, some people in technocratic roles, some in government, some in external policy think tank style research, some in advocacy and lobby groups, etc. If this idea is only about directing people into government I am much less excited by it than a broader conception of field building that includes institution building and lobbying work.
Thanks, appreciate your comment and the compliment!
On your questions:
2. The research process does consider cost-effectiveness as a key factor – e.g., the weighted factor model we used included both an “impact potential” and a “cost” item, so projects were favoured if they had high estimated impact potential and/or a low estimated cost. “Impact potential” here means “impact with really successful (~90th percentile) execution” – we’re focusing on the extreme rather than the average case because we expect most of our expected impact to come from tail outcomes (but have a separate item in the model to account for downside risk). The “cost” score was usually based on a rough proxy, but the “impact potential” score was basically just a guess – so it’s quite different from how CE (presumably) uses cost-effectiveness, in that we don’t make an explicit cost-effectiveness estimate and in that we don’t consult evidence from empirical studies (which typically don’t exist for the kinds of projects we consider).
Re: “For each of the ideas do you feel like you have a sense of why this thing has not happened already?” – we didn’t consider this explicitly in the process (though it somewhat indirectly featured as part of considering tractability and impact potential). I feel like I have a rough sense for each of the projects listed – and we wouldn’t include projects where we didn’t think it was plausible that the project would be feasible, that there’d be a good founder out there etc. – but I could easily be missing important reasons. Definitely an important question – would be curious to hear how CE takes it into account.
3. Appreciate the input! The idea here wouldn’t be to just shove people into government jobs, but also making sure that they have the right context, knowledge, skills and opportunities to have a positive impact once there. I agree that policy is an ecosystem and that people are needed in many kinds of roles. I think it could make sense for an individual project to focus just/primarily on one or a few types of role (analogously to how the Horizon Institute focuses primarily on technocratic staffer and executive branch roles + think tank roles), but am generally in favour of high-quality projects in multiple policy-related areas (including advocacy/lobbying and developing think tank pipelines).
This is a really inspiring list, thanks for posting! I’m curating.
Thanks for publishing this! I added it to this list of impactful org/project ideas
Super post!
In response to “Independent researcher infrastructure”:
I honestly think the ideal is just to give basic income to the researchers that both 1) express an interest in having absolute freedom in their research directions, and 2) you have adequate trust for.
I don’t think much valuable gets done in the mode where people look to others to figure out what they should do. There are arguments, many of which are widely-known-but-not-taken-seriously, and I realise writing more about it here would take more time than I planned for.
Anyway, the basic income thing. People can do good research on 30k USD a year. If they don’t think that’s sufficient for continuing to work on alignment, then perhaps their motivations weren’t on the right track in the first place. And that’s a signal they probably weren’t going to be able to target themselves precisely at what matters anyway. Doing good work on fuzzy problems requires actually caring.
People can do good research on even less than 30k USD a year at CEEALAR (EA Hotel).
Well, such a low pay creates additional mental pressure to resist temptation to get 5-10x money in a normal job. I’d rather select people carefully, but then provide them with at least a ~middle class wage
The problem is that if you select people cautiously, you miss out on hiring people significantly more competent than you. The people who are much higher competence will behave in ways you don’t recognise as more competent. If you were able to tell what right things to do are, you would just do those things and be at their level. Innovation on the frontier is anti-inductive.
If good research is heavy-tailed & in a positive selection-regime, then cautiousness actively selects against features with the highest expected value.[1]
That said, “30k/year” was just an arbitrary example, not something I’ve calculated or thought deeply about. I think that sum works for a lot of people, but I wouldn’t set it as a hard limit.
Based on data sampled from looking at stuff. :P Only supposed to demonstrate the conceptual point.
Your “deference limit” is the level of competence above your own at which you stop being able to tell the difference between competences above that point. For games with legible performance metrics like chess, you get a very high deference limit merely by looking at Elo ratings. In altruistic research, however...
I’m sorry I didn’t express myself clearly. By “select people carefully”, I meant selecting for correct motivations, that you have tried to filter for using the subsistence salary. I would prefer using some other selection mechanism (like references), and then provide a solid paycheck (like MIRI does).
It’s certainly noble to give away everything beyond 30k like Singer and MacAskill do, but I think it should be a choice rather than a requirement.
This is helpful, thanks!
I notice you didn’t mention fundraising for AI safety.
Recently, many have mentioned that the funding bar for AI safety projects has increased quite a bit (especially for projects not based in the Bay and not already well connected to funders) and response times from funders such as EA Funds LTFF can be very long (median 2 months afaik), which suggests we should look for more additional funding sources such as new high net worth donors, governments, non-EA foundations etc.
Do you have any thoughts on that? How valuable does this seem to you compared to your ideas?
Thanks for the question. At the time we were generating the initial list of ideas, it wasn’t clear that AI safety was funding-constrained rather than talent-constrained (or even idea-constrained). As you’ve pointed out, it seems more plausible now that finding additional funding sources could be valuable for a couple of reasons:
Helps respond to the higher funding bar that you’ve mentioned
Takes advantage of new entrants to AI-safety-related philanthropy, notably the mainstream foundations that have now become interested in the space.
I don’t have a strong view on whether additional funding should be used to start a new fund or if it is more efficient to direct it towards existing grantmakers. I’m pretty excited about new grantmakers like Manifund getting set up recently that are trying out new ways for grantmakers to be more responsive to potential grantees. I don’t have a strong view about whether ideas around increasing funding for AI safety are more valuable than those listed above. I’d be pretty excited about the right person doing something around educating mainstream donors about AI safety opportunities.
Did anyone consider ELOing the longtermist projects for “Retrospective grant evaluations”. Seems like it could be relatively fast by crowdsourcing comparisons, and orthogonal to other types of estimates
Thanks for sharing!
Is there any reason for not sharing the scores of the weighted factor model?
The quick explanation is that I don’t want people to over-anchor on it, given that the inputs are extremely uncertain, and that I think that a ranked list produced by a relatively well-respected research organisation is the kind of thing people could very easily over-anchor on, even if you caveat it heavily
Regarding AI lab coordination, it seems like the governance teams of major labs are a lot better placed to help with this, since they will have an easier time getting buy in from their own lab as well as being listened to by other labs. Also, the frontier models forum seems to be aiming at exactly this.
I’m glad to see that growing the field of AI governance is being prioritized. I would (and in a series of posts elsewhere on the forum, will) argue that, as part of the field development, the distinction between domestic and international governance could be valuable. Both can be covered by field-growing initiatives of course, but each involve distinct expertises and probably have different enough candidate profiles that folks thinking of getting into either may need to pursue divergent paths at a certain point.
Of course, I may be speaking too much from “inside” the field as it were. Was this distinction considered when articulating that point? Or does that strike you as too fine a point to focus on at this stage?
Hi Matt, I think it’s right that there’s some distinction between domestic and international governance. Unless otherwise specified, our project ideas were usually developed with the US in mind. When evaluating the projects, I think our overall view was (and still is) that the US is probably the most important national actor for AI risk outcomes and that international governance around AI is substantially less tractable since effective international governance will need to involve China. I’d probably favour more effort going into field-building focused on the US, then the EU, then the UK, in that order, before focusing on field-building initiatives aimed at international orgs.
In the short term, it seems like prospects for international governance on AI are low, with the political gridlock in the UN since the Russian invasion of Ukraine. I think there could be some particular international governance opportunities that are high-leverage, e.g. making the OECD AI incidents database very good, but we haven’t looked into that much.
You’ve done some wonderful research and this is a great list. It does seem very heavy on AI, with only a few projects addressing other existential risks. I’d be curious to know about some other ideas that didn’t make the cut for this post, but are aimed at addressing other risks.
Thank you!
Worth noting that our input was also very unevenly distributed – our original idea list includes ~40% AI-related ideas, ~15% bio, ~25% movement building / community infrastructure, and only ~20% other. (this was mainly due to us having better access to AI-related project ideas via our networks). If you’re interested in pursuing biosecurity- or movement building-related projects, feel free to get in touch and I can share some of our additional ideas – for the other areas I think we don’t necessarily have great ideas.
This list seems interesting and reasonable, but is it really justified to call it “research” when it’s essentially three persons sitting together for a few days doing a bit of thinking?