Did Eric Drexler not describe ideas like this in Engines of Creation? Either way, I would guess that Drexler has thought of similar ideas before (sans the phrase “diamondoid bacteria”) and has also likely communicated these ideas to Eliezer (albeit perhaps in an informal context). Though it’s also possible Eliezer came up with it independently, as it seems like a relatively natural idea to consider once you already assume diamondoid mechanosynthesis creating atomically-precise nanobots.
Daniel_Eth
Karma: 2,647
I don’t think this is a fair representation of what happened. The only thing that Eliezer appears to have “made up” is the name “diamondoid bacteria” – the idea of diamondoid mechanosynthesis doesn’t come from him, nor does the idea of applying diamondoid mechanosynthesis to building nanorobots, nor that bacteria provide a proof-of-concept for the possibility of nanorobots, nor that artificial nanorobots may share some resemblance to bacteria. Eliezer also doesn’t claim to have come up with any of these ideas. You can debate the merits of any of these ideas, as well as whether it was a smart move for Eliezer to come up with the term “diamondoid bacteria” (and his arguably unclear communication about having come up with the term), but that’s different from him having just made up the ideas.
Granted, in principle you could also have a situation where they’re less cautious than management but more cautious than policymakers and it winds up being net positive, though I think that situation is pretty unlikely. Agree the consideration you raised is worth paying attention to.
I think there are a few reasons why this is very unlikely to work in practice, at least in society today (maybe it would work if most people were longtermists of some sort):
Focusing on such far-off issues would be seen as “silly,” and would thereby hurt the political careers of those who focused on them
Political inertia is generally strong, meaning that, by default, most things in politics don’t happen, even if there’s slightly more support than opposition. Here, you’d be shifting the policy proposal to one that wouldn’t affect the lives of constituents, and taking away any argument for urgency, making it even harder for the policy to gather enough momentum to overcome the political inertia
Due to cognitive dissonance/motivated reasoning/reciprocity/etc, most people who benefit from policies that are on net harmful come to believe that the policy is actually good (e.g., rich homeowners conveniently becoming ideologically NIMBY), and so approaching this from a transactional perspective of “okay, we’ll let you personally benefit but let’s end the policy after that so it stop hurting everyone else” will likely fail to gain any headway with the current beneficiaries of the policy
Etcetera
As an aside, regarding this point:
For instance, voting researchers widely consider the presidential voting system in America to be inferior to many alternatives. But if you want to change it, you require consent from Democrats and Republicans—i.e. the very people who benefit from the status quo.
I decently strongly suspect that the main opposition to changing the presidential system in the US (say, to a parliamentary system) is not that the Democratic and Republican politicians would lose power (I think they’d wind up relatively similarly powerful under that system, though possibly parties would fragment), but instead some combination of status quo bias/Chesterton’s Fence conservatism/sentimentality towards the current system/political inertia/etc.
A union for AI workers such as data scientists, hardware and software engineers could organise labor to counterbalance the influence of shareholders or political masters.
It’s not obvious to me that AI workers would want a more cautious approach than AI shareholders, AI bosses, and so on. Whether or not this would be the case seems to me to be the main crux behind whether this would be net positive or net harmful.
Do people actually think that Google+OpenAI+Anthropic (for sake of argument) are the US? Do they think the US government/military can/will appropriate those staff/artefacts/resources at some point?
I’m pretty sure what most (educated) people think is they are part of the US (in the sense that they are “US entities”, among other things), that they will pay taxes in the US, will hire more people in the US than China (at least relative to if they were Chinese entities), will create other economic and technological spillover effects in greater amount in the US than in China (similar to how the US’s early lead on the internet did), will enhance the US’s national glory and morale, will provide strategically valuable assets to the US and deny these assets to China (at least in a time of conflict), will more likely embody US culture and norms than Chinese culture and norms, and will be subject to US regulation much more than Chinese regulation.
Most people don’t expect these companies will be nationalized (though that does remain a possibility, and presumably more so if they were Chinese companies than US companies, due to the differing economic and political systems), but there are plenty of other ways that people expect the companies to advantage their host country[’s government, population, economy, etc].
In my mind there are 2 main differences:
Economic degrowth is undesirable, vs pausing AI is at least arguably desirable – climate change is very unlikely to lead to a literal existential catastrophe, “business as usual” tech improvements and policy changes (i.e., without overthrowing capitalism) will likely lead to a clean energy transition as is, economic degrowth would probably kill many more people than it would save, etc. Meanwhile, AI presents large existential risk in my mind, and I think a pause would probably lower this risk by a non-negligible amount.
Economic degrowth is much less politically feasible than an AI pause – first, because people are loss averse, so degrowth would be taking something away from them vs a pause would just be asking them to forgo future progress; second, because the fear of actual existential risk (from AI) may motivate more extreme policies.
I will say, if I thought p(doom | climate) > 10%, with climate timelines of 12 years, then I would be in favor of degrowth policies that seemed likely to reduce this risk. I just think that in reality, the situation is very different than this.
So I notice Fox ranks pretty low on that list, but if you click through to the link, they rank very high among Republicans (second to only the weather channel). Fox definitely uses rhetoric like that. After Fox (among Republicans) are Newsman and OAN, which similarly both use rhetoric like that. (And FWIW, I also wouldn’t be super surprised to see somewhat similar rhetoric from WSJ or Forbes, though probably said less bluntly.)
I’d also note that the left-leaning media uses somewhat similar rhetoric for conservative issues that are supported by large groups (e.g., Trumpism in general, climate denialism, etc), so it’s not just a one-directional phenomena.
I don’t recall seeing many reputable publications label large-scale progressive movements (e.g., BLM, Extinction Rebellion, or #MeToo) as “uninformed mobs”
So progressive causes will generally be portrayed positively by progressive-leaning media, but conservative-leaning media, meanwhile, has definitely portrayed all those movements as ~mobs (especially for BLM and Extinction Rebellion), and predecessor movements, such as for Civli Rights, were likewise often portrayed as mobs by detractors. Now, maybe you don’t personally find conservative media to be “reputable,” but (at least in the US, perhaps less so in the UK) around half the power will generally be held by conservatives (and perhaps more than half going forward).
the piece has an underlying narrative of a covert group exercising undue influence over the government
My honest perspective is if you’re an lone individual affecting policy, detractors will call you a wannabe-tyrant, if you’re a small group, they’ll call you a conspiracy, and if you’re a large group, they’ll call you an uninformed mob. Regardless, your political opponents will attempt to paint your efforts as illegitimate, and while certain lines of criticism may be more effective than others, I wouldn’t expect scrutiny to simply focus on the substance either way.
I agree that we should have more of an outside game in addition to an inside game, but I’d also note that efforts at developing an outside game could similarly face harsh criticism (e.g., “appealing to the base instincts of random individuals, taking advantage of these individuals’ confusion on the topic, to make up for their own lack of support from actual experts”).
either because they see the specific interventions as a risk to their work, or because they feel policy is being influenced in a major way by people who are misguided
or because they feel it as a threat to their identity or self-image (I expect these to be even larger pain points than the two you identified)
“how well resourced scientific research institutes are and thus able to get organs that they need for research.”
Hmm, are they allowed to buy organs, though? Otherwise, the fact that they’re well resourced might not matter much for their access to organs.
My guess is this is mostly just a product of success, and insofar as the political system increasingly takes AI X-risk seriously, we should expect to see stuff like this from time to time. If the tables were flipped and Sunak was instead pooh-poohing AI X-risk and saying things like “the safest path forward for AI is accelerating progress as fast as we can – slowing down would be Luddism” then I wouldn’t be surprised to see articles saying “How Silicon Valley accelerationists are shaping Rishi Sunak’s AI plans”. Doesn’t mean we should ignore the negative pieces, and there very well may be things we can do to decrease it at the margin, but ultimately, I’d be surprised if there was a way around it. I also think it’s notable how much press there is that agrees with AI X-risk concerns; it’s not like there’s a consensus in the media that it should be dismissed.
Speaking personally, I think there is a possibility of money becoming obsolete, but I also think there’s a possibility of money mattering more, as (for instance) AI might allow for an easier ability to turn money into valuable labor. In my mind, it’s hard to know how this all shakes out on net.
I think there are reasons for expecting the value of spending to be approximately logarithmic with total spending for many domains, and spending on research seems to fit this general pattern pretty well, so I suspect that it’s prudent to generally plan to spread spending around a fair bit over the years.
I also just want to note that I wouldn’t expect this whole question to affect behavior of the LTFF much, as we decide what grants to fund, but we don’t make plans to save/invest money for future years anyways (though, of course, it could affect behavior of potential donors to the LTFF).
Annoy away – it’s a good question! Of course, standard caveats to my answer apply, but there’s a few caveats in particular that I want to flag:
It’s possible that by 2028 there will be one (or more) further longtermist billionaires who really open up the spigot, significantly decreasing the value of marginal longtermist money at that time
It’s possible that by 2028, AI would have gotten “weird” in ways that affect the value of money at that time, even if we haven’t reached AGI (e.g., certain tech stocks might have skyrocketed by then, or it might be possible to turn money into valuable research labor via AI)
You might be considering donation opportunities that significantly differ in value from other large funders in the field
This is all pretty opinionated and I’m writing it on the fly, so others on the LTFF may disagree with me (or I might disagree with myself if I thought about it at another time).
In principle, we could try to assign probability distributions to all the important cruxes and Monte Carlo this out. Instead, I’m just going to give my answer based on simplifying assumptions that we still have one major longtermist donor who prioritizes AI safety to a similar amount as today, things haven’t gotten particularly weird, your donation opportunities don’t look that different from others’ and roughly match donation opportunities now,[1] etc.
One marginal funding opportunity to benchmark the value of donations against would be funding the marginal AI alignment researcher, which probably costs ~$100k/yr. Assuming a 10% yearly discount rate (in line with the long-term, inflation-adjusted returns to equities within the US), funding this in perpetuity is equivalent to a lump-sum donation now of $1M, or a donation in 2028 of ($1M)*(1.1^5) = $1.6M.[2]
Then the question becomes, how valuable is the marginal researcher (and how would you expect to compare against them)? Borrowing from Linch’s piece on the value of marginal grants to the LTFF, the marginal alignment grant often is a bit better than something like the following:
a late undergraduate or recent graduate from an Ivy League university or a comparable institution requests a grant to conduct independent research or comparable work in a high-impact field, but we don’t find the specific proposal particularly compelling. For example, the mentee of a fairly prominent AI safety… researcher may request 6-12 months’ stipend to explore a particular research project that their mentor(s) are excited about, but LTFF fund managers and some of our advisors are unexcited about. Alternatively, they may want to take an AGISF course, or to read and think enough to form a detailed world model about which global catastrophic risks are the most pressing, in the hopes of then transitioning their career towards combating existential risk.
In these cases, the applicant often shows some evidence of interest and focus (e.g., participation in EA local groups/EA Global or existential risk reading groups) and some indications of above-average competence or related experience, but nothing exceptional.
[Note: I think this sort of grant is well below the current funding threshold for the LTFF, given that we’re currently in a funding crunch. But I would generally expect, for the longtermist community as a whole over longer timespans, the marginal grant would be only a bit higher than that.]
Note that for many of the people funded on that kind of grant, the main expected benefit would come not from the direct work of the initial grant, but instead on the chance that the researcher winds up being surprisingly good at alignment research; so, in considering the value of the “marginally-funded researcher,” just note that it would be someone with stronger signals for alignment research than described above.
So from this, if you think you’d be around equivalent to the marginally-funded alignment researcher (where it’s worth it to keep funding them based on their research output), I’d think your labor would be worth about $1-2M in donations in 2028.[3] It’s harder to estimate the value for people doing substantially better work than that. I think values 10x that would probably apply to a decent number of people, and numbers 100x higher would be rare but not totally unimaginable.
- ^
Or that current donors are appropriately weighing how much they should spend now vs invest so even if the nature of donation opportunities differs, the (investment-adjusted) value of the donations is comparable. Note that I’m not trying to claim they are actually doing this, but the assumption makes the analysis easier.
- ^
There’s a few simplifying assumptions here: I’m neglecting to consider how the cost of living/wages may raise this cost, and I’m also neglecting to consider how the labor wouldn’t last in perpetuity but only for the remainder of the person’s career (presumably either until they reach retirement, or until AI forces them into retirement or takes over).
- ^
In principle, a very-marginally worth it to fund person may be (on net) worth much less than this, since funding them would also cost the money to fund them. In practice, I think this rough calculation still gives us a good general ballpark for people to index on, as very few people are presumably almost exactly at the point of indifference.
So I’m imagining, for instance, AGIs with some shards of caring about human ~autonomy, but also other (stronger) shards that are for caring about (say) paperclips (also this was just meant as an example). I was also thinking that this might be what “a small population in a ‘zoo’” would look like – the Milky Way is small compared to the reachable universe! (Though before writing out my response, I almost wrote it as “our solar system” instead of “the Milky Way,” so I was imagining a relatively expansive set within this category; I’m not sure if distorted “pet” versions of humans would qualify or not.)
FWIW, I think specific changes here are unlikely to be cruxy for the decisions we make.
[Edited to add: I think if we could know with certainty that AGI was coming in 202X for a specific X, then that would be decision-relevant for certain decisions we’d face. But a shift of a few years for the 10% mark seems less decision relevant]
Another reason that the higher funding bar is likely increasing delays – borderline decisions are higher stakes, as we’re deciding between higher EV grants. It seems to me like this is leading to more deliberation per grant, for instance.
Presumably this will differ a fair bit for different members of the LTFF, but speaking personally, my p(doom) is around 30%,[1] and my median timelines are ~15 years (though with high uncertainty). I haven’t thought as much about 10% timelines, but it would be some single-digit number of years.
- ^
Though a large chunk of the remainder includes outcomes that are much “better” than today but which are also very suboptimal – e.g., due to “good-enough” alignment + ~shard theory + etc, AI turns most of the reachable universe into paperclips but leaves humans + our descendants to do what we want with the Milky Way. This is arguably an existential catastrophe in terms of opportunity cost, but wouldn’t represent human extinction or disempowerment of humanity in the same way as “doom.”
- ^
I was also surprised by how highly the EMH post was received, for a completely different reason – the fact that markets aren’t expecting AGI in the next few decades seems unbelievably obvious, even before we look at interest rates. If markets were expecting AGI, AI stocks would presumably be (much more, at least compared to non-AI stocks) to the moon than they are now, and market analysts would presumably (at least occasionally) cite the possibility of AGI as the reason why. But we weren’t seeing any of that, and we already knew from just general observation of the zeitgeist that, until a few months ago, the prospect of AGI was overwhelmingly not taken seriously outside of a few niche sub-communities and AI labs (how to address this reality has been a consistent, well-known hurdle within the AI safety community).
So I’m a little confused at what exactly judges thought was the value provided by the post – did they previously suspect that markets were taking AGI seriously, and this post significantly updated them towards thinking markets weren’t? Maybe instead judges thought that the post was valuable for some other reason unrelated to the main claim of “either reject EMH or reject AGI in the next few decades”, in which case I’d be curious to hear about what that reason is (e.g., if the post causes OP to borrow a bunch of money, that would be interesting to know).
Granted, it’s an interesting analysis, but that seems like a different question, and many of the other entries (including both those that did and didn’t win prizes) strike me as having advanced the discourse more, at least if we’re focusing on the main claims.