AI strategy & governance. Blog: Not Optional.
Zach Stein-Perlman
Karma: 4,410
It’s our policy to not discuss the specifics of people’s applications with other people besides them. I don’t think it would be appropriate for me to give more detail about why you were rejected publicly, so it is hard to really reply to the substance of this post, and share the other side of this story.
This of course is correct as a default policy. But if Constance explicitly said she wants to have this conversation more publicly, would you comment publicly? Or could you comment in a private message to her, and endorse her sharing the message if she chose to?
(Good luck with EAG DC in the meantime.)
Data point: in the three cases I know of, undergraduates with around 30–40 hours of engagement with EA ideas were accepted to EAG London 2022.
I’ve left AI Impacts; I’m looking for jobs/projects in AI governance. I have plenty of runway; I’m looking for impact, not income. Let me know if you have suggestions!
(Edit to clarify: I had a good experience with AI Impacts.)
PSA about credentials (in particular, a bachelor’s degree): they’re important even for working in EA and AI safety.
When I dropped out of college to work on AI safety, I thought credentials are mostly important as evidence-of-performance, for people who aren’t familiar with my work, and are necessary in high-bureaucracy institutions (academia, government). It turns out that credentials are important—for working with even many people who know you (such that the credential provides no extra evidence) and are willing to defy conventions—for rational, optics-y reasons. It seems even many AI governance professionals/orgs are worried (often rationally) about appearing unserious by hiring or publicly-collaborating-with the uncredentialed, or something. Plus irrationally-credentialist organizations are very common/important, and may even comprise a substantial fraction of EA jobs and x-risk-focused AI governance jobs (which I expected to be more convention-defying), and sometimes an organization/institution is credentialist even when it’s led by weird AI safety people (those people operate under constraints).
Disclaimer: the evidence-from-my-experiences for these claims is pretty weak. This point’s epistemic status is more considerations + impressions from a few experiences than facts/PSA.
Upshot: I’d caution people against dropping out of college to increase impact unless they have a great plan.
(Edit to clarify: this paragraph is not about AI Impacts — it’s about everyone else.)
I used a diff checker to find the differences between the current post and the original post. There seem to be two:
“Alice worked there from November 2021 to June 2022” became “Alice travelled with Nonlinear from November 2021 to June 2022 and started working for the org from around February”
“using Lightcone funds” became “using personal funds”
So it seems Kat’s comment is wrong and Emerson’s is misleading/wrong. They are free to point to another specific edit if it exists.
Update: Kat guesses she was thinking of changes from a near-final draft rather than changes from the first published version.
This was the press release; the actual order has now been published.
One safety-relevant part:
4.2. Ensuring Safe and Reliable AI. (a) Within 90 days of the date of this order, to ensure and verify the continuous availability of safe, reliable, and effective AI in accordance with the Defense Production Act, as amended, 50 U.S.C. 4501 et seq., including for the national defense and the protection of critical infrastructure, the Secretary of Commerce shall require:
(i) Companies developing or demonstrating an intent to develop potential dual-use foundation models to provide the Federal Government, on an ongoing basis, with information, reports, or records regarding the following:
(A) any ongoing or planned activities related to training, developing, or producing dual-use foundation models, including the physical and cybersecurity protections taken to assure the integrity of that training process against sophisticated threats;
(B) the ownership and possession of the model weights of any dual-use foundation models, and the physical and cybersecurity measures taken to protect those model weights; and
(C) the results of any developed dual-use foundation model’s performance in relevant AI red-team testing based on guidance developed by NIST pursuant to subsection 4.1(a)(ii) of this section, and a description of any associated measures the company has taken to meet safety objectives, such as mitigations to improve performance on these red-team tests and strengthen overall model security. Prior to the development of guidance on red-team testing standards by NIST pursuant to subsection 4.1(a)(ii) of this section, this description shall include the results of any red-team testing that the company has conducted relating to lowering the barrier to entry for the development, acquisition, and use of biological weapons by non-state actors; the discovery of software vulnerabilities and development of associated exploits; the use of software or tools to influence real or virtual events; the possibility for self-replication or propagation; and associated measures to meet safety objectives; and
(ii) Companies, individuals, or other organizations or entities that acquire, develop, or possess a potential large-scale computing cluster to report any such acquisition, development, or possession, including the existence and location of these clusters and the amount of total computing power available in each cluster.
(b) The Secretary of Commerce, in consultation with the Secretary of State, the Secretary of Defense, the Secretary of Energy, and the Director of National Intelligence, shall define, and thereafter update as needed on a regular basis, the set of technical conditions for models and computing clusters that would be subject to the reporting requirements of subsection 4.2(a) of this section. Until such technical conditions are defined, the Secretary shall require compliance with these reporting requirements for:
(i) any model that was trained using a quantity of computing power greater than 1026 integer or floating-point operations, or using primarily biological sequence data and using a quantity of computing power greater than 1023 integer or floating-point operations; and
(ii) any computing cluster that has a set of machines physically co-located in a single datacenter, transitively connected by data center networking of over 100 Gbit/s, and having a theoretical maximum computing capacity of 1020 integer or floating-point operations per second for training AI.
I’m a fan of donation swapping, but I don’t think this is legal under US campaign finance law. (If someone else knows for sure, please tell us. Edit: Peter says it’s not legal. Edit: Peter now only says it’s illegal for non-Americans, but I also think it’s illegal for Americans as a way to get around the individual contribution limit.)
Edit, meta: several people have downvoted Caleb’s comment after he no longer endorsed it, and some have downvoted his reply too (both to below zero). This isn’t right, epistemically or in terms of desert. Downvoting a retracted idea doesn’t improve the conversation, and Caleb’s comments are clearly good- and truth-seeking. If you want to punish the author for saying something that turned out to be unpopular, you should consider the effects of that policy (here and more generally) on the community’s epistemic culture. See also Oliver’s comment.
Edit: Caleb’s comments are safely back in nonnegative territory, for now, but I’ll leave the above note since it’s still worth saying.
I don’t know him, but I really want Carrick in Congress. I think donating to his campaign is a not-unreasonable thing to do as a hits-based-giving-opportunity, since it would be great for him to be in the House...
...but I don’t think most readers of this post would appreciate how unlikely he is to win. People who would be great politicians often aren’t great candidates. Unless there’s relevant private information (e.g., he’s expecting endorsements from major Democrats), it’s quite unlikely that Carrick—whose name isn’t known in the district and who doesn’t have government experience—will get more votes in the Democratic primary than the candidates with name recognition, state government experience, and endorsements from many state government officials. (And if he doesn’t win, marginal performance in a failed House primary isn’t very helpful to future pursuits.) I wish I knew how Carrick plans to win: I wish we lived in the world where you could win elections just by dazzling voters with your policy chops, but we don’t.
I’m an elections junkie. I wish I could vote for Carrick, and I really hope he wins. But I would not feel comfortable recommending others donate or volunteer until his campaign gives us reason to believe that he has a real chance.
Edit, 2.5 days later. I still think Carrick’s chances are pretty low and that some people in these comments are excessively optimistic due to misweighting the relevant factors (which is ok—not everyone needs to be knowledgeable about elections—but skews the sentiment in these comments). And I still have meta-level concerns about how we decide to pursue certain interventions (which I plan to share after the primary). But I now think that donating is a highly effective thing to do in expectation (although on balance I would rather give to the Long-Term Future Fund), because it seems quite high-value for Carrick to win.
I’m a noted fan of Holly, but I disagree with your hagiography.
Her published work at RP seems typical
I’m not familiar with her work at Harvard EA and I doubt you are either
Elsewhere you say “Holly has launched a GoFundMe to fund her work independently; it’s this kind of entrepreneurial spirit that gives me confidence she’ll do well as a movement organizer!”; I disagree that launching a GoFundMe is nontrivial evidence of success
I feel like I detect a missing mood from you where you’re skeptical of pausing (for plausible-to-me reasons), but you’re not conflicted about it like I am and you don’t e.g. look for ways to buy time or ways for regulation to help without the downsides of a pause. (Sorry if this sounds adversarial.) Relatedly, this post is one-sided and so feels soldier-mindset-y. Likely this is just due to the focus on debating AI pause. But I would feel reassured if you said you’re sympathetic to: labs not publishing capabilities research, labs not publishing model weights, dangerous-capability-model-eval-based regulation, US and allies slowing other states and denying them compute, and/or other ways to slow AI or for regulation to help. If you’re unsympathetic to these, I would doubt that the overhang nuances you discuss are your true rejection (but I’d be interested in hearing more about your take on slowing and regulation outside of “pause”).
Edit: man, I wrote this after writing four object-level comments but this got voted to the top. Please note that I’m mostly engaging on the object level and I think object-level discussion is generally more productive—and I think Nora’s post makes several good points.
From the comments in the NYT, two notes on communicating longtermism to people-like-NYT-readers:
Many readers are confused by the focus on humans.
Some readers are confused by the suggestion that longtermism is weird (Will: “It took me a long time to come around to longtermism”) rather than obvious.
Re 2, I do think it’s confusing to act like longtermism is nonobvious unless you’re emphasizing weird implications like our calculations being dominated by the distant future and x-risk and things at least as weird as digital minds filling the universe.
I agree that there are some psychological and practical benefits to making and having more money, but I don’t think you’re “essentially donating 98–99%,” since even if you create value 50–100 times your salary, there’s no way for you to capture 50–100 times your salary, even if you were totally selfish. The fraction you’re “essentially donating” is more like , where “max possible salary” is the amount you would earn if you were totally selfish.
Carrick repeatedly said that he’s not a crypto guy, hasn’t met SBF, etc.
He didn’t hide his pandemic-preparedness agenda—in fact, he ran on it as a major qualification
Bostrom [claims] that even a small credence in the VWH requires continuously controlling and surveilling everyone on earth.
Bostrom does not claim this. Period. (Your reading is good-faith, but Bostrom is frequently misread on this topic by people criticizing him in bad faith, so it’s worth emphasizing—and it’s just an important point.) He narrowly claims here that mass surveillance would be necessary given “a biotechnological black ball that is powerful enough that a single malicious use could cause a pandemic that would kill billions of people.”
Another relevant Bostrom quote:
Comprehensive surveillance and global governance would thus offer protection against a wide spectrum of civilizational vulnerabilities. This is a considerable reason in favor of bringing about those conditions. The strength of this reason is roughly proportional to the probability that the vulnerable world hypothesis is true.
He goes on to discuss the downsides of surveillance and global governance. So your quotes like “Bostrom’s plan [purports to be a] one-size-fits-all antidote” are not correct, and Bostrom would agree with you that totalitarianism and surveillance present astronomical risks.
I happen to disagree that possible interventions that greatly improve the expectation of the long-term future will soon all be taken. But regardless, the first quote is just about value, not about what we ought to do.
I think the second principle is basically true. Since the long-term future is 10^(big number) times bigger than the short-term future, our effects on the short-term future mostly matter insofar as they affect the long-term future, unless we have reason to believe that long-term effects somehow cancel out exactly. You’re right that humans are not psychologically capable of always following it directly, but we can pursue proxies and instrumental goals that we think improve the long-term future. (But also, this principle is about describing our actions, not telling us what to do, so what’s relevant isn’t our capability to estimate long-term effects, but rather what we would think about our actions if we were omniscient.)
The conjunction
AGI has existed for at least 5 years.
Technology is advancing rapidly and AI is transforming the world sector by sector.
would be quite surprising to me, since I strongly expect superintelligence within a couple years after AGI, and I strongly expect a technological singularity at that time. So I do not believe that a story consistent with the rules can be plausible. (I also expect more unipolarity by 5 years after AGI, but even multipolar scenarios don’t give us a future as prosaic as the rules require.)
Common beliefs/attitudes/dispositions among [highly engaged EAs/rationalists + my friends] which seem super wrong to me:
Meta-uncertainty:
Giving a range of probabilities when you should give a probability + giving confidence intervals over probabilities + failing to realize that probabilities of probabilities just reduce to simple probabilities
But thinking in terms of probabilities over probabilities is sometimes useful, e.g. you have a probability distribution over possible worlds/models and those worlds/models are probabilistic
Unstable beliefs about stuff like AI timelines in the sense of I’d be pretty likely to say something pretty different if you asked tomorrow
Instability in the sense of being likely to change beliefs if you thought about it more is fine; fluctuating predictably (dutch-book-ably) is not
Ethics:
Axiologies besides ~utilitarianism
Possibly I’m actually noticing sloppy reasoning about how to go from axiology to decision procedure, possibly including just not taking axiology seriously
Veg(etari)anism for terminal reasons; veg(etari)anism as ethical rather than as a costly indulgence
Thinking personal flourishing (or something else agent-relative) is a terminal goal worth comparable weight to the impartial-optimization project
Cause prioritization:
Cause prioritization that doesn’t take seriously the cosmic endowment is astronomical, likely worth >10^60 happy human lives and we can nontrivially reduce x-risk
E.g. RP’s Cross-Cause Cost-Effectiveness Model doesn’t take the cosmic endowment seriously
Deciding in advance to boost a certain set of causes [what determines that set??], or a “portfolio approach” without justifying the portfolio-items
E.g. multiple CEA staff donate by choosing some cause areas and wanting to help in each of those areas
Related error: agent-relativity
Related error: considering difference from status quo rather than outcomes in a vacuum
Related error: risk-aversion in your personal contributions (much more egregious than risk-averse axiology)
Instead you should just argmax — find the marginal value of your resources in each cause (for your resources that can funge between causes), then use them in the best possible way
Intra-cause offsetting: if you do harm in area X [especially if it’s avoidable/unnecessary/intentional], you should fix your harm in that area, even if you could do more good in another area
Maybe very few of my friends actually believe this
Misc:
Not noticing big obvious problems with impact certificates/markets
Naively using calibration as a proxy for forecasting ability
Thinking you can (good-faith) bet on the end of the world by borrowing money
Many examples, e.g. How to place a bet on the end of the world
I think most of us understand the objection you can do better by just borrowing money at market rates — I think many people miss that utility is about ∫consumption not ∫bankroll (note the bettor typically isn’t liquidity-constrained). The bet only makes sense if you spend all your money before you’d have to pay back.
[Maybe something deep about donations; not sure]
[Maybe something about compensating altruists or compensating for roles often filled by altruists; not sure]
[Maybe something about status; not sure]
Possibly I’m wrong about which attitudes are common.
For now I’m just starting a list, not trying to be legible, much less change minds. I know I haven’t explained my views.
Edit: I’m sharing controversial beliefs, without justification and with some framed provocatively. If one of these views makes you think worse of me to a nontrivial degree, please ask for elaboration; maybe there’s miscommunication or it’s more reasonable than it seems. Edit 2: there are so many comments; I may not respond to requests-for-elaboration but will at least notice them as a bid-for-elaboration-at-some-point.
Wait what. What alternative is supposed to be better (in general or for solving the there’s a bad actor but many people don’t know problem)?
This all seems reasonable. But here’s why I’m going to continue thinking of myself as a longtermist.
(1) I generally agree, but at some point you become sufficiently certain in a set of beliefs, and they are sufficiently relevant to your decisions, that it’s useful to cache them. I happen to feel quite certain in each of the four beliefs you mention in (0), such that spending time questioning them generally has little expected value. But if given reason to doubt one, of course my identity “longtermist” shouldn’t interfere, and I think I’m sufficiently epistemically virtuous that I’d update.
(2) I agree that we should beware of “group-belief baggage.” Yesterday I told some EA friends that I strongly identify as an EA and with the EA project while being fundamentally agnostic on the EA movement/community. If someone identifies as an EA, or longtermist, or some other identity not exclusively determined by beliefs, I think they should be epistemically cautious. But I think it is possible to have such an identity while maintaining caution.
Put another way, I think your argument proves too much: it’s not clear what distinguishes “longtermist” from almost any other identity that implicates beliefs, so an implication of your argument seems to be that you shouldn’t adopt any such identity. And whatever conditions could make it reasonable for a person to adopt an identity, I think those conditions hold for me and longtermism.
(And when I think “I am a longtermist,” I roughly mean “I think the long-term future is a really big deal, both independent of my actions and in terms of what I should do, and I aspire to act accordingly.”)
I used a diff checker to find the differences between the current post and the original post. There seem to be two:
“Alice worked there from November 2021 to June 2022” became “Alice travelled with Nonlinear from November 2021 to June 2022 and started working for the org from around February”
“using Lightcone funds” became “using personal funds”
Your claim
Importantly, his original post (he’s quietly redacted a lot of points since publishing) had a lot of falsehoods that he knew were not true. He has since removed some of them after the fact, but those have still been causing us damage.
seems false. Possibly I made a mistake, or Ben made edits and you saw them and then Ben reverted them—if so, I encourage you/anyone to point to another specific edit, possibly on other archive.org versions.
Update: Kat guesses she was thinking of changes from a near-final draft rather than changes from the first published version.
Thanks for this post.
A few data points and reactions from my somewhat different experiences with EA:
I’ve known many EAs. Many have been vegan and many have not (I’m not). I’ve never seen anyone “treat [someone] as non-serious (or even evil)” based on their diet.
A significant minority achieves high status across EA contexts while loudly disagreeing with utilitarianism.
You claim that EA takes as given “Not living up to this list is morally bad. Also sort of like murder.” Of course failing to save lives is sort of like murder, for sufficiently weak “sort of.” But at the level of telling people what to do with their lives, I’ve always seen community leaders endorse things like personal wellbeing and non-total altruism (and not always just for instrumental, altruistic reasons). The rank-and-file and high-status alike talk (online and offline) about having fun. The vibe I get from the community is that EA is more of an exciting opportunity than a burdensome obligation. (Yes, that’s probably an instrumentally valuable vibe for the community to have—but that means that ‘having fun is murder’ is not endorsed by the community, not the other way around.)
[Retracted; I generally support noting disagreements even if you’re not explaining them; see Zvi’s reply]
It feels intellectually lazy to “strongly disagree” with principles like “The best way to do good yourself is to act selflessly to do good” and then not explain why. To illustrate, here’s my confused reading of your disagreement. Maybe you narrowly disagree that selflessness is the optimal psychological strategy for all humans. But of course EA doesn’t believe that either. Maybe you think it does. Or maybe you have a deeper criticism about “The best way to do good yourself”… but I can’t guess what that is.Relatedly, you claim that you are somehow not allowed to say real critiques. “There are also things one is not socially allowed to question or consider, not in EA in particular but fully broadly. Some key considerations are things that cannot be said on the internet, and some general assumptions that cannot be questioned are importantly wrong but cannot be questioned.” “Signals are strong [real criticism] is unwelcome and would not be rewarded.” I just don’t buy it. My experiences strongly suggest that the community goes out of its way to be open to good-faith criticism, in more than a pat-ourselves-on-the-back-for-being-open-to-criticism manner. I guess insofar as you have had different experiences that you decline to discuss explicitly, fine, you’ll arrive at different beliefs. But your observations aren’t really useful to the rest of us if you don’t say them, including the meta-observation that you’re supposedly not allowed to say certain things.
I think you gesture toward useful criticism. It would be useful for me if you actually made that criticism. You might change my mind about something! This post doesn’t seem written to make it easy for even epistemically virtuous EAs to be able to change their minds, even if you’ve correctly identified some good criticism, though, since you don’t share it.