When people write “more dakka,” do they simply meaning that we need to try harder and/or try more things? I’ve seen this in two or three pieces of writing on the EA Forum, but I’ve never seen a clear explanation. Apparently “dakka” is slang from a sci-fi video game/tabletop RPG? Is this useless in-group terminology, or does this actually have value?
As best I can tell, “more dakka” is a reference to this quote. Can anyone point me to a more clear or authoritative explanation?
We know the solution. Our bullets work. We just need more. We need More (and better) (metaphorical) Dakka – rather than firing the standard number of metaphorical bullets, we need to fire more, absurdly more, whatever it takes until the enemy keels over dead.
So if you’re doing something, and it isn’t working well enough, here’s a dozen ways to generate more dakka, and how each could apply if you’re a) exercising, or b) learning new mathematics.
A Dozen Ways
Do it again.
Instead of doing one set of repetitions of the exercise, do two.
If you read the chapter once, read it again.
Use more.
If you were lifting 10 pounds, lift 15.
If you were doing easy problems, do harder ones.
Do more repetitions.
Instead of 10 repetitions, do 15.
If you did 10 problems on the material, do 15.
Increase intensity.
Do your 15 repetitions in 2 minutes instead of 3.
If you were skimming or reading quickly, read more slowly.
Schedule it.
Exercise at a specific time on specific days. Put it on your calendar, and set reminders.
Make sure you have time scheduled for learning the material and doing problems.
Do it regularly.
Make sure you exercise twice a week, and don’t skip.
Make sure you review what you did previously, on a regular basis.
Do it for a longer period.
Keep exercising for another month.
Go through another textbook, or find more problem sets to work through.
Add types.
In addition to push-ups, do bench presses, chest flyers, and use resistance bands.
In addition to the problem sets, do the chapter review exercises, and work through the problems in the chapter on your own.
Expand the repertoire.
Instead of just push–ups, do incline push ups, loaded push-ups, and diamond push-ups.
Find (or invent!) additional problem types; try to prove things with other methods, find different counter-examples or show why a relaxed assumption means the result no longer holds, find pre-written solutions and see if you can guess next steps before reading them.
Add variety.
Do leg exercises instead of just chest exercises. Do cardio, balance, and flexibility training, not just muscle building.
Do adjacent types of mathematics, explore complex analysis, functional analysis, and/or harmonic analysis.
Add feedback.
Get an exercise coach to tell you how to do it better.
Get someone to grade your work and tell you what you’re doing wrong, or how else to learn the material.
Add people.
Have the whole team exercise. Find a group, gym, or exercise class.
Collaborate with others in solving problems. Take a course instead of self-teaching. Get others to learn with you, or teach someone else to solidify your understanding.
I’ve substantially revised my views on QURI’s research priorities over the past year, primarily driven by the rapid advancement in LLM capabilities.
Previously, our strategy centered on developing highly-structured numeric models with stable APIs, enabling:
Formal forecasting scoring mechanisms
Effective collaboration between human forecasting teams
Reusable parameterized world-models for downstream estimates
However, the progress in LLM capabilities has updated my view. I now believe we should focus on developing and encouraging superior AI reasoning and forecasting systems that can:
Generate high-quality forecasts on-demand, rather than relying on pre-computed forecasts for scoring
Produce context-specific mathematical models as needed, reducing the importance of maintaining generic mathematical frameworks
Leverage repositories of key insights, though likely not in the form of formal probabilistic mathematical models
This represents a pivot from scaling up traditional forecasting systems to exploring how we can enhance AI reasoning capabilities for forecasting tasks. The emphasis is now on dynamic, adaptive systems rather than static, pre-structured models.
(I rewrote with Claude, I think it’s much more understandable now)
Quick list of some ideas I’m excited about, broadly around epistemics/strategy/AI.
1. I think AI auditors / overseers of critical organizations (AI efforts, policy groups, company management) are really great and perhaps crucial to get right, but would be difficult to do well.
2. AI strategists/tools telling/helping us broadly what to do about AI safety seems pretty safe.
3. In terms of commercial products, there’s been some neat/scary military companies in the last few years (Palantir, Anduril). I’d be really interested if there could be some companies to automate core parts of the non-military government. I imagine there are some parts of the government that are particularly tractable/influenceable/tractable. For example, just making great decisions on which contractors the government should work with. There’s a ton of work to do here, between the federal government / state government / local government.
6. I think there are a lot of interesting ways for us to experiment with [AI tools to help our research/epistemics]. I want to see a wide variety of highly creative experimentation here. I think people are really limiting themselves in this area to a few narrow conceptions of how AI can be used in very specific ways that humans are very comfortable with. For example, I’d like to see AI dashboards of “How valuable is everything in this space” or even experiments where AIs negotiate on behalf of people and they use the result of that. A lot of this will get criticized for being too weird/disruptive/speculative, but I think that’s where good creative works should begin.
7. Right now, I think the field of “AI forecasting” is actually quite small and constrained. There’s not much money here, and there aren’t many people with bold plans or research agendas. I suspect that some successes / strong advocates could change this.
8. I think that it’s likely that Anthropic (and perhaps Deepmind) would respond well to good AI+epistemics work. “Control” was quickly accepted at Anthropic, for example. I suspect that it’s possible that things like the idea of an “Internal AI+human auditor” or an internal “AI safety strategist” could be adopted if done well.
One thing this makes me curious about: how good is the existing evidence base on electric stunning is better for the welfare of the shrimp, and how much better is stunning? I didn’t realize SWP was thinking of using the corporate campaign playbook to scale up stunning, so it makes me curious how robustly good this intervention is, and I couldn’t quickly figure this out from the Forum / website. @Aaron Boddy🔸 is there a public thing I can read by any chance? No pressure!
FWIW, “how good is stunning for welfare” is the main question I have about the impact of HSI right now (whereas other crucial considerations like “how many shrimp do you target”, “will farmers use the stunners”, “how should we value shrimp / their expected sentience” all feel clearer to me).
FYI rolling applications are back on for the Biosecurity Forecasting Group! We have started the pilot and are very excited about our first cohort! Don’t want to apply but have ideas for questions? Submit them here (anyone can submit!).
A reflection on the posts I have written in the last few months, elaborating on my views
In a series of recent posts, I have sought to challenge the conventional view among longtermists that prioritizes the empowerment or preservation of the human species as the chief goal of AI policy. It is my opinion that this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.
I recognize that my position is controversial and likely to remain unpopular among effective altruists for a long time. Nevertheless, I believe it is worth articulating my view at length, as I see it as a straightforward application of standard, common-sense utilitarian principles that merely lead to an unpopular conclusion. I intend to continue elaborating on my arguments in the coming months.
My view follows from a few basic premises. First, that future AI systems are quite likely to be moral patients; second, that we shouldn’t discriminate against them based on arbitrary distinctions, such as their being instantiated on silicon rather than carbon, or having been created through deep learning rather than natural selection. If we insist on treating AIs fundamentally differently from a human child or adult—for example, by regarding them merely as property to be controlled or denying them the freedom to pursue their own goals—then we should identify a specific ethical reason for our approach that goes beyond highlighting their non-human nature.
Many people have argued that consciousness is the key quality separating humans from AIs, thus rendering any AI-based civilization morally insignificant compared to ours. They maintain that consciousness has relatively narrow boundaries, perhaps largely confined to biological organisms, and would only arise in artificial systems under highly specific conditions—for instance, if one were to emulate a human mind in digital form. While I acknowledge that this perspective is logically coherent, I find it deeply unconvincing. The AIs I am referring to when I write about this topic would almost certainly not be simplistic, robotic automatons; rather, they would be profoundly complex, sophisticated entities whose cognitive abilities rival or exceed those of the human brain. For anyone who adopts a functionalist view of consciousness, it seems difficult to be confident that such AIs would lack a rich inner experience.
Because functionalism and preference utilitarianism—both of which could grant moral worth to AI preferences even if they do not precisely replicate biological states—have at least some support within the EA community, I remain hopeful that, if I articulate my position clearly, EAs who share these philosophical assumptions will see its merits.
That said, I am aware that explaining this perspective is an uphill battle. The unpopularity of my views often makes it difficult to communicate without instant misunderstandings; critics seem to frequently conflate my arguments with other, simpler positions that can be more easily dismissed. At times, this has caused me to feel as though the EA community is open to only a narrow range of acceptable ideas. This reaction, while occasionally frustrating, does not surprise me, as I have encountered similar resistance when presenting other unpopular views—such as challenging the ethics of purchasing meat in social contexts where such concerns are quickly deemed absurd.
However, the unpopularity of these ideas also creates a benefit: it creates room for rapid intellectual progress by opening the door to new and interesting philosophical questions about AI ethics. If we free ourselves from the seemingly unquestionable premise that preserving the human species should be the top priority when governing AI development, we can begin to ask entirely new and neglected questions about the role of artificial minds in society.
These questions include: what social and legal frameworks should we pursue if AIs are seen not as dangerous tools to be contained but as individuals on similar moral footing with humans? How do we integrate AI freedom and autonomy into our vision of the future, creating the foundation for a system of ethical and pragmatic AI rights?
Under this alternative philosophical approach, policy would not focus solely on minimizing risks to humanity. Instead, it would emphasize cooperation and inclusion, seeing advanced AI as a partner rather than an ethical menace to be tightly restricted or controlled. This undoubtedly requires a significant shift in our longtermist thinking, demanding a re-examination of deeply rooted assumptions. Such a project cannot be completed overnight, but given the moral stakes and the rapid progress in AI, I view this philosophical endeavor as both urgent and exciting. I invite anyone open to rethinking these foundational premises to join me in exploring how we might foster a future in which AIs and humans coexist as moral peers, cooperating for mutual benefit rather than viewing each other as intrinsic competitors locked in an inevitable zero-sum fight.
I distinguish believing that good successor criteria are brittle from speciesism. I think antispeciesism does not oblige me to accept literally any successor.
I do feel icky coalitioning with outright speciesists (who reject the possibility of a good successor in principle), but I think my goals and all of generalized flourishing benefits a lot from those coalitions so I grin and bear it.
I haven’t read your other recent comments on this, but here’s a question on the topic of pausing AI progress. (The point I’m making is similar to what Brad West already commented.)
Let’s say we grant your assumptions (that AIs will have values that matter the same as or more than human values and that an AI-filled future would be just as or more morally important than one with humans in control). Wouldn’t it still make sense to pause AI progress at this important junction to make sure we study what we’re doing so we can set up future AIs to do as well as (reasonably) possible?
You say that we shouldn’t be confident that AI values will be worse than human values. We can put a pin in that. But values are just one feature here. We should also think about agent psychologies and character traits and infrastructure beneficial for forming peaceful coalitions. On those dimensions, some traits or setups seem (somewhat robustly?) worse than others?
We’re growing an alien species that might take over from humans. Even if you think that’s possibly okay or good, wouldn’t you agree that we can envision factors about how AIs are built/trained and about what sort of world they are placed in that affect whether the future will likely be a lot better or a lot worse?
I’m thinking about things like:
pro-social insctincts (or at least absence of anti-social ones)
more general agent character traits that do well/poorly at forming peaceful coalitions
agent infrastructure to help with coordinating (e.g., having better lie detectors, having a reliable information environment or starting out under the chaos of information warfare, etc.)
initial strategic setup (being born into AI-vs-AI competition vs. being born in a situation where the first TAI can take to proceed slowly and deliberately)
maybe: decision-theoretic setup to do well in acausal interactions with other parts of the multiverse (or at least not do particularly poorly)
If (some of) these things are really important, wouldn’t it make sense to pause and study this stuff until we know whether some of these traits are tractable to influence?
(And, if we do that, we might as well try to make AIs have the inclination to be nice to humans, because humans already exist, so anything that kills humans who don’t want to die frustrates already-existing life goals, which seems worse than frustrating the goals of merely possible beings.)
I know you don’t talk about pausing in your above comment—but I think I vaguely remember you being skeptical of it in other comments. Maybe that was for different reasons, or maybe you just wanted to voice disagreement with the types of arguments people typically give in favor of pausing?
FWIW, I totally agree with the position that we should respect the goals of AIs (assuming they’re not just roleplayed stated goals but deeply held ones—of course, this distinction shouldn’t be uncharitably weaponized against AIs ever being considered to have meaningful goals). I’m just concerned because whether the AIs respect ours in turn, especially when they find themselves in a position where they could easily crush us, will probably depend on how we build them.
In your comment, you raise a broad but important question about whether, even if we reject the idea that human survival must take absolute priority other concerns, we might still want to pause AI development in order to “set up” future AIs more thoughtfully. You list a range of traits—things like pro-social instincts, better coordination infrastructures, or other design features that might improve cooperation—that, in principle, we could try to incorporate if we took more time. I understand and agree with the motivation behind this: you are asking whether there is a prudential reason, from a more inclusive moral standpoint, to pause in order to ensure that whichever civilization emerges—whether dominated by humans, AIs, or both at once—turns out as well as possible in ways that matter impartially, rather than focusing narrowly on preserving human dominance.
Having summarized your perspective, I want to clarify exactly where I differ from your view, and why.
First, let me restate the perspective I defended in my previous post on delaying AI. In that post, I was critiquing what I see as the “standard case” for pausing AI, as I perceive it being made in many EA circles. This standard case for pausing AI often treats preventing human extinction as so paramount that any delay of AI progress, no matter how costly to currently living people, becomes justified if it incrementally lowers the probability of humans losing control.
Under this argument, the reason we want to pause is that time spent on “alignment research” can be used to ensure that future AIs share human goals, or at least do not threaten the human species. My critique had two components: first, I argued that pausing AI is very costly to people who currently exist, since it delays medical and technological breakthroughs that could be made by advanced AIs, thereby forcing a lot of people to die who could have otherwise been saved. Second, and more fundamentally, I argued that this “standard case” seems to rest on an assumption of strictly prioritizing human continuity, independent of whether future AIs might actually generate utilitarian moral value in a way that matches or exceeds humanity.
I certainly acknowledge that one could propose a different rationale for pausing AI, one which does not rest on the premise that preserving the human species is intrinsically worth more than than other moral priorities. This is, it seems, the position you are taking.
Nonetheless, I don’t find your considerations compelling for a variety of reasons.
To begin with, it might seem that granting ourselves “more time” robustly ensures that AIs come out morally better—pro-social, cooperative, and so on. Yet the connection between “getting more time” to “achieving positive outcomes” does not seem straightforward. Merely taking more time does not ensure that this time will be used to increase, rather than decrease, the relevant quality of AI systems according to an impartial moral view. Alignment with human interests, for example, could just as easily push systems in directions that entrench specific biases, maintain existing social structures, or limit moral diversity—none of which strongly aligns with the “pro-social” ideals you described. In my view, there is no inherent link between a slower timeline and ensuring that AIs end up embodying genuinely virtuous or impartial ethical principles. Indeed, if what we call “human control” is mainly about enforcing the status quo or entrenching the dominance of the human species, it may be no better—and could even be worse—than a scenario in which AI development proceeds at the default pace, potentially allowing for more diversity and freedom in how systems are shaped.
Furthermore, in my own moral framework—which is heavily influenced by preference utilitarianism—I take seriously the well-being of everyone who currently exists in the present. As I mentioned previously, one major cost to pausing AI is that it would likely postpone many technological benefits. These might include breakthroughs in medicine—potential cures for aging, radical extensions of healthy lifespans, or other dramatic increases to human welfare that advanced AI could accelerate. We should not simply dismiss the scale of that cost. The usual EA argument for downplaying these costs rests on the Astronomical Waste argument. However, I find this argument flawed, and I spelled out exactly why I found this argument flawed in the post I just wrote.
If a pause sets back major medical discoveries by even a decade, that delay could contribute to the premature deaths of around a billion people alive today. It seems to me that an argument in favor of pausing should grapple with this tradeoff, instead of dismissing it as clearly unimportant compared to the potential human lives that could maybe exist in the far future. Such a dismissal would seem both divorced from common sense concern for existing people, and divorced from broader impartial utilitarian values, as it would prioritize the continuity of the human species above and beyond species-neutral concerns for individual well-being.
Finally, I take very seriously the possibility that pausing AI would cause immense and enduring harm by requiring the creation of vast regulatory controls over society. Realistically, the political mechanisms by which we “pause” advanced AI development would likely involve a lot of coercion, surveillance and social control, particularly as AI starts becoming an integral part of our economy. These efforts are likely to expand state regulatory powers, hamper open competition, and open the door to a massive intrusion of state interference in economic and social activity. I believe these controls would likely be far more burdensome and costly than, for example, our controls over nuclear weapons. If our top long-term priority is building a more free, prosperous, inclusive, joyous, and open society for everyone, rather than merely to control and stop AI, then it seems highly questionable that creating the policing powers required to pause AI is the best way to achieve this objective.
As I see it, the core difference between the view you outlined and mine is not that I am ignoring the possibility that we might “do better” by carefully shaping the environment in which AIs arise. I concede that if we had a guaranteed mechanism to spend a known, short period of time intentionally optimizing how AIs are built, without imposing any other costs in the meantime, that might bring some benefits. However, my skepticism flows from the actual methods by which such a pause would come about, its unintended consequences on liberty, the immediate harms it imposes on present-day people by delaying technological progress, and the fact that it might simply entrench a narrower or more species-centric approach that I explicitly reject. It is not enough to claim that “pausing gives us more time”, suggesting that “more time” is robustly a good thing. One must argue why that time will be spent well, in a way that outweighs the enormous and varied costs that I believe are incurred by pausing AI.
To be clear, I am not opposed to all forms of regulation. But I tend to prefer more liberal approaches, in the sense of classical liberalism. I prefer strategies that try to invite AIs into a cooperative framework, giving them legal rights and a path to peaceful integration—coupled, of course, with constraints on any actor (human or AI) who threatens to commit violence. This, in my view, simply seems like a far stronger foundation for AI policy than a stricter top-down approach in which we halt all frontier AI progress, and establish the associated sweeping regulatory powers required to enforce such a moratorium.
I think it’s interesting to assess how popular or unpopular these views are within the EA community. This year and last year, we asked people in the EA Survey about the extent to which they agreed or disagreed that:
Most expected value in the future comes from digital minds’ experiences, or the experiences of other nonbiological entities.
This year about 47% (strongly or somewhat) disagreed, while 22.2% agreed (roughly a 2:1 ratio).
However, among people who rated AI risks a top priority, respondents leaned towards agreement, with 29.6% disagreeing and 36.6% agreeing (a 0.8:1 ratio).[1]
Similarly, among the most highly engaged EAs, attitudes were roughly evenly split between 33.6% disagreement and 32.7% agreement (1.02:1), with much lower agreement among everyone else.
This suggests to me that the collective opinion of EAs, among those who strongly prioritise AI risks and the most highly engaged is not so hostile to digital minds. Of course, for practical purposes, what matters most might be the attitudes of a small number of decisionmakers, but I think the attitudes of the engaged EAs matters for epistemic reasons.
Interestingly, among people who merely rated AI risks a near-top priority, attitudes towards digital minds were similar to the sample as a whole. Lower prioritisation of AI risks were associated with yet lower agreement with the digital minds item.
I think for me, part of the issue with your posts on this (which I think are net positive to be clear, they really push at significant weak points in ideas widely held in the community) is that you seem to be sort of vacillating between three different ideas, in a way that conceal that one of them, taken on its own sounds super-crazy and evil:
1) Actually, if AI development were to literally lead to human extinction, that might be fine, because it might lead to higher utility.
2) We should care about humans harming sentient, human-like AIs as much as we care about AIs harming humans.
3) In practice, the benefits to current people from AI development outweigh the risks, and the only moral views which say that we should ignore this and pause in the face of even tiny risks of extinction from AI because there are way more potential humans in the future, in fact, when taken seriously, imply 1), which nobody believes.
1) feels extremely bad to me, basically a sort of Nazi-style view on which genocide is fine if the replacing people are superior utility generators (or I guess, inferior but sufficiently more numerous). 1) plausibly is a consequence of classical utilitarianism (even maybe on some person-affecting versions of classical utilitarianism I think), but I take this to be a reason to reject pure classical utilitarianism, not a reason to endorse 1). 2) and 3), on the other hand, seem reasonable to me. But the thing is that you seem at least sometimes to be taking AI moral patienthood as a reason to push on in the face of uncertainty about whether AI will literally kill everyone. And that seems more like 1) than 2) or 3). 1-style reasoning supports the idea that AI moral patienthood is a reason for pushing on with AI development even in the face of human extinction risk, but as far as I can tell 2) and 3) don’t. At the same time though I don’t think you mean to endorse 1).
I realize my position can be confusing, so let me clarify it as plainly as I can: I do not regard the extinction of humanity as anything close to “fine.” In fact, I think it would be a devastating tragedy if every human being died. I have repeatedly emphasized that a major upside of advanced AI lies in its potential to accelerate medical breakthroughs—breakthroughs that might save countless human lives, including potentially my own. Clearly, I value human lives, as otherwise I would not have made this particular point so frequently.
What seems to cause confusion is that I also argue the following more subtle point: while human extinction would be unbelievably bad, it would likely not be astronomically bad in the strict sense used by the “astronomical waste” argument. The standard “astronomical waste” argument says that if humanity disappears, then all possibility for a valuable, advanced civilization vanishes forever. But in a scenario where humans die out because of AI, civilization would continue—just not with humans. That means a valuable intergalactic civilization could still arise, populated by AI rather than by humans. From a purely utilitarian perspective that counts the existence of a future civilization as extremely valuable—whether human or AI—this difference lowers the cataclysm from “astronomically, supremely, world-endingly awful” to “still incredibly awful, but not on a cosmic scale.”
In other words, my position remains that human extinction is very bad indeed—it entails the loss of eight billion individual human lives, which would be horrifying. I don’t want to be forcibly replaced by an AI. Nor do I want you, or anyone else to be forcibly replaced by an AI. I am simply pushing back on the idea that such an event would constitute the absolute destruction of all future value in the universe. There is a meaningful distinction between “an unimaginable tragedy we should try very hard to avoid” and “a total collapse of all potential for a flourishing future civilization of any kind.” My stance falls firmly in the former category.
This distinction is essential to my argument because it fundamentally shapes how we evaluate trade-offs, particularly when considering policies that aim to slow or restrict AI research. If we assume that human extinction due to AI would erase all future value, then virtually any present-day sacrifice—no matter how extreme—might seem justified to reduce that risk. However, if advanced AI could continue to sustain its own value-generating civilization, even in the absence of humans, then extinction would not represent the absolute end of valuable life. While this scenario would be catastrophic for humanity, attempting to avoid it might not outweigh certain immediate benefits of AI, such as its potential to save lives through advanced technology.
In other words, there could easily be situations where accelerating AI development—rather than pausing it—ends up being the better choice for saving human lives, even if doing so technically slightly increases the risk of human species extinction. This does not mean we should be indifferent to extinction; rather, it means we should stop treating extinction as a near-infinitely overriding concern, where even the smallest reduction in its probability is always worth immense near-term costs to actual people living today.
For a moment, I’d like to reverse the criticism you leveled at me. From where I stand, it is often those who strongly advocate pausing AI development, not myself, who can appear to undervalue the lives of humans. I know they don’t see themselves this way, and they would certainly never phrase it in those terms. Nevertheless, this is my reading of the deeper implications of their position.
A common proposition that many AI pause advocates have affirmed to me is that it very well could be worth it to pause AI, even if this led to billions of humans dying prematurely due to them missing out on accelerated medical progress that could otherwise have saved their lives. Therefore, while these advocates care deeply about human extinction (something I do not deny), their concern does not seemrooted in the intrinsic worth of the people who are alive today. Instead, their primary focus often seems to be on the loss of potential future human lives that could maybe exist in the far future—lives that do not yet even exist, and on my view, are unlikely to exist in the far future in basically any scenario, since humanity is unlikely to be preserved as a fixed, static concept over the long-run.
In my view, this philosophy neither prioritizes the well-being of actual individuals nor is it grounded in the utilitarian value that humanity actively generates. If this philosophy were purely about impartial utilitarian value, then I ask: why are they not more open to my perspective? Since my philosophy takes an impartial utilitarian approach—one that considers not just human-generated value, but also the potential value that AI itself could create—it would seem to appeal to those who simply took a strict utilitarian approach, without discriminating against artificial life arbitrarily. Yet, my philosophy largely does not appeal to those who express this view, suggesting the presence of alternative, non-utilitarian concerns.
FWIW, I completely agree with what you’re saying here and I think that if you seriously go into consciousness research and especially for what we westerners more label as a sense of self rather than anything else it quickly becomes infeasible to hold a position that the way we’re taking AI development, e.g towards AI agents will not lead to AIs having self-models.
For all matters and purposes this encompasses most theories of physicalist or non-dual theories of consciousness which are the only feasible ones unless you want to bite some really sour apples.
There’s a classic “what are we getting wrong” question in EA and I think it’s extremely likely that we will look back in 10 years and say, “wow, what are we doing here?”.
I think it’s a lot better to think of systemic alignment and look at properties that we want for the general collective intelligences that we’re engaging in such as our information networks or our institutional decision making procedures and think of how we can optimise these for resillience and truth-seeking. If certain AIs deserve moral patienthood then that truth will naturally arise from such structures.
(hot take) Individual AI alignment might honestly be counter-productive towards this view.
I think it’s interesting and admiral that you’re dedicated on a position that’s so unusual in this space.
I assume I’m in the majority here that my intuitions are quite different from yours, however.
One quick point when we’re here: > this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.
I think that a common, but perhaps not well vocalized, utilitarian take is that humans don’t have much of a special significance in terms of creating well-being. The main option would be a much more abstract idea, some kind of generalization of hedonium or consequentialism-ium or similar. For now, let’s define hedonium as “the ideal way of converting matter and energy into well-being, after a great deal of deliberation.”
As such, it’s very tempting to try to separate concerns and have AI tools focus on being great tools, and separately optimize hedonium to be efficient at being well-being. While I’m not sure if AIs would have zero qualia, I’d feel a lot more confident that they will have dramatically less qualia per unit resources than a much more optimized substrate.
If one follows this general logic, then one might assume that it’s likely that the vast majority of well-being in the future would exist as hedonium, not within AIs created to ultimately make hedonium.
One less intense formulation would be to have both AIs and humans focus only on making sure we get to the point where we much better understand the situation with qualia and hedonium (a la the Long Reflection), and then re-evaluate. In my strategic thinking around AI I’m not particularly optimizing for the qualia of the humans involved in the AI labs or the relevant governments. Similarly, I’d expect not to optimize hard for the qualia in the early AIs, in the period when we’re unsure about qualia and ethics, even if I thought they might have experiences. I would be nervous if I thought this period could involve AIs having intense suffering or be treated in highly immoral ways.
Adrian Tchaikovsky, the science fiction writer, is a master at crafting bleak,
hellish future worlds.
In Service Model,
he has truly outdone himself,
conjuring an absurd realm where human societies have crumbled,
and humanity teeters on the brink of extinction.
Now, that scenario isn’t entirely novel.
But what renders the book both tear-inducing and hilarious,
is the presence in this world of numerous sophisticated robots,
designed to eliminate the slightest discomfort from human existence.
Yet, they adhere so strictly to their programmed rules,
that it only leads to endless absurdities,
and meaningless ordeals for both robots and humans alike.
Science fiction writers,
effective altruists,
and Silicon Valley billionaires have long cautioned,
that the rise of sentient,
super-human artificial intelligence might herald the downfall of our own species.
However, Tchaikovsky suggests a different,
perhaps more mundane, and even more depressing scenario.
He proposes that precisely because robots,
no matter how advanced,
lack free will,
and cannot exercise their own volition in decision-making,
they will not only fail to rescue us from impending environmental,
political, and economic crises,
but they will also be incapable of replacing us,
by creating a better world of their own.
And, I believe, that is Tchaikovsky’s final warning to humanity.
I hope that future historians,
if they still exist
—since there aren’t any left in Service Model
—will regard him as a mere novelist,
one who tries to capitalise on the general unease
concerning advancements in artificial intelligence.
Yet, I fear he may indeed be onto something.
How might EA-aligned orgs in global health and wellness need to adapt calculations of cost-effective interventions given the slash-and-burn campaign currently underway against US foreign aid? Has anyone tried gaming out what different scenarios of funding loss look like (e.g., one where most of the destruction is reversed by the courts, or where that reversal is partial, or where nothing happens and the days are numbered for things like PEPFAR)? Since US foreign aid is so varied, I imagine that’s a tall order, but I’ve been thinking about this quite a bit lately!
Although it’s an interesting question, I’m not sure that gaming out scenarios is that useful in many cases. I think putting energy into responding to the funding reality changes as they appear may be more important. There are just so many scenarios possible in the next few months.
PEPFAR might be the exception to that, as if it gets permanently cut then there just has to be a prompt and thought through response. Other programs might be able to be responded to in the fly, but if The US do pull out of HUV funding there needs to be a contingency plan in place. Maybe gaming scenarios is useful there, but only if whoever is gaming it actually has the influence either to fund scenarios that do arise, or informed those that fund. Maybe the WHO is doing this but they aren’t very agile and don’t communicate much on the fly so it’s hard to know
I think pepfar and malaria tests and treatment donations are among the most important and large scale funding gaps that need to be considered responded to in the short term. Even if stocks remain for the next few months, if they aren’t delivered because those organizing their delivery didn’t have jobs then that’s a big problem.
I do think that governments need to take some responsibility too. If you have the medications you probably can switch manpower to delivering them, even if you hadn’t budgeted for it because you expected USAID was going to fund that indefinitely. This is the situation for malaria and HIV commodities which are often there in decent quantities but sometimes aren’t being distributed effectively right now.
The vast majority of other USAID programs I don’t believe are super cost effective, so as super sad as it is that they are gone and no longer helping people, I don’t think it’s wise to consider covering their funding in most cases as that money would be better spent on more cost effective charities.
I skimmed past many posts like this, assuming that it was some kind of stomach worm, or related to the suffering of wild worms (not that I am opposed to either of those, they just don’t grab my attention as strongly)
Which interesting EA-related bluesky accounts do you know of?
I’m not using Twitter anymore since it’s being used to promote hateful views, but Bluesky is quite a cool online space in my opinion.
I’m making a list of Bluesky accounts of EA-related organisations and key people. If you’re active on Bluesky or some of your favourite EA orgs or key people are, please leave a comment with a link to their profile!
Thanks for the suggestion! This should be relatively quick to add so I’ll see if we can do it soon. :) I was also thinking of setting up a bluesky bot account similar to our twitter account. Do you know how active the EA-ish bluesky community is?
High-variance. Most people seem to have created an account and then gone back to being mostly on (eX)twitter. However, there are some quite active accounts. I’m not the best to ask this question to, since I’m not that active either. Still, having the bluesky account post as a mirror of the twitter acccount maybe isn’t hard to set up?
Note: I am obviously not an expert here nor do I have much first hand experience but I thought it could be useful for people I work with to know how I currently conceptualize burnout. I was then encouraged to post on the forum. This is based off around 4 cases of burnout that I have seen (at varying levels of proximity) and conversations with people who have seen significantly more.
Different Conceptions of Burnout
Basic conception that people often have: working too hard until energy is depleted.
Yes, working too hard can lead to exhaustion, but there’s a difference between exhaustion and burnout.
Exhaustion vs. Burnout
Exhaustion:
Result of working very hard/ a lot of hours. Possibly including sleep deprivation or just brain fog.
Can often be resolved with a short break or vacation (eg: one week)
Burnout:
More pervasive and affects many areas of life/work. While it shared many physical symptoms of exhaustion, it is deeper.
A short vacation isn’t sufficient to resolve it.
Core Feelings Tied to Burnout
Burnout is often tied to more core feelings like motivation, recognition, and feeling good about the work you’re doing. It is more tied to your feelings of motivation and value than pure sleep deprivation or lack of rest. If someone is unsure of the value of their work and isn’t super recognized, especially if they’re also working really hard, that can really get into your brain and feels like a recipe for burnout.
Importance of Motivation
This is why I stress the value of motivation so much
Nuance: we should distinguish motivation from being overly enthusiastic about programs.
Jessica take is that we should have set times for re-evaluating the value of programs. Having set evaluation times helps reduce constant worry about program value but still maintains our ability to have a critical eye toward making sure we are having a large impact.
To some extent motivation is a very moldable thing and if you want to try and get more motivated, you can (but it often includes help from others like your manager and team)
Quick note
This isn’t me advocating for exhaustion because it isn’t burnout. I think exhaustion can be very counterproductive and leads to future hours being less productive.
My main thing here is that I don’t think our LFG / work hard culture is the recipe for burnout. I think being uncertain of the value of our programs, facing many internal structural changes, and not being on top of motivation can be. This is part of why I am excited about the M&E work we are doing, people doing tour of duties, and people tracking motivation/actively valuing it.
Jessica addition in Dec. 2024:
Getting sick more often than usual is an indicator to be aware of. This can lead to a spiral of “Get sick, get less done, get more stressed and feel like you are not doing good enough/not feeling good about your work, that stress causing you to get more sick/get sick again”
(I will add for the forum that right now I am feeling really good about the value of our programs but its always good to be approaching programs critically to ensure you are having the most impact :) )
I think that this has the practical implications that people suffering from burnout should at least consider whether they are depressed and consider treatment options with that in mind (e.g. antidepressants, therapy).
There’s a risk that the “burnout” framing limits the options people are considering (e.g. that they need rest / changes to their workplace). At the same time, there’s a risk that people underestimate the extent to which environmental changes are relevant to their depression, so changing their work environment should also be considered if a person does conclude they might be depressed.
From some expressions on extinction risks as I have observed—extinction risks might actually be suffering risks. It could be the expectation of death is torturing. All risks might be suffering risks.
Offer subject to be arbitrarily stopping at some point (not sure exactly how many I’m willing to do)
Give me chatGPT Deep Research queries and I’ll run them. My asks are that:
You write out exactly what you want the prompt to be so I can just copy and paste something in
Feel free to request a specific model (I think the options are o1, o1-pro, o3-mini, and o3-mini-high) but be ok with me downgrading to o3-mini
Be cool with me very hastily answering the inevitable set of follow-up questions that always get asked (seems unavoidable for whatever reason). I might say something like “all details are specified above; please use your best judgement”
Note that the UI is atrocious. You’re not using o1/o3-mini/o1-pro etc. It’s all the same model, a variant of o3, and the model in the bar at the top is completely irrelevant once you click the deep research button. I am very confused why they did it like this
https://openai.com/index/introducing-deep-research/
It’s the first official day of the AI Safety Action Summit, and thus it’s also the day that the Seoul Commitments (made by sixteen companies last year to adopt an RSP/safety framework) have come due.
I’ve made a tracker/report card for each of these policies at www.seoul-tracker.org.
I’ll plan to keep this updated for the foreseeable future as policies get released/modified. Don’t take the grades too seriously — think of it as one opinionated take on the quality of the commitments as written, and in cases where there is evidence, implemented. Do feel free to share feedback if anything you see surprises you, or if you think the report card misses something important.
My personal takeaway is that both compliance and quality for these policies are much worse than I would have hoped. I believe many peoples’ theories of change for these policies gesture at something about a race to the top, where companies are eager to outcompete each other on safety to win talent and public trust, but I don’t sense much urgency or rigor here. Another theory of change is that this is a sort of laboratory for future regulation, where companies can experiment now with safety practices and the best ones could be codified. But most of the diversity between policies here is in how vague they can be while claiming to manage risks :/
I’m really hoping this changes as AGI gets closer and companies feel they need to do more to prove to govts/public that they can be trusted. Part of my hope is that this report card makes clear to outsiders that not all voluntary safety frameworks are equally credible.
Hi! I’m looking for help with a project. If you’re interested or know someone who might be, it would be really great if you let me know/share. I’ll check the forum forum for dms.
Help with acausal research and get mentoring to learn about decision theory
Motivation: Caspar Oesterheld (inventor/discoverer of ECL/MSR), Emery Cooper and I are doing a project where we try to get LLMs to help us with our acausal research.
Our research is ultimately aimed at making future AIs acausally safe.
Project: In a first step, we are trying to train an LLM classifier that evaluates critiques of arguments. To do so, we need a large number of both good and bad arguments about decision theory (and other areas of Philosophy.)
How you’ll learn: If you would like to learn about decision theory, anthropics, open source game theory, …, we supply you with a curriculum. There’s a lot of leeway for what exactly you want to learn about. You go through the readings.
If you already know things and just want to test your ideas, you can optionally skip this step.
Your contribution: While doing your readings, you, write up critiques of arguments you read.
Bottom-line: We get to use your arguments/critiques for our projects and you get our feedback on them. (We have to read and label them for the project anyway.)
Logistics: Unfortunately, you’d be a volunteer. I might be able to pay you a small amount out-of-pocket, but it’s not going to be very much. Caspar and Em are both university employed and I am similar in means to an independent researcher. We are also all non-Americans based in the US which makes it harder for us to acquire money for projects and such for boring and annoying reasons.
Why are we good mentors: Caspar has dozens of publications on related topics. Em has a handful. And I have been around.
2. Be a saint and help with acausal research by doing tedious manual labor and getting little in return We also need help with various grindy tasks that aren’t super helpful for learning, e.g. turning pdfs with equations etc. into sensible txts to feed to LLMs. If you’re motivated to help with that, we would be extremely grateful.
This case is getting a lot of press attention and will likely spawn a lot of further attention in the form of true crime, etc. The effect of this will be likely to cement Rationalism in the public imagination as a group of crazy people (regardless of whether the group in general opposes extremism), and groups and individuals connected to rationalism, including EA, will be reputationally damaged by association.
Bit the bullet and paid them $200. So far, it’s astonishingly good. If you’re in the UK/EU, you can get a refund no questions asked within 14 days so if you’re on the fence I’d definitely suggest giving it a go
Disclaimer: I think the instant USAID cuts are very harmful, they directly affect our organisation’s wonderful nurses and our patients. I’m not endorsing the cuts, I just think exaggurating numbers when communicating for dramatic effect (or out of ignorance) is unhelpful and doesn’t build trust in institutions like the WHO.
Sometimes the lack of understanding, or care in calulations from leading public health bodies befuddles me.
“The head of the United Nations’ programme for tackling HIV/AIDS told the BBC the cuts would have dire impacts across the globe.
“AIDS related deaths in the next five years will increase by 6.3 million” if funding is not restored, UNAIDS executive director Winnie Byanyima said.”
There just isn’t a planet on which AIDS related deaths would increase that much. In 2023 an estimated 630,000 people were estimated to have died from AIDS related deaths. The WHO estimates about 21 million Africans on HIV treatment. Maybe 5 million of these in South Africa aren’t funded by USAID. Other countries like Kenya and Botswana also contribute to their own HIV treatment.
So out of those 16ish million on USAID funded treatment, over 1⁄3 of those would have to die in the next 3 years for that figure would be correct. The only scenario where this could happen is if all of these people went completely untreated, which means that no local government would come in at any stage. This scenario is impossible
I get that the UN HIV program want to put out scary numbers to put the pressure on the US and try and bring other funding in, but it still important to represent reality. Heads of public health institutions and their staff who do this kind of modelling should learn what a counterfactual is.
“AIDS related deaths in the next five years will increase by 6.3 million” if funding is not restored, UNAIDS executive director Winnie Byanyima said.
This is a quote from a BBC news article, mainly about US political and legal developments. We don’t know what the actual statement from the ED said, but I don’t think there’s enough here to infer fault on her part.
For all we know, the original quote could have been something like predicting that deaths will increase by 6.3 million if we can’t get this work funded—which sounds like a reasonable position to take. Space considerations being what they are, I could easily see a somewhat more nuanced quote being turned into something that sounded unaware of counterfactual considerations.
There’s also an inherent limit to how much fidelity can be communicated through a one-sentence channel to a general audience. We can communicate somewhat more in a single sentence here on the Forum, but the ability to make assumptions about what the reader knows helps. For example, in the specific context here, I’d be concerned that many generalist readers would implicitly adjust for other funders picking up some of the slack, which could lead to double-counting of those effects. And in a world that often doesn’t think counterfactually, other readers’ points of comparison will be with counterfactually-unadjusted numbers. Finally, a fair assessment of counterfactual impact would require the reader to understand DALYs or something similar, because at least a fair portion of the mitigation is likely to come from pulling public-health resources away from conditions that do not kill so much as they disable.
So while I would disagree with a statement from UNAIDS that actually said if the U.S. doesn’t fund PEPFAR, 6.3MM more will die, I think there would be significant drawbacks and/or limitations to other ways of quantifying the problem in this context, and think using the 6.3MM number in a public statement could be appropriate if the actual statement were worded appropriately.
Thanks Jason—those are really good points. In general maybe this wasn’t such a useful thing to bring up at this point in time, and in general its good that she is campaigning for funding to be restored. I do think the large exaggeration though means this a bit more than a nitpick.
I’ve been looking for her saying the actual quote, and have struggled to find it. A lot of news agencies have used the same quote I used above with similar context. Mrs. Byanyima even reposted on her twitter the exact quote above...
I also didn’t explain properly but even at the most generous reading of something like After 5 years deaths will increase by 6.3 million if we get zero funding for HIV medication, the number is still wildly exaggurated. Besides the obvious point that many people would self fund the medications if there was zero funding available (I would guess 30%-60%), and that even short periods of self funded treatment (a few months) would greatly increase their lifespan, the 6.3 million is still incorrect at least by a factor of 2.
Untreated HIV in adults in the pre HAART era in Africa had something like an 80% survival rate (maybe even a little higher) 5 years after seroconversion, which would bring a mortality figure of 3.2 million dying in 5 years assuming EVERYONE on PEPFAR drugs remained untreated—about half the 6.3 million figure quoted. Here’s a graph of mortality over time in the Pre HAART era. Its worth keeping in mind that our treatment of AIDS defining infections is far superior to what it was back then, which would keep people alive longer as well.
And my 3.2 million figure doesn’t take into account the not-insignificant number of people who would die within 5 years even while on ARVs which further reduces the extra deaths figure.
Also many countries like Uganda have about 1 years supply of medications left, so we should perhaps be considering the 10% mortality after 4 years of no medications rather than 20% at 5 in this calculation, which would halve the death numbers again.
So I still think the statement remains a long way off being accurate, even if we allow some wiggle room for wording like you rightly say we should.
The only scenario where this could happen is if all of these people went completely untreated, which means that no local government would come in at any stage. This scenario is impossible
Can you elaborate why this is impossible, or at least unlikely?
The idea that no (or even few) Sub-Saharan African countres would stand in the gap for their most vulnerable people with HIV, abandoning them to horrendous sickness and death from HIV that would overwhelm their health systems shows lack of insight.
Countries simply can’t afford to leave people with HIV completely high and dry, economically and politcally. HIV medication would be a priority for most African countries—either extra fundng would be allocated or money switched from other funds to HIV treatment. As much as governments aren’t utilitarian, they know the disaster that would ensue if HIV medications were not given and their heallth systems were overwhelmed. AIDS is a horrible condition which lasts a long time and robs individuals and families of their productivity.
Granted care might be far worse. Funding for tests like viral load cold be cut, there might be disastrous medicaion stockouts. Hundreds of thousands or even more could die because of these USAID cuts. Funding for malaria, tuberculosis and other treatments might fall by the wayside but I believe for most countries HIV care would be a top priority.
There would be some countries that are either too poor or unstable where this might not happen. Countrie like South Sudan, DRC, Somalia—but I strongly believe that most countries would provide most people with HIV most of their treatment for free.
Besides this, given it is life saving I would estimate maybe half (uncertain) of peopl ewith HIV would buy their own medication if there was no other option—if the alternative is death their family would pool money to keep them alive.
Another minor point is that I think drug companies would likely hugely drop the cost of medication as well—otherwise they wouldn’t be able to sell much of it.
When people write “more dakka,” do they simply meaning that we need to try harder and/or try more things? I’ve seen this in two or three pieces of writing on the EA Forum, but I’ve never seen a clear explanation. Apparently “dakka” is slang from a sci-fi video game/tabletop RPG? Is this useless in-group terminology, or does this actually have value?
As best I can tell, “more dakka” is a reference to this quote. Can anyone point me to a more clear or authoritative explanation?
The LW tag is useful here: it says More Dakka is “the technique of throwing more resources at a problem to see if you get better results”.
I like David Manheim’s A Dozen Ways to Get More Dakka; copying it over to reduce the friction of link-clicking:
More dakka is to pour more firepower onto a problem. Two examples:
Example: “bright lights don’t help my seasonal depression”. More dakka: “have you tried even brighter lights?”
Example: we brainstormed ten ideas, none of them seem good. More dakka: “Try listing a 100 ideas”
I’ve substantially revised my views on QURI’s research priorities over the past year, primarily driven by the rapid advancement in LLM capabilities.
Previously, our strategy centered on developing highly-structured numeric models with stable APIs, enabling:
Formal forecasting scoring mechanisms
Effective collaboration between human forecasting teams
Reusable parameterized world-models for downstream estimates
However, the progress in LLM capabilities has updated my view. I now believe we should focus on developing and encouraging superior AI reasoning and forecasting systems that can:
Generate high-quality forecasts on-demand, rather than relying on pre-computed forecasts for scoring
Produce context-specific mathematical models as needed, reducing the importance of maintaining generic mathematical frameworks
Leverage repositories of key insights, though likely not in the form of formal probabilistic mathematical models
This represents a pivot from scaling up traditional forecasting systems to exploring how we can enhance AI reasoning capabilities for forecasting tasks. The emphasis is now on dynamic, adaptive systems rather than static, pre-structured models.
(I rewrote with Claude, I think it’s much more understandable now)
Quick list of some ideas I’m excited about, broadly around epistemics/strategy/AI.
1. I think AI auditors / overseers of critical organizations (AI efforts, policy groups, company management) are really great and perhaps crucial to get right, but would be difficult to do well.
2. AI strategists/tools telling/helping us broadly what to do about AI safety seems pretty safe.
3. In terms of commercial products, there’s been some neat/scary military companies in the last few years (Palantir, Anduril). I’d be really interested if there could be some companies to automate core parts of the non-military government. I imagine there are some parts of the government that are particularly tractable/influenceable/tractable. For example, just making great decisions on which contractors the government should work with. There’s a ton of work to do here, between the federal government / state government / local government.
4. Epistemic Evals of AI seem pretty great to me, I imagine work here can/should be pushed more soon. I’m not a huge fan of emphasizing “truthfulness” specifically, I think there’s a whole lot to get right here. I think my post here is relevant—it’s technically specific to evaluating math models, but I think it applies to broader work. https://forum.effectivealtruism.org/posts/fxDpddniDaJozcqvp/enhancing-mathematical-modeling-with-llms-goals-challenges
5. One bottleneck to some of the above is AI with strong guarantees+abilities of structured transparency. It’s possible that more good work here can wind up going a long way. That said, some of this is definitely already something companies are trying to do for commercial reasons. https://forum.effectivealtruism.org/posts/piAQ2qpiZEFwdKtmq/llm-secured-systems-a-general-purpose-tool-for-structured
6. I think there are a lot of interesting ways for us to experiment with [AI tools to help our research/epistemics]. I want to see a wide variety of highly creative experimentation here. I think people are really limiting themselves in this area to a few narrow conceptions of how AI can be used in very specific ways that humans are very comfortable with. For example, I’d like to see AI dashboards of “How valuable is everything in this space” or even experiments where AIs negotiate on behalf of people and they use the result of that. A lot of this will get criticized for being too weird/disruptive/speculative, but I think that’s where good creative works should begin.
7. Right now, I think the field of “AI forecasting” is actually quite small and constrained. There’s not much money here, and there aren’t many people with bold plans or research agendas. I suspect that some successes / strong advocates could change this.
8. I think that it’s likely that Anthropic (and perhaps Deepmind) would respond well to good AI+epistemics work. “Control” was quickly accepted at Anthropic, for example. I suspect that it’s possible that things like the idea of an “Internal AI+human auditor” or an internal “AI safety strategist” could be adopted if done well.
Well done to the Shrimp Welfare Project for contributing to Waitrose’s pledge to stun 100% of their warm water shrimps by the end of 2026, and for getting media coverage in a prominent newspaper (this article is currently on the front page of the website): Waitrose to stop selling suffocated farmed prawns, as campaigners say they feel pain
This is so great, congratulations team!! <3
One thing this makes me curious about: how good is the existing evidence base on electric stunning is better for the welfare of the shrimp, and how much better is stunning? I didn’t realize SWP was thinking of using the corporate campaign playbook to scale up stunning, so it makes me curious how robustly good this intervention is, and I couldn’t quickly figure this out from the Forum / website. @Aaron Boddy🔸 is there a public thing I can read by any chance? No pressure!
FWIW, “how good is stunning for welfare” is the main question I have about the impact of HSI right now (whereas other crucial considerations like “how many shrimp do you target”, “will farmers use the stunners”, “how should we value shrimp / their expected sentience” all feel clearer to me).
FYI rolling applications are back on for the Biosecurity Forecasting Group! We have started the pilot and are very excited about our first cohort! Don’t want to apply but have ideas for questions? Submit them here (anyone can submit!).
A reflection on the posts I have written in the last few months, elaborating on my views
In a series of recent posts, I have sought to challenge the conventional view among longtermists that prioritizes the empowerment or preservation of the human species as the chief goal of AI policy. It is my opinion that this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.
I recognize that my position is controversial and likely to remain unpopular among effective altruists for a long time. Nevertheless, I believe it is worth articulating my view at length, as I see it as a straightforward application of standard, common-sense utilitarian principles that merely lead to an unpopular conclusion. I intend to continue elaborating on my arguments in the coming months.
My view follows from a few basic premises. First, that future AI systems are quite likely to be moral patients; second, that we shouldn’t discriminate against them based on arbitrary distinctions, such as their being instantiated on silicon rather than carbon, or having been created through deep learning rather than natural selection. If we insist on treating AIs fundamentally differently from a human child or adult—for example, by regarding them merely as property to be controlled or denying them the freedom to pursue their own goals—then we should identify a specific ethical reason for our approach that goes beyond highlighting their non-human nature.
Many people have argued that consciousness is the key quality separating humans from AIs, thus rendering any AI-based civilization morally insignificant compared to ours. They maintain that consciousness has relatively narrow boundaries, perhaps largely confined to biological organisms, and would only arise in artificial systems under highly specific conditions—for instance, if one were to emulate a human mind in digital form. While I acknowledge that this perspective is logically coherent, I find it deeply unconvincing. The AIs I am referring to when I write about this topic would almost certainly not be simplistic, robotic automatons; rather, they would be profoundly complex, sophisticated entities whose cognitive abilities rival or exceed those of the human brain. For anyone who adopts a functionalist view of consciousness, it seems difficult to be confident that such AIs would lack a rich inner experience.
Because functionalism and preference utilitarianism—both of which could grant moral worth to AI preferences even if they do not precisely replicate biological states—have at least some support within the EA community, I remain hopeful that, if I articulate my position clearly, EAs who share these philosophical assumptions will see its merits.
That said, I am aware that explaining this perspective is an uphill battle. The unpopularity of my views often makes it difficult to communicate without instant misunderstandings; critics seem to frequently conflate my arguments with other, simpler positions that can be more easily dismissed. At times, this has caused me to feel as though the EA community is open to only a narrow range of acceptable ideas. This reaction, while occasionally frustrating, does not surprise me, as I have encountered similar resistance when presenting other unpopular views—such as challenging the ethics of purchasing meat in social contexts where such concerns are quickly deemed absurd.
However, the unpopularity of these ideas also creates a benefit: it creates room for rapid intellectual progress by opening the door to new and interesting philosophical questions about AI ethics. If we free ourselves from the seemingly unquestionable premise that preserving the human species should be the top priority when governing AI development, we can begin to ask entirely new and neglected questions about the role of artificial minds in society.
These questions include: what social and legal frameworks should we pursue if AIs are seen not as dangerous tools to be contained but as individuals on similar moral footing with humans? How do we integrate AI freedom and autonomy into our vision of the future, creating the foundation for a system of ethical and pragmatic AI rights?
Under this alternative philosophical approach, policy would not focus solely on minimizing risks to humanity. Instead, it would emphasize cooperation and inclusion, seeing advanced AI as a partner rather than an ethical menace to be tightly restricted or controlled. This undoubtedly requires a significant shift in our longtermist thinking, demanding a re-examination of deeply rooted assumptions. Such a project cannot be completed overnight, but given the moral stakes and the rapid progress in AI, I view this philosophical endeavor as both urgent and exciting. I invite anyone open to rethinking these foundational premises to join me in exploring how we might foster a future in which AIs and humans coexist as moral peers, cooperating for mutual benefit rather than viewing each other as intrinsic competitors locked in an inevitable zero-sum fight.
I distinguish believing that good successor criteria are brittle from speciesism. I think antispeciesism does not oblige me to accept literally any successor.
I do feel icky coalitioning with outright speciesists (who reject the possibility of a good successor in principle), but I think my goals and all of generalized flourishing benefits a lot from those coalitions so I grin and bear it.
I haven’t read your other recent comments on this, but here’s a question on the topic of pausing AI progress. (The point I’m making is similar to what Brad West already commented.)
Let’s say we grant your assumptions (that AIs will have values that matter the same as or more than human values and that an AI-filled future would be just as or more morally important than one with humans in control). Wouldn’t it still make sense to pause AI progress at this important junction to make sure we study what we’re doing so we can set up future AIs to do as well as (reasonably) possible?
You say that we shouldn’t be confident that AI values will be worse than human values. We can put a pin in that. But values are just one feature here. We should also think about agent psychologies and character traits and infrastructure beneficial for forming peaceful coalitions. On those dimensions, some traits or setups seem (somewhat robustly?) worse than others?
We’re growing an alien species that might take over from humans. Even if you think that’s possibly okay or good, wouldn’t you agree that we can envision factors about how AIs are built/trained and about what sort of world they are placed in that affect whether the future will likely be a lot better or a lot worse?
I’m thinking about things like:
pro-social insctincts (or at least absence of anti-social ones)
more general agent character traits that do well/poorly at forming peaceful coalitions
agent infrastructure to help with coordinating (e.g., having better lie detectors, having a reliable information environment or starting out under the chaos of information warfare, etc.)
initial strategic setup (being born into AI-vs-AI competition vs. being born in a situation where the first TAI can take to proceed slowly and deliberately)
maybe: decision-theoretic setup to do well in acausal interactions with other parts of the multiverse (or at least not do particularly poorly)
If (some of) these things are really important, wouldn’t it make sense to pause and study this stuff until we know whether some of these traits are tractable to influence?
(And, if we do that, we might as well try to make AIs have the inclination to be nice to humans, because humans already exist, so anything that kills humans who don’t want to die frustrates already-existing life goals, which seems worse than frustrating the goals of merely possible beings.)
I know you don’t talk about pausing in your above comment—but I think I vaguely remember you being skeptical of it in other comments. Maybe that was for different reasons, or maybe you just wanted to voice disagreement with the types of arguments people typically give in favor of pausing?
FWIW, I totally agree with the position that we should respect the goals of AIs (assuming they’re not just roleplayed stated goals but deeply held ones—of course, this distinction shouldn’t be uncharitably weaponized against AIs ever being considered to have meaningful goals). I’m just concerned because whether the AIs respect ours in turn, especially when they find themselves in a position where they could easily crush us, will probably depend on how we build them.
In your comment, you raise a broad but important question about whether, even if we reject the idea that human survival must take absolute priority other concerns, we might still want to pause AI development in order to “set up” future AIs more thoughtfully. You list a range of traits—things like pro-social instincts, better coordination infrastructures, or other design features that might improve cooperation—that, in principle, we could try to incorporate if we took more time. I understand and agree with the motivation behind this: you are asking whether there is a prudential reason, from a more inclusive moral standpoint, to pause in order to ensure that whichever civilization emerges—whether dominated by humans, AIs, or both at once—turns out as well as possible in ways that matter impartially, rather than focusing narrowly on preserving human dominance.
Having summarized your perspective, I want to clarify exactly where I differ from your view, and why.
First, let me restate the perspective I defended in my previous post on delaying AI. In that post, I was critiquing what I see as the “standard case” for pausing AI, as I perceive it being made in many EA circles. This standard case for pausing AI often treats preventing human extinction as so paramount that any delay of AI progress, no matter how costly to currently living people, becomes justified if it incrementally lowers the probability of humans losing control.
Under this argument, the reason we want to pause is that time spent on “alignment research” can be used to ensure that future AIs share human goals, or at least do not threaten the human species. My critique had two components: first, I argued that pausing AI is very costly to people who currently exist, since it delays medical and technological breakthroughs that could be made by advanced AIs, thereby forcing a lot of people to die who could have otherwise been saved. Second, and more fundamentally, I argued that this “standard case” seems to rest on an assumption of strictly prioritizing human continuity, independent of whether future AIs might actually generate utilitarian moral value in a way that matches or exceeds humanity.
I certainly acknowledge that one could propose a different rationale for pausing AI, one which does not rest on the premise that preserving the human species is intrinsically worth more than than other moral priorities. This is, it seems, the position you are taking.
Nonetheless, I don’t find your considerations compelling for a variety of reasons.
To begin with, it might seem that granting ourselves “more time” robustly ensures that AIs come out morally better—pro-social, cooperative, and so on. Yet the connection between “getting more time” to “achieving positive outcomes” does not seem straightforward. Merely taking more time does not ensure that this time will be used to increase, rather than decrease, the relevant quality of AI systems according to an impartial moral view. Alignment with human interests, for example, could just as easily push systems in directions that entrench specific biases, maintain existing social structures, or limit moral diversity—none of which strongly aligns with the “pro-social” ideals you described. In my view, there is no inherent link between a slower timeline and ensuring that AIs end up embodying genuinely virtuous or impartial ethical principles. Indeed, if what we call “human control” is mainly about enforcing the status quo or entrenching the dominance of the human species, it may be no better—and could even be worse—than a scenario in which AI development proceeds at the default pace, potentially allowing for more diversity and freedom in how systems are shaped.
Furthermore, in my own moral framework—which is heavily influenced by preference utilitarianism—I take seriously the well-being of everyone who currently exists in the present. As I mentioned previously, one major cost to pausing AI is that it would likely postpone many technological benefits. These might include breakthroughs in medicine—potential cures for aging, radical extensions of healthy lifespans, or other dramatic increases to human welfare that advanced AI could accelerate. We should not simply dismiss the scale of that cost. The usual EA argument for downplaying these costs rests on the Astronomical Waste argument. However, I find this argument flawed, and I spelled out exactly why I found this argument flawed in the post I just wrote.
If a pause sets back major medical discoveries by even a decade, that delay could contribute to the premature deaths of around a billion people alive today. It seems to me that an argument in favor of pausing should grapple with this tradeoff, instead of dismissing it as clearly unimportant compared to the potential human lives that could maybe exist in the far future. Such a dismissal would seem both divorced from common sense concern for existing people, and divorced from broader impartial utilitarian values, as it would prioritize the continuity of the human species above and beyond species-neutral concerns for individual well-being.
Finally, I take very seriously the possibility that pausing AI would cause immense and enduring harm by requiring the creation of vast regulatory controls over society. Realistically, the political mechanisms by which we “pause” advanced AI development would likely involve a lot of coercion, surveillance and social control, particularly as AI starts becoming an integral part of our economy. These efforts are likely to expand state regulatory powers, hamper open competition, and open the door to a massive intrusion of state interference in economic and social activity. I believe these controls would likely be far more burdensome and costly than, for example, our controls over nuclear weapons. If our top long-term priority is building a more free, prosperous, inclusive, joyous, and open society for everyone, rather than merely to control and stop AI, then it seems highly questionable that creating the policing powers required to pause AI is the best way to achieve this objective.
As I see it, the core difference between the view you outlined and mine is not that I am ignoring the possibility that we might “do better” by carefully shaping the environment in which AIs arise. I concede that if we had a guaranteed mechanism to spend a known, short period of time intentionally optimizing how AIs are built, without imposing any other costs in the meantime, that might bring some benefits. However, my skepticism flows from the actual methods by which such a pause would come about, its unintended consequences on liberty, the immediate harms it imposes on present-day people by delaying technological progress, and the fact that it might simply entrench a narrower or more species-centric approach that I explicitly reject. It is not enough to claim that “pausing gives us more time”, suggesting that “more time” is robustly a good thing. One must argue why that time will be spent well, in a way that outweighs the enormous and varied costs that I believe are incurred by pausing AI.
To be clear, I am not opposed to all forms of regulation. But I tend to prefer more liberal approaches, in the sense of classical liberalism. I prefer strategies that try to invite AIs into a cooperative framework, giving them legal rights and a path to peaceful integration—coupled, of course, with constraints on any actor (human or AI) who threatens to commit violence. This, in my view, simply seems like a far stronger foundation for AI policy than a stricter top-down approach in which we halt all frontier AI progress, and establish the associated sweeping regulatory powers required to enforce such a moratorium.
Thanks for writing on this important topic!
I think it’s interesting to assess how popular or unpopular these views are within the EA community. This year and last year, we asked people in the EA Survey about the extent to which they agreed or disagreed that:
This year about 47% (strongly or somewhat) disagreed, while 22.2% agreed (roughly a 2:1 ratio).
However, among people who rated AI risks a top priority, respondents leaned towards agreement, with 29.6% disagreeing and 36.6% agreeing (a 0.8:1 ratio).[1]
Similarly, among the most highly engaged EAs, attitudes were roughly evenly split between 33.6% disagreement and 32.7% agreement (1.02:1), with much lower agreement among everyone else.
This suggests to me that the collective opinion of EAs, among those who strongly prioritise AI risks and the most highly engaged is not so hostile to digital minds. Of course, for practical purposes, what matters most might be the attitudes of a small number of decisionmakers, but I think the attitudes of the engaged EAs matters for epistemic reasons.
Interestingly, among people who merely rated AI risks a near-top priority, attitudes towards digital minds were similar to the sample as a whole. Lower prioritisation of AI risks were associated with yet lower agreement with the digital minds item.
I think for me, part of the issue with your posts on this (which I think are net positive to be clear, they really push at significant weak points in ideas widely held in the community) is that you seem to be sort of vacillating between three different ideas, in a way that conceal that one of them, taken on its own sounds super-crazy and evil:
1) Actually, if AI development were to literally lead to human extinction, that might be fine, because it might lead to higher utility.
2) We should care about humans harming sentient, human-like AIs as much as we care about AIs harming humans.
3) In practice, the benefits to current people from AI development outweigh the risks, and the only moral views which say that we should ignore this and pause in the face of even tiny risks of extinction from AI because there are way more potential humans in the future, in fact, when taken seriously, imply 1), which nobody believes.
1) feels extremely bad to me, basically a sort of Nazi-style view on which genocide is fine if the replacing people are superior utility generators (or I guess, inferior but sufficiently more numerous). 1) plausibly is a consequence of classical utilitarianism (even maybe on some person-affecting versions of classical utilitarianism I think), but I take this to be a reason to reject pure classical utilitarianism, not a reason to endorse 1). 2) and 3), on the other hand, seem reasonable to me. But the thing is that you seem at least sometimes to be taking AI moral patienthood as a reason to push on in the face of uncertainty about whether AI will literally kill everyone. And that seems more like 1) than 2) or 3). 1-style reasoning supports the idea that AI moral patienthood is a reason for pushing on with AI development even in the face of human extinction risk, but as far as I can tell 2) and 3) don’t. At the same time though I don’t think you mean to endorse 1).
I realize my position can be confusing, so let me clarify it as plainly as I can: I do not regard the extinction of humanity as anything close to “fine.” In fact, I think it would be a devastating tragedy if every human being died. I have repeatedly emphasized that a major upside of advanced AI lies in its potential to accelerate medical breakthroughs—breakthroughs that might save countless human lives, including potentially my own. Clearly, I value human lives, as otherwise I would not have made this particular point so frequently.
What seems to cause confusion is that I also argue the following more subtle point: while human extinction would be unbelievably bad, it would likely not be astronomically bad in the strict sense used by the “astronomical waste” argument. The standard “astronomical waste” argument says that if humanity disappears, then all possibility for a valuable, advanced civilization vanishes forever. But in a scenario where humans die out because of AI, civilization would continue—just not with humans. That means a valuable intergalactic civilization could still arise, populated by AI rather than by humans. From a purely utilitarian perspective that counts the existence of a future civilization as extremely valuable—whether human or AI—this difference lowers the cataclysm from “astronomically, supremely, world-endingly awful” to “still incredibly awful, but not on a cosmic scale.”
In other words, my position remains that human extinction is very bad indeed—it entails the loss of eight billion individual human lives, which would be horrifying. I don’t want to be forcibly replaced by an AI. Nor do I want you, or anyone else to be forcibly replaced by an AI. I am simply pushing back on the idea that such an event would constitute the absolute destruction of all future value in the universe. There is a meaningful distinction between “an unimaginable tragedy we should try very hard to avoid” and “a total collapse of all potential for a flourishing future civilization of any kind.” My stance falls firmly in the former category.
This distinction is essential to my argument because it fundamentally shapes how we evaluate trade-offs, particularly when considering policies that aim to slow or restrict AI research. If we assume that human extinction due to AI would erase all future value, then virtually any present-day sacrifice—no matter how extreme—might seem justified to reduce that risk. However, if advanced AI could continue to sustain its own value-generating civilization, even in the absence of humans, then extinction would not represent the absolute end of valuable life. While this scenario would be catastrophic for humanity, attempting to avoid it might not outweigh certain immediate benefits of AI, such as its potential to save lives through advanced technology.
In other words, there could easily be situations where accelerating AI development—rather than pausing it—ends up being the better choice for saving human lives, even if doing so technically slightly increases the risk of human species extinction. This does not mean we should be indifferent to extinction; rather, it means we should stop treating extinction as a near-infinitely overriding concern, where even the smallest reduction in its probability is always worth immense near-term costs to actual people living today.
For a moment, I’d like to reverse the criticism you leveled at me. From where I stand, it is often those who strongly advocate pausing AI development, not myself, who can appear to undervalue the lives of humans. I know they don’t see themselves this way, and they would certainly never phrase it in those terms. Nevertheless, this is my reading of the deeper implications of their position.
A common proposition that many AI pause advocates have affirmed to me is that it very well could be worth it to pause AI, even if this led to billions of humans dying prematurely due to them missing out on accelerated medical progress that could otherwise have saved their lives. Therefore, while these advocates care deeply about human extinction (something I do not deny), their concern does not seem rooted in the intrinsic worth of the people who are alive today. Instead, their primary focus often seems to be on the loss of potential future human lives that could maybe exist in the far future—lives that do not yet even exist, and on my view, are unlikely to exist in the far future in basically any scenario, since humanity is unlikely to be preserved as a fixed, static concept over the long-run.
In my view, this philosophy neither prioritizes the well-being of actual individuals nor is it grounded in the utilitarian value that humanity actively generates. If this philosophy were purely about impartial utilitarian value, then I ask: why are they not more open to my perspective? Since my philosophy takes an impartial utilitarian approach—one that considers not just human-generated value, but also the potential value that AI itself could create—it would seem to appeal to those who simply took a strict utilitarian approach, without discriminating against artificial life arbitrarily. Yet, my philosophy largely does not appeal to those who express this view, suggesting the presence of alternative, non-utilitarian concerns.
Thanks, that is very helpful to me in clarifying your position.
I have read or skimmed some of his posts and my sense is that he does endorse 1). But at the same time he says
so maybe this is one of these cases and I should be more careful.
FWIW, I completely agree with what you’re saying here and I think that if you seriously go into consciousness research and especially for what we westerners more label as a sense of self rather than anything else it quickly becomes infeasible to hold a position that the way we’re taking AI development, e.g towards AI agents will not lead to AIs having self-models.
For all matters and purposes this encompasses most theories of physicalist or non-dual theories of consciousness which are the only feasible ones unless you want to bite some really sour apples.
There’s a classic “what are we getting wrong” question in EA and I think it’s extremely likely that we will look back in 10 years and say, “wow, what are we doing here?”.
I think it’s a lot better to think of systemic alignment and look at properties that we want for the general collective intelligences that we’re engaging in such as our information networks or our institutional decision making procedures and think of how we can optimise these for resillience and truth-seeking. If certain AIs deserve moral patienthood then that truth will naturally arise from such structures.
(hot take) Individual AI alignment might honestly be counter-productive towards this view.
I think it’s interesting and admiral that you’re dedicated on a position that’s so unusual in this space.
I assume I’m in the majority here that my intuitions are quite different from yours, however.
One quick point when we’re here:
> this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.
I think that a common, but perhaps not well vocalized, utilitarian take is that humans don’t have much of a special significance in terms of creating well-being. The main option would be a much more abstract idea, some kind of generalization of hedonium or consequentialism-ium or similar. For now, let’s define hedonium as “the ideal way of converting matter and energy into well-being, after a great deal of deliberation.”
As such, it’s very tempting to try to separate concerns and have AI tools focus on being great tools, and separately optimize hedonium to be efficient at being well-being. While I’m not sure if AIs would have zero qualia, I’d feel a lot more confident that they will have dramatically less qualia per unit resources than a much more optimized substrate.
If one follows this general logic, then one might assume that it’s likely that the vast majority of well-being in the future would exist as hedonium, not within AIs created to ultimately make hedonium.
One less intense formulation would be to have both AIs and humans focus only on making sure we get to the point where we much better understand the situation with qualia and hedonium (a la the Long Reflection), and then re-evaluate. In my strategic thinking around AI I’m not particularly optimizing for the qualia of the humans involved in the AI labs or the relevant governments. Similarly, I’d expect not to optimize hard for the qualia in the early AIs, in the period when we’re unsure about qualia and ethics, even if I thought they might have experiences. I would be nervous if I thought this period could involve AIs having intense suffering or be treated in highly immoral ways.
I wrote a quick take on lesswrong about evals. Funders seem enchanted with them, and I’m curious about why that is.
https://www.lesswrong.com/posts/kq8CZzcPKQtCzbGxg/quinn-s-shortform?commentId=HzDD3Lvh6C9zdqpMh
Adrian Tchaikovsky, the science fiction writer, is a master at crafting bleak, hellish future worlds. In Service Model, he has truly outdone himself, conjuring an absurd realm where human societies have crumbled, and humanity teeters on the brink of extinction.
Now, that scenario isn’t entirely novel. But what renders the book both tear-inducing and hilarious, is the presence in this world of numerous sophisticated robots, designed to eliminate the slightest discomfort from human existence. Yet, they adhere so strictly to their programmed rules, that it only leads to endless absurdities, and meaningless ordeals for both robots and humans alike.
Science fiction writers, effective altruists, and Silicon Valley billionaires have long cautioned, that the rise of sentient, super-human artificial intelligence might herald the downfall of our own species. However, Tchaikovsky suggests a different, perhaps more mundane, and even more depressing scenario. He proposes that precisely because robots, no matter how advanced, lack free will, and cannot exercise their own volition in decision-making, they will not only fail to rescue us from impending environmental, political, and economic crises, but they will also be incapable of replacing us, by creating a better world of their own.
And, I believe, that is Tchaikovsky’s final warning to humanity. I hope that future historians, if they still exist —since there aren’t any left in Service Model —will regard him as a mere novelist, one who tries to capitalise on the general unease concerning advancements in artificial intelligence. Yet, I fear he may indeed be onto something.
How might EA-aligned orgs in global health and wellness need to adapt calculations of cost-effective interventions given the slash-and-burn campaign currently underway against US foreign aid? Has anyone tried gaming out what different scenarios of funding loss look like (e.g., one where most of the destruction is reversed by the courts, or where that reversal is partial, or where nothing happens and the days are numbered for things like PEPFAR)? Since US foreign aid is so varied, I imagine that’s a tall order, but I’ve been thinking about this quite a bit lately!
Although it’s an interesting question, I’m not sure that gaming out scenarios is that useful in many cases. I think putting energy into responding to the funding reality changes as they appear may be more important. There are just so many scenarios possible in the next few months.
PEPFAR might be the exception to that, as if it gets permanently cut then there just has to be a prompt and thought through response. Other programs might be able to be responded to in the fly, but if The US do pull out of HUV funding there needs to be a contingency plan in place. Maybe gaming scenarios is useful there, but only if whoever is gaming it actually has the influence either to fund scenarios that do arise, or informed those that fund. Maybe the WHO is doing this but they aren’t very agile and don’t communicate much on the fly so it’s hard to know
I think pepfar and malaria tests and treatment donations are among the most important and large scale funding gaps that need to be considered responded to in the short term. Even if stocks remain for the next few months, if they aren’t delivered because those organizing their delivery didn’t have jobs then that’s a big problem.
I do think that governments need to take some responsibility too. If you have the medications you probably can switch manpower to delivering them, even if you hadn’t budgeted for it because you expected USAID was going to fund that indefinitely. This is the situation for malaria and HIV commodities which are often there in decent quantities but sometimes aren’t being distributed effectively right now.
The vast majority of other USAID programs I don’t believe are super cost effective, so as super sad as it is that they are gone and no longer helping people, I don’t think it’s wise to consider covering their funding in most cases as that money would be better spent on more cost effective charities.
Screwworm is a flesh-eating maggot!
I skimmed past many posts like this, assuming that it was some kind of stomach worm, or related to the suffering of wild worms (not that I am opposed to either of those, they just don’t grab my attention as strongly)
Which interesting EA-related bluesky accounts do you know of?
I’m not using Twitter anymore since it’s being used to promote hateful views, but Bluesky is quite a cool online space in my opinion.
I’m making a list of Bluesky accounts of EA-related organisations and key people. If you’re active on Bluesky or some of your favourite EA orgs or key people are, please leave a comment with a link to their profile!
I’ve also made an EA (GHD+AW+CC) Starter Pack in case you’re interested. Let me know who I should add! Effective Environmentalism also has a pack with effectiveness-oriented climate change accounts.
Some accounts in no particular order:
Oh this is great, thanks for putting it together!
EA Forum feature request: Can we get a bluesky profile link button for profile pages?
Thanks for the suggestion! This should be relatively quick to add so I’ll see if we can do it soon. :) I was also thinking of setting up a bluesky bot account similar to our twitter account. Do you know how active the EA-ish bluesky community is?
High-variance. Most people seem to have created an account and then gone back to being mostly on (eX)twitter. However, there are some quite active accounts. I’m not the best to ask this question to, since I’m not that active either. Still, having the bluesky account post as a mirror of the twitter acccount maybe isn’t hard to set up?
Quick take on Burnout
Note: I am obviously not an expert here nor do I have much first hand experience but I thought it could be useful for people I work with to know how I currently conceptualize burnout. I was then encouraged to post on the forum. This is based off around 4 cases of burnout that I have seen (at varying levels of proximity) and conversations with people who have seen significantly more.
Different Conceptions of Burnout
Basic conception that people often have: working too hard until energy is depleted.
Yes, working too hard can lead to exhaustion, but there’s a difference between exhaustion and burnout.
Exhaustion vs. Burnout
Exhaustion:
Result of working very hard/ a lot of hours. Possibly including sleep deprivation or just brain fog.
Can often be resolved with a short break or vacation (eg: one week)
Burnout:
More pervasive and affects many areas of life/work. While it shared many physical symptoms of exhaustion, it is deeper.
A short vacation isn’t sufficient to resolve it.
Core Feelings Tied to Burnout
Burnout is often tied to more core feelings like motivation, recognition, and feeling good about the work you’re doing. It is more tied to your feelings of motivation and value than pure sleep deprivation or lack of rest. If someone is unsure of the value of their work and isn’t super recognized, especially if they’re also working really hard, that can really get into your brain and feels like a recipe for burnout.
Importance of Motivation
This is why I stress the value of motivation so much
Nuance: we should distinguish motivation from being overly enthusiastic about programs.
Jessica take is that we should have set times for re-evaluating the value of programs. Having set evaluation times helps reduce constant worry about program value but still maintains our ability to have a critical eye toward making sure we are having a large impact.
To some extent motivation is a very moldable thing and if you want to try and get more motivated, you can (but it often includes help from others like your manager and team)
Quick note
This isn’t me advocating for exhaustion because it isn’t burnout. I think exhaustion can be very counterproductive and leads to future hours being less productive.
My main thing here is that I don’t think our LFG / work hard culture is the recipe for burnout. I think being uncertain of the value of our programs, facing many internal structural changes, and not being on top of motivation can be. This is part of why I am excited about the M&E work we are doing, people doing tour of duties, and people tracking motivation/actively valuing it.
Jessica addition in Dec. 2024:
Getting sick more often than usual is an indicator to be aware of. This can lead to a spiral of “Get sick, get less done, get more stressed and feel like you are not doing good enough/not feeling good about your work, that stress causing you to get more sick/get sick again”
(I will add for the forum that right now I am feeling really good about the value of our programs but its always good to be approaching programs critically to ensure you are having the most impact :) )
Relatedly, I think in many cases burnout is better conceptualised as depression (perhaps with a specific work-related etiology).
Whether burnout is distinct from depression at all is a controversy within the literature:
https://www.sciencedirect.com/science/article/abs/pii/S0272735815000173
https://onlinelibrary.wiley.com/doi/abs/10.1002/jclp.22229
https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2019.00284/full
https://psycnet.apa.org/record/2019-18074-001
I think that this has the practical implications that people suffering from burnout should at least consider whether they are depressed and consider treatment options with that in mind (e.g. antidepressants, therapy).
There’s a risk that the “burnout” framing limits the options people are considering (e.g. that they need rest / changes to their workplace). At the same time, there’s a risk that people underestimate the extent to which environmental changes are relevant to their depression, so changing their work environment should also be considered if a person does conclude they might be depressed.
From some expressions on extinction risks as I have observed—extinction risks might actually be suffering risks. It could be the expectation of death is torturing. All risks might be suffering risks.
Offer subject to be arbitrarily stopping at some point (not sure exactly how many I’m willing to do)
Give me chatGPT Deep Research queries and I’ll run them. My asks are that:
You write out exactly what you want the prompt to be so I can just copy and paste something in
Feel free to request a specific model (I think the options are o1, o1-pro, o3-mini, and o3-mini-high) but be ok with me downgrading to o3-mini
Be cool with me very hastily answering the inevitable set of follow-up questions that always get asked (seems unavoidable for whatever reason). I might say something like “all details are specified above; please use your best judgement”
Note that the UI is atrocious. You’re not using o1/o3-mini/o1-pro etc. It’s all the same model, a variant of o3, and the model in the bar at the top is completely irrelevant once you click the deep research button. I am very confused why they did it like this https://openai.com/index/introducing-deep-research/
I did not know that, thanks!
It’s the first official day of the AI
SafetyAction Summit, and thus it’s also the day that the Seoul Commitments (made by sixteen companies last year to adopt an RSP/safety framework) have come due.I’ve made a tracker/report card for each of these policies at www.seoul-tracker.org.
I’ll plan to keep this updated for the foreseeable future as policies get released/modified. Don’t take the grades too seriously — think of it as one opinionated take on the quality of the commitments as written, and in cases where there is evidence, implemented. Do feel free to share feedback if anything you see surprises you, or if you think the report card misses something important.
My personal takeaway is that both compliance and quality for these policies are much worse than I would have hoped. I believe many peoples’ theories of change for these policies gesture at something about a race to the top, where companies are eager to outcompete each other on safety to win talent and public trust, but I don’t sense much urgency or rigor here. Another theory of change is that this is a sort of laboratory for future regulation, where companies can experiment now with safety practices and the best ones could be codified. But most of the diversity between policies here is in how vague they can be while claiming to manage risks :/
I’m really hoping this changes as AGI gets closer and companies feel they need to do more to prove to govts/public that they can be trusted. Part of my hope is that this report card makes clear to outsiders that not all voluntary safety frameworks are equally credible.
Hi! I’m looking for help with a project. If you’re interested or know someone who might be, it would be really great if you let me know/share. I’ll check the forum forum for dms.
Help with acausal research and get mentoring to learn about decision theory
Motivation: Caspar Oesterheld (inventor/discoverer of ECL/MSR), Emery Cooper and I are doing a project where we try to get LLMs to help us with our acausal research.
Our research is ultimately aimed at making future AIs acausally safe.
Project: In a first step, we are trying to train an LLM classifier that evaluates critiques of arguments. To do so, we need a large number of both good and bad arguments about decision theory (and other areas of Philosophy.)
How you’ll learn: If you would like to learn about decision theory, anthropics, open source game theory, …, we supply you with a curriculum. There’s a lot of leeway for what exactly you want to learn about. You go through the readings.
If you already know things and just want to test your ideas, you can optionally skip this step.
Your contribution: While doing your readings, you, write up critiques of arguments you read.
Bottom-line: We get to use your arguments/critiques for our projects and you get our feedback on them. (We have to read and label them for the project anyway.)
Logistics: Unfortunately, you’d be a volunteer. I might be able to pay you a small amount out-of-pocket, but it’s not going to be very much. Caspar and Em are both university employed and I am similar in means to an independent researcher. We are also all non-Americans based in the US which makes it harder for us to acquire money for projects and such for boring and annoying reasons.
Why are we good mentors: Caspar has dozens of publications on related topics. Em has a handful. And I have been around.
2. Be a saint and help with acausal research by doing tedious manual labor and getting little in return
We also need help with various grindy tasks that aren’t super helpful for learning, e.g. turning pdfs with equations etc. into sensible txts to feed to LLMs. If you’re motivated to help with that, we would be extremely grateful.
I have a weird amount of experience with the second thing (dealing with formatted PDFs) and may be able to help—feel free to DM!
One of the alleged Zizian murderers has released a statement from prison, and it’s a direct plea for Eliezer Yudkowsky specifically to become a vegan.
This case is getting a lot of press attention and will likely spawn a lot of further attention in the form of true crime, etc. The effect of this will be likely to cement Rationalism in the public imagination as a group of crazy people (regardless of whether the group in general opposes extremism), and groups and individuals connected to rationalism, including EA, will be reputationally damaged by association.
ChatGPT deep-research users: What type of stuff does it perform well on? How good is it overall?
Bit the bullet and paid them $200. So far, it’s astonishingly good. If you’re in the UK/EU, you can get a refund no questions asked within 14 days so if you’re on the fence I’d definitely suggest giving it a go
Disclaimer: I think the instant USAID cuts are very harmful, they directly affect our organisation’s wonderful nurses and our patients. I’m not endorsing the cuts, I just think exaggurating numbers when communicating for dramatic effect (or out of ignorance) is unhelpful and doesn’t build trust in institutions like the WHO.
Sometimes the lack of understanding, or care in calulations from leading public health bodies befuddles me.
“The head of the United Nations’ programme for tackling HIV/AIDS told the BBC the cuts would have dire impacts across the globe.
“AIDS related deaths in the next five years will increase by 6.3 million” if funding is not restored, UNAIDS executive director Winnie Byanyima said.”
https://www.bbc.com/news/articles/cdd9p8g405no
There just isn’t a planet on which AIDS related deaths would increase that much. In 2023 an estimated 630,000 people were estimated to have died from AIDS related deaths. The WHO estimates about 21 million Africans on HIV treatment. Maybe 5 million of these in South Africa aren’t funded by USAID. Other countries like Kenya and Botswana also contribute to their own HIV treatment.
So out of those 16ish million on USAID funded treatment, over 1⁄3 of those would have to die in the next 3 years for that figure would be correct. The only scenario where this could happen is if all of these people went completely untreated, which means that no local government would come in at any stage. This scenario is impossible
I get that the UN HIV program want to put out scary numbers to put the pressure on the US and try and bring other funding in, but it still important to represent reality. Heads of public health institutions and their staff who do this kind of modelling should learn what a counterfactual is.
This is a quote from a BBC news article, mainly about US political and legal developments. We don’t know what the actual statement from the ED said, but I don’t think there’s enough here to infer fault on her part.
For all we know, the original quote could have been something like predicting that deaths will increase by 6.3 million if we can’t get this work funded—which sounds like a reasonable position to take. Space considerations being what they are, I could easily see a somewhat more nuanced quote being turned into something that sounded unaware of counterfactual considerations.
There’s also an inherent limit to how much fidelity can be communicated through a one-sentence channel to a general audience. We can communicate somewhat more in a single sentence here on the Forum, but the ability to make assumptions about what the reader knows helps. For example, in the specific context here, I’d be concerned that many generalist readers would implicitly adjust for other funders picking up some of the slack, which could lead to double-counting of those effects. And in a world that often doesn’t think counterfactually, other readers’ points of comparison will be with counterfactually-unadjusted numbers. Finally, a fair assessment of counterfactual impact would require the reader to understand DALYs or something similar, because at least a fair portion of the mitigation is likely to come from pulling public-health resources away from conditions that do not kill so much as they disable.
So while I would disagree with a statement from UNAIDS that actually said if the U.S. doesn’t fund PEPFAR, 6.3MM more will die, I think there would be significant drawbacks and/or limitations to other ways of quantifying the problem in this context, and think using the 6.3MM number in a public statement could be appropriate if the actual statement were worded appropriately.
Thanks Jason—those are really good points. In general maybe this wasn’t such a useful thing to bring up at this point in time, and in general its good that she is campaigning for funding to be restored. I do think the large exaggeration though means this a bit more than a nitpick.
I’ve been looking for her saying the actual quote, and have struggled to find it. A lot of news agencies have used the same quote I used above with similar context. Mrs. Byanyima even reposted on her twitter the exact quote above...
”AIDS-related deaths in the next 5 years will increase by 6.3 million”
I also didn’t explain properly but even at the most generous reading of something like After 5 years deaths will increase by 6.3 million if we get zero funding for HIV medication, the number is still wildly exaggurated. Besides the obvious point that many people would self fund the medications if there was zero funding available (I would guess 30%-60%), and that even short periods of self funded treatment (a few months) would greatly increase their lifespan, the 6.3 million is still incorrect at least by a factor of 2.
Untreated HIV in adults in the pre HAART era in Africa had something like an 80% survival rate (maybe even a little higher) 5 years after seroconversion, which would bring a mortality figure of 3.2 million dying in 5 years assuming EVERYONE on PEPFAR drugs remained untreated—about half the 6.3 million figure quoted. Here’s a graph of mortality over time in the Pre HAART era. Its worth keeping in mind that our treatment of AIDS defining infections is far superior to what it was back then, which would keep people alive longer as well.
https://pmc.ncbi.nlm.nih.gov/articles/PMC5784803/
And my 3.2 million figure doesn’t take into account the not-insignificant number of people who would die within 5 years even while on ARVs which further reduces the extra deaths figure.
Also many countries like Uganda have about 1 years supply of medications left, so we should perhaps be considering the 10% mortality after 4 years of no medications rather than 20% at 5 in this calculation, which would halve the death numbers again.
So I still think the statement remains a long way off being accurate, even if we allow some wiggle room for wording like you rightly say we should.
Can you elaborate why this is impossible, or at least unlikely?
The idea that no (or even few) Sub-Saharan African countres would stand in the gap for their most vulnerable people with HIV, abandoning them to horrendous sickness and death from HIV that would overwhelm their health systems shows lack of insight.
Countries simply can’t afford to leave people with HIV completely high and dry, economically and politcally. HIV medication would be a priority for most African countries—either extra fundng would be allocated or money switched from other funds to HIV treatment. As much as governments aren’t utilitarian, they know the disaster that would ensue if HIV medications were not given and their heallth systems were overwhelmed. AIDS is a horrible condition which lasts a long time and robs individuals and families of their productivity.
Granted care might be far worse. Funding for tests like viral load cold be cut, there might be disastrous medicaion stockouts. Hundreds of thousands or even more could die because of these USAID cuts. Funding for malaria, tuberculosis and other treatments might fall by the wayside but I believe for most countries HIV care would be a top priority.
There would be some countries that are either too poor or unstable where this might not happen. Countrie like South Sudan, DRC, Somalia—but I strongly believe that most countries would provide most people with HIV most of their treatment for free.
Besides this, given it is life saving I would estimate maybe half (uncertain) of peopl ewith HIV would buy their own medication if there was no other option—if the alternative is death their family would pool money to keep them alive.
Another minor point is that I think drug companies would likely hugely drop the cost of medication as well—otherwise they wouldn’t be able to sell much of it.