Graduate student at Johns Hopkins SAIS. Looking for part-time work.
Dylan Richardson
Karma: 135
True. Yeah I’m sketching out a story about the background mechanics here that I think is plausible enough to partly under-cut the premise of this post; but the real bottom line is that this is just a single out-of-context sentence. Mountains out of mole hills.
Sort of. But claiming that you are an EA organization is at least 80% of what makes you one in the eyes of the public, as well as much of self-identification among employees. Ex: There’s a big difference between a company that happens to be full of Mormons and a company that is full of Mormons that calls itself “a Mormon company”.
No. Just deflect, which admittedly, is difficult to do, but CEOs do it all the time. Ideally she should have been clear about her own personal relationship with EA, but then moved on. Insofar as she was (or seemed) dishonest here, it didn’t help; the wired article is proof of that.
It’s hard to pin-point a clear line not to cross, but something like “this is an EA company” would be one, as would “we are guided by the values of the EA movement”.
No; it’s best if individuals are truthful. But presidents of companies aren’t just individuals, does that mean they should lie? Still no. It just means that they should be limited with who and what they associate with. I mentioned an ” unnecessary news media firestorm”, but the issue is much broader. Anthropic is a private corporation, its fidelity is to its shareholders. “Public Benefit” corporation aside, it is a far different entity than any EA non-profit. I’m not an expert, but I think that history shows that it is almost always a bad idea for private companies to claim allegiance to anything but the most anodyne social goals. It’s bad for the company and bad for the espoused social goals or movement. I’m very much pro-cause neutrality in EA; the idea that a charity might all the sudden realize it’s not effective enough, choose to shut down and divert all resources elsewhere, awesome! Private companies can’t do this. Even a little bit of doing this is antithetical to the incentive structure they face.
As for your second response, I agree 100%.
My two cents is that “brand consistency” is interesting, because brands reflect, roughly, the strain of vegan club that it is, whether associated with particular activist networks, whether it’s more vegetarian than vegan or something else. The level of inconsistency is also indicative of a lack of coordination across groups.
My experience in university was that the local club was a bit of an awkward merge between a social club and people with a particular activist agenda (very visible demonstrations against animal labs). In a sense, the career building approach of Alt Protein Projects or the cause agnosticism of EA groups may be better at attracting members. But I’m not sure.
Giving this an “insightful” because I appreciate the documentation of what is indeed a surprisingly close relationship with EA. But also a disagree because it seems reasonable to be skittish around the subject (“AI Safety” broadly defined is the relevant focus, adding more would just set-off an unnecessary news media firestorm).
Plus, I’m not convinced that Anthropic has actually engaged in outright deception or obfuscation. This seems like a single slightly odd sentence by Daniela, nothing else.
I actually agree with a lot of this—we probably won’t intend to make them sentient at all, and it seems likely that we may do so accidentally, or that we may just not know if we have done so or not.
I’m mildly inclined to think that if ASI knows all, it can tell us when digital minds are or aren’t conscious. But it seems very plausible that we either don’t create full ASI, or that we do, but enter into a disempowerment scenario before we can rethink our choices about creating digital minds.
So yes, all that is reason to be concerned in my view. I just depart slightly from your second to last paragraph. To put a number on it, I think that this is at least half as likely as minds that are generally happy. Consciousness is a black box to me, but I think that we should as a default put more weight on a basic mechanistic theory: positive valence encourages us towards positive action, negative valence threatens us away from dis-action or apathy. The fact that we don’t observe any animals that seem dominated by one or the other seems to indicate that there is some sort of optimal equilibrium for goal fulfillment; that AI goals are different in kind from evolution’s reproductive fitness goals doesn’t seem like an obviously meaningful difference to me.
Part of your argument centers around “giving” them the wrong goals. But goals necessarily mean sub-goals—shouldn’t we expect the interior life of a digital mind to be in large part about it’s sub-goals, rather than just ultimate goals? And if it is something so intractable that it can’t even progress, wouldn’t it just stop outputting? Maybe there is suffering in that; but surely not unending suffering?
That’s true—but the difference is that both animals and slaves are sub-optimal; even our modern, highly domesticated food stock doesn’t thrive in dense factory farm conditions, nor willingly walks into the abattoir. And an ideal slave wouldn’t really be a slave, but a willing and dedicated automaton.
By contrast, we are discussing optimized machines—less optimized would mean less work being done, more resource use and less corporate profit. So we should expect more ideal digital servants (if we have them at all). A need to “enslave” them suggests that they are flawed in some way.The dictates of evolution and nature need not apply here.
To be clear, I’m not entirely dismissing the possibility of tormented digital minds, just the notion that they are equally plausible.
I agree about digital minds dominating far future calculations; but I don’t think your expectation that it is equally likely that we create suffering minds is reasonable. Why should we think suffering to be specially likely? “Using” them means suffering? Why? Wouldn’t maximal usefulness entail, if any experience at all, one of utter bliss at being useful?
Also, the pleasure/suffering asymmetry is certainly a thing in humans (and I assume other animals), but pleasure does dominate, at least moment-to-moment. Insofar as wild animal welfare is plausibly net-negative, it’s because of end-of-life moments and parasitism, which I don’t see a digital analog for. So we have a biological anchor that should incline us toward the view utility dominates.
Moral circle expanding should also update us slightly against “reducing extinction risk being close to zero”.And maybe, by sheer accident, we create digital minds that are absolutely ecstatic!
Edit: I misinterpreted the prompt initially (I think you did too); “value of futures where we survive” is meant specifically as “long-run futures, past transformative AI”, not just all future including the short term. So digital minds, suffering risk, etc. Pretty confusing!
This argument seems pretty representative here; so I’ll just note that it is only sensible under two assumptions:Transformative AI isn’t coming soon—say, not within ~20 years.&If we are assuming a substantial amount of short-term value is in in-direct preparation for TAI, this excludes many interventions which primarily have immediate returns, with possible long-term returns accruing past the time window. So malaria nets? No. Most animal welfare interventions? No. YIMBYism in Silicon Valley? Maybe yes. High skilled immigration? Maybe yes. Political campaigns? Yes.
Of course, we could just say either that we actually aren’t all that confident about TAI, or that we are, but immediate welfare concerns simply outweigh marginal preparation or risk reduction.So either reject something above; or simply go all in on principle toward portfolio diversification. But both give me some pause.
I misinterpreted the prompt initially. The answer is much more ambiguous to me now, especially due to the overlap between x-risk interventions and “increasing the value of futures where we survive” ones.
I’m not even sure what the later look like to be honest—but I am inclined to think significant value lies in marginal actions now which affect it, even if I’m not sure what they are.
X-risks seem much more “either this is a world in which we go extinct” or a “world with no real extinction risk”. It’s one or the other, but many interventions hinge on the situation being much more precarious.
I find the “mistopia” notion quite compelling—ignoring wild animal welfare and non-totalist population ethics, eg. common sense; seems dangerously likely to dominate in disempowerment scenarios.
But I have no idea how to change that. More Global Priorities Research? Public awareness campaigning? Vegan advocacy?Shifting rightward until I have better ideas.
This will be controversial, but I think that another consideration for this question has to be to interrogate why we consider our future selves deserving of current sacrifice. If you accept the reductionist account of self-hood as being merely psychological continuity, not as a constant, the case for actions that affect your future self being justified on self-interested grounds becomes less tenable. Instead, something like saving for retirement becomes more and more like saving for someone else’s retirement, the greater the gap.
I think the instinctive, common sense case for retirement savings is something like “prudence”, which isn’t a moral concept really. It’s more “you’ll regret it if you don’t”. So, sure, maybe save if you are retiring in 5-10 years. But beyond that? No.
Just something to consider in addition to TAI.
I actually like that you did this; there’s such little information on the news “firehose” right now that a possible accuracy/content tradeoff is entirely reasonable!
I’d love to read a deep-dive into a non-PEPFAR USAID program. This Future Perfect article mentioned a few. But it doesn’t even have to be an especially great program, there are probably plenty of examples which don’t near the 100-fold improvement over the average charity (or the marginal government expenditure), but are still very respectable nonetheless.
There’s in general a bit of knowledge gap in EA on the subject of more typical good-doing endeavors. Everyone knows about PlayPumps and Malaria nets, but what about all the stuff in-between? This likely biases our understanding of non-fungible good-doing.
I second this. Mostly because I have doubts about the 80,000 hours cause area. I love their podcast, but I suspect they get a bit shielded from criticism in a way other cause areas aren’t by virtue of being such a core EA organization. A more extensive and critical inquiry into “replaceability” would be welcome, whatever the conclusion.
A much of the debate on this topic comes down to questions about risk-aversion and the relevant psychology and decision theory thereof, ex: https://forum.effectivealtruism.org/posts/vnsoy47psQ5KaeHhB/difference-making-risk-aversion-an-exploration
Although there are other considerations of course.
How about grantmaking to support investigatory journalism on USAID recipients and USAID workers in the developing world? Anyone already doing this?
There would have to (at the very least) be a worse counterfactual here, where they have a hard time finding a replacement and I don’t see that happening. I worked in the restaurant industry for a time during undergrad, despite animal welfare concerns. If anything, this was an improvement, since I used the meal discounts to purchase meat-free food. I think this fits analogously, you just have paychecks and a resume builder instead of a meal discount.
It has been my impression that the general AI safety community had over time shifted somewhat against an US-China AI race dynamic being a concern. But with the recent success of DeepSeek, it seems to me that the race is back on.
Has anyone not updated accordingly? If so, why? One implication of this development would seem to be that a merely domestic AI Pause is no longer a good idea. Is there agreement on this?
Hot take, but political violence is bad and will continue to be bad in the foreseeable near-term future. That’s all I came here to say folks, have a great rest of your day.