In that case your strategy is just feeding the labs talent and poisoning the ability of their circles to oppose them.
It seems like your model only has such influence going one way. The lab worker will influence their friends, but not the other way around. I think two-way influence is a more accurate model.
Another option is to ask your friends to monitor you so you don’t get ideologically captured, and hold an intervention if it seems appropriate.
I see no meaningful path to impact on safety working as an AI lab researcher
This is a very strong statement. I’m not following technical alignment research that closely, but my general sense is that exciting work is being done. I just wrote this comment advertising a line of research which strikes me as particularly promising.
I noticed the other day that the people who are particularly grim about AI alignment also don’t seem to be engaging much with contemporary technical alignment research. That missing intersection seems suspicious. I’m interested in any counterexamples that come to mind.
My subjective sense is there’s a good chance we lose because all the necessary insights to build aligned AI were lying around, they just didn’t get sufficiently developed or implemented. This seems especially true for techniques like gradient routing which would need to be baked in to a big, expensive training run.
(I’m also interested in arguments for why unlearning won’t work. I’ve thought about this a fair amount, and it seems to me that sufficiently good unlearning kind of just oneshots AI safety, as elaborated in the comment I linked.)
With regard to Deepseek, it seems to me that the success of mixture-of-experts could be considered an update towards methods like gradient routing. If you could localize specific kinds of knowledge to specific experts in a reliable way, you could dynamically toggle off / ablate experts with unnecessary dangerous knowledge. E.g. toggle off experts knowledgeable in human psychology so the AI doesn’t manipulate you.
I like this approach because if you get it working well, it’s a general tool that could help address a lot of different catastrophe stories in a way that seems pretty robust. E.g. to mitigate a malicious AI from gaining root access to its datacenter, ablate knowledge of the OS it’s running on. To mitigate sophisticated cooperation between AIs that are supposed to be monitoring one another, ablate knowledge of game theory. Etc. (The broader point is that unlearning seems very generally useful. But the “Expand-Route-Ablate” style approach from the gradient routing paper strikes me as a particularly promising, and could harmonize well with MoE.)
I think a good research goal would be to try to eventually replicate Deepseek’s work, except with highly interpretable experts. The idea is to produce a “high-assurance” model which can be ablated so undesired behaviors, like deception, are virtually impossible to jailbreak out of it (since the weights that perform the behavior are inaccessible). I think the gradient routing paper is a good start. To achieve sufficient safety we’ll need new methods that are more robust and easier to deploy, which should probably be prototyped on toy problems first.
it’s possible Trump is better placed to negotiate a binding treaty with China (similar to the idea that ‘only Nixon could go to China’), even if it’s not clear he’ll want to do so.
Some related points:
Trump seems like someone who likes winning, but hates losing, and also likes making deals. If Deepseek increases the probability that the US will lose, that makes it more attractive to negotiate an end to the race. This seems true from both a Trump-psychology perspective, and a rational-game-theorist perspective.
Elon Musk seems to have a good relationship with Chinese leadership.
Releasing open-source AI seems like more of a way to prevent someone else from winning than a way to win yourself.
I suppose one possible approach would be to try to get some sort of back-channel dialogue going, to start drafting a treaty which can be invoked if political momentum appears.
That’s a good point about patient privacy. On X, you suggested that PEPFAR has had a sizable macro impact on AIDS in Africa. Maybe Africans who are old enough to remember could talk about what AIDS was like in Africa before and after PEPFAR, as a way to illustrate that macro impact without violating the privacy of any individual patients.
Of course, insofar as individual patients are willing to speak about their experiences, possibly with some light anonymization such as looking away from the camera, that seems really good to me too.
A couple other thoughts:
If access to ARV drugs is interrupted, I imagine that could lead to drug-resistant HIV strains, same way your doctor tells you to take your antibiotics consistently.
Having a big population of immunocompromised people in Africa seems bad in terms of mutation speed for future pandemics. Here’s a paper I just found on Google, about COVID variants which are thought to have arisen from immunocompromised patients.
I wonder if the response will be seen as more credible if it’s driven by Africans?
Insofar as there’s a motivation behind cutting foreign aid, I suspect that skepticism of the NGO sector is playing a role. I can imagine Trump supporters thinking: This program supposedly helps millions of poor Africans, yet the primary voices advocating for it online are rich-world progressives. Seems fishy.
I’ve received a bunch of Whatsapp messages from Ugandan friends who are very very worried about what this might mean not only for patients (the main issue), but also for jobs and the livelihood of many local NGOS.
I think it could be pretty effective for your Ugandan friends to record videos interviewing people who have been helped by the program, and post the videos online.
A lot of skepticism of foreign aid is driven by the fear that the aid is being captured somewhere along the way, either here in the US or in the target country. Hearing directly from the aid recipients helps address that fear.
As an American, it feels like good things the US does are taken for granted, and the US will be criticized relentlessly by people in other countries no matter what we do. So I wouldn’t suggest scolding the US for withdrawing funds. And in any case, rewards usually work better than punishments for modifying behavior. I think saying thanks, and talking about what it would mean for the program to continue, would be a lot more powerful than scolding. Even for Americans who aren’t particularly altruistic, this would provide tangible evidence that the program is improving America’s reputation.
One mental model I have is that the US is suffering from something akin to EA burnout, and the solution to both “US burnout” and “EA burnout” is a stronger culture of gratitude. In principle, I could imagine that a really good response to this PEPFAR situation actually ends up motivating the US to fund additional effective aid programs.
I sold all my NVIDIA stock, since their moat looks weak to me:
I think your reasoning is generally correct. Another argument: If you believe things look sufficiently grim under short timelines, maybe you should invest under the assumption that a recession, or something else, will pop the AI bubble and gives us longer timelines.
“AI bad” gets misgeneralized into skepticism about current and future AI capabilities.
This point is interesting. I almost wonder if it’s better to not argue against this. If we argue against it, maybe the left gets attached to this position, and becomes slower to update even as unemployment increases.
One could also reason that the left can be counted on to be anti-AI going forwards, and the objective for EA should be to foster anti-AI forces on the right. The H1B split shows that tech leaders don’t have ideological control over the right. Sam Altman and Elon Musk don’t get along either. In fact, Sam Altman doesn’t seem to have a strong popular constituency in either party at this point.
Maybe once Sam says it, Dario kinda has to say it to stay competitive for funding?
I guess my view is that low-effort sharing of new ideas is not rewarded/engaged with, and is undersupplied relative to what’s optimal. When I have a new idea that seems like it could have a big impact, and I quickly post it on the EA Forum, it’s much more out of a sense of duty than a sense of excitement.
I see the “Let’s think about slowing down AI” post as support for my position. We shouldn’t have required a 45-minute read by a senior community member before slowing down AI was taken seriously as a possibility. In a world where Katja is too busy to make that effort-post, I think there’s a chance that EA takes far longer to consider a pivot.
I think high-effort contributions for new ideas aren’t necessarily optimal. I put a fair amount of effort into this post, which looks like a big waste of time in retrospect. In this case, Linch explicitly told me I put too much effort in, and his short comment to that effect got more upvotes than my effort-comment.
The upvote-snowballing mechanism means a small difference in the rate of vote-gaining creates a large difference in attention. It seems like the top 10% of vote-gainers tend to be high-effort, non-controversial stuff, which ideally has some sort of prestige affiliation (“10 pages defending a thesis that is obvious”). So we see a lot of that stuff on the Forum. People copy what’s upvoted, and the Forum ends up rather bland.
The “10 pages defending a thesis that is obvious”-type posts tend to be ones where the author tries to anticipate and respond to every possible criticism or deficiency. An author can’t necessarily predict in advance which axes readers will want more effort on. If you try to predict and address them all, I suspect that contributes to bland writing. It’d be better to address those issues through dialogue than monologue.
Maybe there is a good place for low-effort sharing of new EA ideas elsewhere, and I’m not aware of it. But there does seem to be a suspicious lack of new ideas on the Forum—especially given how fast the world is changing, which should naturally produce new ideas for how to do good. I think lack of new ideas on the Forum is evidence that there’s no good place for low-effort sharing of new EA ideas elsewhere. If EA was good at low-effort sharing of new ideas elsewhere, I would expect some of those ideas to trickle into high-effort “new idea” posts on the Forum, and I’m largely not seeing those posts.
Compare this comment with this comment. The second comment was posted about a week later. I’m glad the second comment was posted, and I’m glad the suggestion was packaged in a way that made it appealing to Forum users, but I do notice that the packaging seemed to matter a fair amount (“rigor” flavoring). I don’t think overconfidence was a major factor here.
I post a fair number of offbeat ideas like this, and they don’t generally receive much attention, which leaves me feeling demoralized. And then I wrote the grandparent comment, where I got downvoted/disagreevoted for asking if there’s a better place to post offbeat ideas, which is even more demoralizing. Like, what do you guys want from me?
I notice your framing
I would guess that weird EA ideas that were appropriately caveated would do reasonably well here
basically acknowledges that this is a hypothetical, and new ideas mostly don’t get posted here. I feel like I have a pretty good understanding of why! Again, maybe this is an OK or even desirable state of affairs. But I wish we could at least acknowledge it.
If it’s literally a waste product, I don’t see why that should matter. The costs should just be for processing and transportation, which would also apply to ordinary milk? Yet in my area whey protein powder costs about 30x as much as milk per unit weight.
It seems that users on this forum want to upvote content which is rigorous and true. So the way to gain karma is to write 10 pages defending a thesis that is obvious, as opposed to writing half a page introducing a thesis that is revolutionary.
That’s not necessarily a problem. I feel like the EA Forum wants to be the end of an idea pipeline, the last step where ideas get final scrutiny, and are stamped for epistemic rigor and community consensus. Yet the beginning and middle of the pipeline sort of don’t seem to exist? At least not on the public internet.
Anyway, let me know if there’s a better place to post my weird EA ideas. My general sense is that weird ideas are not super welcome here.
If what you’re saying is true, why is whey protein so expensive?
Do you know of anything which directly addresses the argument I made here?
My vague impression is that the optimism I see is based on outside-view type forecasts, and people have mostly not taken inside views. I haven’t thought much about bird flu recently though.
Thanks.
Regarding decreased demand for GPUs—I agree Deepseek v3 is evidence that demand will decrease. But o3 seems like evidence that demand will increase, since we’ll be doing more test-time compute. Unclear which evidence is more compelling.
Interesting. Seems like the obvious way to resolve this disagreement would be to check benchmarks for SCALE. 6 months ago, in this reddit thread, they said benchmarks will “become publicly available in the near future”. So, join their discord if you’re holding Nvidia stock, I guess?
The track record of anticapitalist advocacy seems quite poor. See this free book: Socialism: The failed idea that never dies
If you’re doing anticapitalist advocacy for EA reasons, I think you need a really clear understanding of why such advocacy has caused so much misery in the past, and how your advocacy will avoid those traps.
I’d say what’s needed is not anticapitalist advocacy, so much as small-scale prototyping of alternative economic systems that have strong theoretical arguments for how they will align incentives better, and scale way past Dunbar’s number.
You don’t need a full replacement for capitalism to test ideas and see results. For example, central planning often fails due to corruption. A well-designed alternative system will probably need a solution for corruption. And such a solution could be usefully applied to an ordinary capitalist democracy.
I concede that AI companies are behaving in a harmful way, but I doubt that anticapitalist advocacy is a particularly tractable way to address that, at least in the short term.