Really valuable work—thank you. Looking forward to the Animal Welfare Alignment Newsletter!
Ben_Norman
Karma: 98
Ben N’s Quick takes
Does anyone have good resources on where value might accrue in worlds where AI can do most (or all) cognitive labour (this Q also applies to physical labour, but is harder)?
Context is I’m interested in learning more about the question of whether “all/most value accrues to AI companies”. My current guess is that value probably just accrues to whatever remains hard to replicate, not necessarily to the model layer.
If you follow this line of reasoning, here are some candidates for where value might end up:
Distribution and customer trust (I’d assume there’s a decent chance users stick with institutions they already trust, especially in high-stakes domains?)
Proprietary workflows/data (i.e not just static datasets, but also embedded org context, feedback loops, tacit knowledge etc)
Regulatory permissions (e.g. medicine, law, finance, government procurement stuff)
Physical-world access (robots, factories, human relationships, and so on)
Compute/energy/inference (but TBC only insofar as these remain structurally scarce, not just temporarily supply-constrained).
Frontier model capability (only if a small number of AI companies maintain a persistent lead, or if integration costs are really high for some reason—which seems unikely due to strong commercial incentives for them to be easy)
(to be clear I think a world where value concentrates in frontier AI companies would be a worse world compared to one where it is more diffuse, due to power concentration concerns + generally more dynamism and resilience in that world).
Thanks for this! Firstly these seem like two separate Qs (“which actors am I most worried about?” vs the specific risks posed by defender-first access programmes).
My current guess (as a relatively uninformed outsider) is that the most worrying near-term category actors with some combination of pre-existing domain knowledge, operational access, and (obviously) intent. So in bio I guess this would mean: state/state-adjacent programmes, small highly motivated teams with wet-lab access, and maybe compromised actors inside institutions that give them access. I definitely don’t currently think frontier models are good enough that a literal random person could cause catastrophe if they wanted to (though this could change).
Re defender-first access specifically: yeah makes sense that this could add risk. E.g. if there’s stuff like insider misuse, account compromise/coercion, leakage of outputs/workflows, etc.
I don’t really have more concrete thoughts beyond this. It would be interesting to see some analysis (if there isn’t already one internally) on how robust OAI’s Rosalind Biodefense and Anthropic’s Project Glasswing are to risks from risky actors getting into the programme.
(Somewhat relatedly I remember reading somewhere that Mythos reportedly got accessed by some unauthorised users via a third-party private Discord group).
Thoughts on Governing Catastrophic AI Misuse Risk
Thanks :)
Great post!
Thanks for your comment.
I’m not able to follow what you mean here, could you explain your points more simply?
Mapping Positive Visions of Post-AGI Futures
So cool that you’re working on this course! Excited to participate and learn more once its released.