Formerly Executive Director at BERI; now Secretary and board member. Current board member at SecureBio and FAR.AI, where Iâm also the Treasurer.
sawyerđ¸
Karma: 1,403
Ah! Yes thatâs a good point and I misinterpreted.Thatâs part of what I meant by âhistorical accidentâ but now I think that it was confusing to say âaccidentâ and I should have said something like âhisotrical activitiesâ.
I agree that theyâre worth calling out somehow, I just think âlabâ is a misleading way to doing so given their current activities. Iâve made some admittedly-clunky suggestions in other threads here.
I completely agree that OpenAI and Deepmind started out as labs and are no longer so.
I agree that those companies are worth distinguishing. I just think calling them âlabsâ is a confusing way to do so. If the purpose was only to distinguish them from other AI companies, you could call them âAI bananasâ and it would be just as useful. But âAI bananasâ is unhelpful and confusing. I think âAI labsâ is the same (to a lesser but still important degree).
I think this is a useful distinction, thanks for raising it. I support terms like, âfrontier AI company,â âcompany making frontier AI,â and âcompany making foundation models,â all of which help distinguish OpenAI from Palantir. Also it seems pretty likely that within a few years, most companies will be AI companies!? So weâll need new terms. I just donât want that term to be âlabâ.
Another thing you might be alluding to is that âlabâ is less problematic when talking to people within the AI safety community, and more problematic the further out you go. I think that, within a community, the terms of art sort of lose their generic connotations over time, as community members build a dense web of new connotations specific to that meaning. I regret to admit that Iâm at the point where the word âlabâ without any qualifiers at all makes me think of OpenAI!
But code switching is hard, and if we use these terms internally, weâll also use them externally. Also external people read things that were more intended for internal people, so the language leaks out.
Interesting point! Iâd be OK with people calling them âevil mad scientist labs,â but I still think the generic âlabâ has more of a positive, harmless connotation than this negative one.
Iâd also be more sympathetic to calling them âlabsâ if (1) we had actual regulations around them or (2) they were government projects. Biosafety and nuclear weapons labs have a healthy reputation for being dangerous and unfriendly, in a way âcomputer labsâ do not. Also, private companies may have biosafety containment labs on premises, and the people working within them are labworkers/âscientists, but we call the companies pharmaceutical companies (or âBig Pharmaâ), not âfrontier medicine labsâ.
Also also if any startup tried to make a nuclear weapons lab they would be shut down immediately and all the founders would be arrested. [citation needed]
From everything Iâve seen, GWWC has totally transformed under your leadership. And I think this transformation has been one of the best things thatâs happened in EA during that time. Iâm so thankful for everything youâve done for this important organization.
Yep! Something like this is probably unavoidable, and itâs what all of my examples below do (BERI, ACE, and MIRI).
There are many examples of organizations with high funding transparency, including BERI (which I run), ACE, and MIRI (transparency page and top contributors page).
I think this dynamic is generally overstated, at least in the existential risk space that I work in. Iâve personally asked all of our medium and large funders for permission, and the vast majority of them have given permission. Most of the funding comes from Open Philanthropy and SFF, both of which publicly announce all of their grantsâwhen recipients decided not to list those funders, itâs not because the funders donât want them to. There are many examples of organizations with high funding transparency, including BERI (which I run), ACE, and MIRI (transparency page and top contributors page).
Nonprofit organizations should make their sources of funding really obvious and clear: How much money you got from which grantmakers, and approximately when. Any time I go on some orgâs website and canât find information about their major funders, itâs a big red flag. At a bare minimum you should have a list of funders, and Iâm confused why more orgs donât do this.
I think people would say that the dog was stronger and faster than all previous dog breeds, not that it was âmore capableâ. Itâs in fact significantly less capable at not attacking its owner, which is an important dog capability. I just think the language of âcapabilityâ is somewhat idiosyncratic to AI research and industry, and Iâm arguing that itâs not particularly useful or clarifying language.
More to my point (though probably orthogonal to your point), I donât think many people would buy this dog, because most people care more about not getting attacked than they do about speed and strength.
As a side note, I donât see why preferences and goals change any of this. Iâm constantly hearing AI (safety) researchers talk about âcapabilities researchâ on todayâs AI systems, but I donât think most of them think those systems have their own preferences and goals. At least not in the sense that a dog has preferences or goals. I just think itâs a word that AI [safety?] researchers use, and I think itâs unclear and unhelpful language.
#taboocapabilities
What is âcapabilitiesâ? What is âsafetyâ? People often talk about the alignment tax: the magnitude of capabilities/âtime/âcost a developer loses by implementing an aligned/âsafe system. But why should we consider an unaligned/âunsafe system âcapableâ at all? If someone developed a commercial airplane that went faster than anything else on the market, but it exploded on 1% of flights, no one would call that a capable airplane.
This idea overlaps with safety culture and safety engineering and is not new. But alongside recent criticism of the terms âsafetyâ and âalignmentâ, Iâm starting to think that the term âcapabilitiesâ is unhelpful, capturing different things for different people.
I played the paperclips game 6-12 months before reading Superintelligence (which is what convinced me to prioritize AI x-risk), and I think the game made these ideas easier for me to understand and internalize.
This is truly crushing news. I met Marisa at a CFAR workshop in 2020. She was open, kind, and grateful to everyone, and it was joyful to be around her. I worked with her a bit revitalizing the EA Operations Slack Workspace in 2020, and had only had a few conversations with her since then, here and there at EA events. Marisa (like many young EAs) made me excited for a future that would benefit from her work, ambition, and positivity. Now sheâs gone. She was a good person, Iâm glad she was alive, and I am so sad sheâs gone.
Good reasoning, well written. Reading this post convinced me to join the next NYC protest. Unfortunately I missed the one literally two days ago because I waited too long to read this. But I plan to be there in September.
One thing I think is often missing from these sorts of conversations is that âalignment with EAâ and âalignment with my organizationâs missionâ are not the same thing! Itâs a mistake to assume that the only people who understand and believe in your organizationâs mission are members of the effective altruism community. EA ideas donât have to come in a complete package. People can believe that one organizationâs mission is really valuable and important, for different reasons, coming from totally different values, and without also believing that a bunch of other EA organizations are similarly valuable.
For âcore EAâ orgs like the Centre for Effective Altruism[1], thereâs probably near-total overlap between these two things. But for lots of other organizations the overlap is only incidental, and what you should really be looking for is âalignment with my organizationâs missionâ. Perceived EA Alignment is an unpredictable measure of that, while also being correlated with a bunch of other things like culture, thinking style, network, and socioeconomic status, each of which you either donât care about or which you donât want to be selecting for in the first place.
Within EA, work on x-risk is very siloed by type of threat: There are the AI people, the bio people, etc. Is this bad, or good?
Which of these is the correct analogy?
âBiology is to science as AI safety is to x-risk,â or
âImmunology is to biology as AI safety is to x-riskâ
EAs seem to implicitly think analogy 1 is correct: some interdisciplinary work is nice (biophysics) but most biologists can just be biologists (i.e. most AI x-risk people can just do AI).
The âexistential risk studiesâ model (popular with CSER, SERI, and lots of other non-EA academics) seems to think that analogy 2 is correct, and that interdisciplinary work is totally criticalâimmunologists alone cannot achieve a useful understanding of the entire system theyâre trying to study, and they need to exchange ideas with other subfields of medicine/âbiology in order to have an impact, i.e. AI x-risk workers are missing critical pieces of the puzzle when they neglect broader x-risk studies.
Thanks for writing this! I have a more philosophical counter that Iâd love for you to respond to.
The idea of haggling doesnât sit well with me or my idea of what a good society should be like. It feels competitive, uncooperative, and zero-sum, when I want to live in a society where people are honest and cooperative. Specifically, it seems to encourage deceptive pricing and reward people who are willing to be manipulative and stretch the truth.
In other words, haggling gives me bad vibes.
When you think about haggling/ânegotiating in altruistic context, do you have a framing that is more positive than this? Put another way: Other than saving money for the good guys (us) and costing money for the bad guys (some business), why is all of this âgoodâ?