Formerly Executive Director at BERI; now Secretary and board member. Current board member at SecureBio and FAR.AI, where Iâm also the Treasurer.
sawyerđ¸
Karma: 1,463
Some perÂsonal thoughts about workÂing at Tarbell
Having not read the article, this threw me and I had to go check. But unfortunately they do seem to be calling the timeline models themselves âbadâ.
I focussed on one section alone: their âtimelines forecastâ code and accompanying methodology section. Not to mince words, I think itâs pretty bad.
I think this is the single most underrated post on the EA Forum.
Thanks for writing this! I have a more philosophical counter that Iâd love for you to respond to.
The idea of haggling doesnât sit well with me or my idea of what a good society should be like. It feels competitive, uncooperative, and zero-sum, when I want to live in a society where people are honest and cooperative. Specifically, it seems to encourage deceptive pricing and reward people who are willing to be manipulative and stretch the truth.
In other words, haggling gives me bad vibes.
When you think about haggling/ânegotiating in altruistic context, do you have a framing that is more positive than this? Put another way: Other than saving money for the good guys (us) and costing money for the bad guys (some business), why is all of this âgoodâ?
Ah! Yes thatâs a good point and I misinterpreted.Thatâs part of what I meant by âhistorical accidentâ but now I think that it was confusing to say âaccidentâ and I should have said something like âhisotrical activitiesâ.
I agree that theyâre worth calling out somehow, I just think âlabâ is a misleading way to doing so given their current activities. Iâve made some admittedly-clunky suggestions in other threads here.
I completely agree that OpenAI and Deepmind started out as labs and are no longer so.
I agree that those companies are worth distinguishing. I just think calling them âlabsâ is a confusing way to do so. If the purpose was only to distinguish them from other AI companies, you could call them âAI bananasâ and it would be just as useful. But âAI bananasâ is unhelpful and confusing. I think âAI labsâ is the same (to a lesser but still important degree).
I think this is a useful distinction, thanks for raising it. I support terms like, âfrontier AI company,â âcompany making frontier AI,â and âcompany making foundation models,â all of which help distinguish OpenAI from Palantir. Also it seems pretty likely that within a few years, most companies will be AI companies!? So weâll need new terms. I just donât want that term to be âlabâ.
Another thing you might be alluding to is that âlabâ is less problematic when talking to people within the AI safety community, and more problematic the further out you go. I think that, within a community, the terms of art sort of lose their generic connotations over time, as community members build a dense web of new connotations specific to that meaning. I regret to admit that Iâm at the point where the word âlabâ without any qualifiers at all makes me think of OpenAI!
But code switching is hard, and if we use these terms internally, weâll also use them externally. Also external people read things that were more intended for internal people, so the language leaks out.
Interesting point! Iâd be OK with people calling them âevil mad scientist labs,â but I still think the generic âlabâ has more of a positive, harmless connotation than this negative one.
Iâd also be more sympathetic to calling them âlabsâ if (1) we had actual regulations around them or (2) they were government projects. Biosafety and nuclear weapons labs have a healthy reputation for being dangerous and unfriendly, in a way âcomputer labsâ do not. Also, private companies may have biosafety containment labs on premises, and the people working within them are labworkers/âscientists, but we call the companies pharmaceutical companies (or âBig Pharmaâ), not âfrontier medicine labsâ.
Also also if any startup tried to make a nuclear weapons lab they would be shut down immediately and all the founders would be arrested. [citation needed]
Stop callÂing them labs
From everything Iâve seen, GWWC has totally transformed under your leadership. And I think this transformation has been one of the best things thatâs happened in EA during that time. Iâm so thankful for everything youâve done for this important organization.
Yep! Something like this is probably unavoidable, and itâs what all of my examples below do (BERI, ACE, and MIRI).
There are many examples of organizations with high funding transparency, including BERI (which I run), ACE, and MIRI (transparency page and top contributors page).
I think this dynamic is generally overstated, at least in the existential risk space that I work in. Iâve personally asked all of our medium and large funders for permission, and the vast majority of them have given permission. Most of the funding comes from Open Philanthropy and SFF, both of which publicly announce all of their grantsâwhen recipients decided not to list those funders, itâs not because the funders donât want them to. There are many examples of organizations with high funding transparency, including BERI (which I run), ACE, and MIRI (transparency page and top contributors page).
Nonprofit organizations should make their sources of funding really obvious and clear: How much money you got from which grantmakers, and approximately when. Any time I go on some orgâs website and canât find information about their major funders, itâs a big red flag. At a bare minimum you should have a list of funders, and Iâm confused why more orgs donât do this.
I think people would say that the dog was stronger and faster than all previous dog breeds, not that it was âmore capableâ. Itâs in fact significantly less capable at not attacking its owner, which is an important dog capability. I just think the language of âcapabilityâ is somewhat idiosyncratic to AI research and industry, and Iâm arguing that itâs not particularly useful or clarifying language.
More to my point (though probably orthogonal to your point), I donât think many people would buy this dog, because most people care more about not getting attacked than they do about speed and strength.
As a side note, I donât see why preferences and goals change any of this. Iâm constantly hearing AI (safety) researchers talk about âcapabilities researchâ on todayâs AI systems, but I donât think most of them think those systems have their own preferences and goals. At least not in the sense that a dog has preferences or goals. I just think itâs a word that AI [safety?] researchers use, and I think itâs unclear and unhelpful language.
#taboocapabilities
What is âcapabilitiesâ? What is âsafetyâ? People often talk about the alignment tax: the magnitude of capabilities/âtime/âcost a developer loses by implementing an aligned/âsafe system. But why should we consider an unaligned/âunsafe system âcapableâ at all? If someone developed a commercial airplane that went faster than anything else on the market, but it exploded on 1% of flights, no one would call that a capable airplane.
This idea overlaps with safety culture and safety engineering and is not new. But alongside recent criticism of the terms âsafetyâ and âalignmentâ, Iâm starting to think that the term âcapabilitiesâ is unhelpful, capturing different things for different people.
Thanks for writing and posting this! Iâve had these sorts of feelings floating around in my head for a while, but this is the best term Iâve heard for it.