Non-EA interests include chess and TikTok (@benthamite). Formerly @ CEA, METR + a couple now-acquired startups.
Ben_Westđ¸
Karma: 16,244
Sure, not the metaphor I would use but I broadly agree that applicants who are willing to plug the metaphorical USB stick into their computer (e.g. by following people they want to work with and applying to the jobs that they post) have a much lower rejection rate.
Sure, âhiring managers being bad at marketing is the bottleneck, not fundingâ is at least partially true. It still implies that if you happen to stumble across a poorly advertised position, you shouldnât expect the acceptance rate to be low!
My version of Mattâs critique that you quoted is something like:
Imagine youâre running a mining company, and you want to start mining Venus. You could either embark on a massive terraforming project to make Venus habitable by biological humans who can work in your mining colony, or you could just build a bunch of robots who can naturally withstand Venusâ climate, think faster than a human, make better decisions than a human, etc. etc. What do you do?
Obviously you are going to choose to send the robots, and the robots arenât going to want to eat meat, so you donât need to worry about factory farming on Venus.I donât think this argument is bulletproof. For example, ports in the U.S. are required to pay human dock workers to sit around and do nothing after their jobs had been automated. I could imagine some sort of analogous regulatory capture in the future which would require mining companies to send humans to other planets even when robots would be more efficient. Preventing this kind of lock-in is one of the few interventions targeting a post-singularity world that I feel positive about.
this claim feels at odds with what I understood your perspective to be from the shallow review you did a while ago
Yeah, I think I did a bad job explaining my views there. Could you say more about what you thought I believed? I should maybe update the post.
However, it seems like this would fall into the bucket of ânot actionable unless you work directly on AI,â so it seems like it might be practically useful to act as if this wasnât going to be the case?
Good question and I claim the answer is ânoâ because you can work on AI! E.g. The Midas Project (founded by a former THL campaigner) is bringing corporate campaign tactics to AI safety. See also e.g. AI Safetyâs Biggest Talent Gap Isnât Researchers. Itâs Generalists.
I think thereâs a decent chance that AI wonât actually be very transformative (or at least wonât be transformative soon) and therefore itâs reasonable to bet your career plans on that assumption. But, to the extent you think AI is actually going to be a big deal, I would suggest considering working on it!
Fair. âIn good timelines some humans continue to exist despite not really being a major force analogous to how the Amish exist today but arenât really a major forceâ is probably the modal view amongst people I talk to.
I shouldnât have used the term âTAIâ here; Iâve clarified. Thanks!
What I said was unclear; I probably should have just quoted Holden Karnofsky:
I think there is a good chance that:
During the century weâre in right now, we will develop technologies that cause us to transition to a state in which humans as we know them are no longer the main force in world events. This is our last chance to shape how that transition happens.
Whatever the main force in world events is (perhaps digital people, misaligned AI, or something else) will create highly stable civilizations that populate our entire galaxy for billions of years to come. The transition taking place this century could shape all of that.
I think itâs very unclear whether this would be a good or bad thing.
There are some people (e.g. Evitable) who are trying to stop this transition, but I think more AI safety people would self-describe their work as âshap[ing] how that transition happens.â
What Matt calls efficient many would call âweirdâ and hence undesirable. Iâm not sure we have a reason to think AI will mitigate this rather than amplify it.
In case you havenât seen it, one argument for why AI might amplify (or at least lock-in) traditions was given by Buck here: Christian homeschoolers in the year 3000.
Thanks for writing this! I appreciate you flagging:
Here, Iâm conditioning on humans as we know them continuing to exist. I take no position on how likely this is to happen, but it seems to me like this is the outcome that EAs are pushing for when trying to reduce existential risk from AI.
I am not aware of anyone who works on AI safety who would say that this is what they are pushing for, with the exception of people who are pushing for a complete pause in AI development.
The rest of us are generally resigned to biological humans disappearing once we have transformative AI [edit: this was unclear, see update], even under the most optimistic scenarios. I expect that this is a major way in which the AI safety people I know (including Matt) find arguments like this and Benthamâs Bulldogsâ uncompelling.
Interesting post, thanks! On this:
Itâs possible that the lack of evidence has been accounted for in other ways. Perhaps someone who initially guesses a 20% chance of extinction is subtly dropping that down to 5% on the grounds of epistemic modesty. But itâs unlikely they are doing so in the exact right way to counteract the effect of the optimizerâs curse.
My understanding is that worldview diversification partially addresses things like this (see this old critique from Holden Karnofsky, which makes a similar point to yours and I think is intellectually upstream of CGâs later thinking), in addition to accounting for e.g. normative uncertainty.
Iâm not exactly sure how worldview diversification works (maybe someone from CG can comment) but I share your skepticism that itâs being done in exactly the right way to counteract these effects.
+1, I think cluelessness-type objections are some of the strongest objections to my own work and I would be excited to see more discussion about it.
Thanks for the questions!
Are you saying that people would read âseniorâ in a job description as meaning âolderâ rather than âmore experiencedâ?
No, Iâm saying that they would interpret it to mean âhaving more years of formal experience (rather than e.g. having had a wider variety of experiences, or having had more useful experiences)â and I instead want a word which means âmore skilledâ.
Can you elaborate on your reluctance to hire an âoldâ person?
No reluctance! I check the â20+ years of experienceâ box on eag applications myself. I just am bemoaning the fact that the word âseniorâ indicates both age and skill, and I want a word which only applies to the latter.
At least, people should say that the field is bottlenecked on highly skilled generalists.
Thoughts on how to do this as a hiring manager? Things Iâve considered:
Title the role âSenior [whatever].â I think this is ok, but in many fields âseniorâ is a synonym for âoldâ, so this title causes talented young people to not apply (and untalented old people to apply).
Say âif you can do X, then you should applyâ. This is ideal, but itâs often hard to give an objective enough test that it doesnât end up just effectively being a test of the candidateâs self-confidence.
Say âif you have done X, then you should applyâ. Easier to evaluate objectively than (2), but artificially limits the candidate pool to only people who have done something very similar before.
Literally say âhighly skilled generalist.â Seems kind of pretentious, and it also seems like itâs effectively a self-confidence test for the candidate.
Ask for referrals from people who I know well enough that I can effectively say âhighly skilled generalistâ and they will apply that criterion in a way that I would endorse. This is good but means I donât hire from outside my circle.
Iâm curious what you think economic orthodoxy is if Hansonâs claim â[AI will cause the world economy to] double every few months or fasterâ is more of a disagreement with EA orthodoxy than economic orthodoxy?
(Iâm also interested in what you think the EA orthodoxy is; âmonths-long doubling timesâ feels like a pretty mainstream view even amongst people working full time in AI safety, though I agree that it differs from, e.g. Eliezer Yudkowsky c. 2008.)
Thanks for writing this! Iâm on the other side (wanting to hire generalists) and I share your skepticism about the âwrite publiclyâ advice. I think the actual advice should be more like âclearly demonstrate success in the thing you are trying to get hired for,â which usually doesnât involve writing.
(Although I do think that generalist roles can be particularly hard to demonstrate success in. For example, I expect that writing about recruiting might actually be one of the better ways to clearly demonstrate your skill in recruiting, even though recruiting per se doesnât usually involve much writing.)
Thanks for writing this up! I hadnât realized that the feb protest was so big.
This is super cool, thanks for doing it.
What does the average scholar think about this situation? âŚThey think that the work falls beneath standards because it has not been through the system of review which nearly all scholars think is required for work to reliably meet minimal scholarly standards.
This is not my experience. The arxiv version of my paper has been cited 97 times; the peer-reviewed version 7 times. The only time I can remember someone saying the paper shouldnât be trusted because of not having been peer-reviewed yet was, ironically, on the EA Forum.
You can compare the peer review comments with those on LessWrong. Neither is a pareto improvement over the other, but what in hindsight has proven to be the strongest critique (that growth is actually superexponential, not exponential, because of e.g. post-training) was only mentioned on LW (and twitter).
Peer review has many things to recommend it but, as might be guessed from a post whose âCriticism of peer reviewâ section consists solely of claims that all criticism of peer review is invalid, this post is overstating the case.
Surprising that your MP was willing to meet with you for so long. Thanks for doing this (and writing it up)!