I think the main benefit of applying to jobs you don’t think you’ll get remains the possibility you might actually get them.
If you’re applying for feedback you’ll usually be disappointed though. Most organizations offer little feedback due to a mix of caution, other things to do and the tendency of “honestly you were pretty OK, the other candidates just stood out as better in a mix of different ways” to be the actual answer, which isn’t really actionable.
Credulous really is the right word. There is a strand of dialogue in EA circles that feels like “we called much of this many years ago” therefor “everything that transpires will mimic our thought experiments perfectly.” The marketing from frontier labs is the offspring of early EA/LW ideas. The potential for confirmation bias here is astronomical.
I also think that when it comes to assessing whether they’re overly trusting of the claims of frontier labs because it fits their broader views, it’s probably more relevant that EAs generally believed Altman and Musk when they said they were founding OpenAI to do philanthropic research when basically everybody else understood what they were really trying to do than that EAs correctly called transformers being a big deal when the average computer scientist was a bit more cautious.
GPT2 was “too dangerous to release” as a marketing strategy too.
I think human responses to pain are more complex than just thresholds too. In addition to your gentle scratching, or the near-zero amount of pain sensations registered when applauding, humans voluntarily inflict non-trivial amounts of pain upon themselves for entertainment and self-worth, and accept inevitability of pain as part of their hobbies. Is it meaningful to argue that a certain number of tattoo artists is more objectionable than a smaller number of puppy kickers, or that selecting someone to play football for 90 minutes which will aggravate their modest limb pain is morally more dubious than intentionally kicking them on that limb? I don’t think so.
And that’s not to mention the benefits of pain which such discussions ignore. For many millenia, one of the most feared diseases was leprosy, which manifests itself in numbness.
When it gets to complex organisms and torture much of it is measured as psychological responses rather than intensity of nervous sensation too, which also doesn’t align well with a neat cardinal pain scale; techniques like waterboarding actually trigger objectively useful reflexes rather than inflicting neurological pain. Is there a certain amount of cumulative experience of unexpected droplets in the nasal system aggregated across many people that approximates a single instance of waterboarding one individual? I would say no. I’d probably also say that there isn’t a certain amount of aggregated BSDM kink that’s worse than one actual torture chamber...
I’d go further and say that the FTX-ish reputation “EA is where extremely wealthy Silicon Valley nerds brag about their generosity whilst mostly funnelling money to people like them and using it as an avenue for self-promotion” also attracts the wrong sort of people—before there were people complaining about FTX being a scam, there were people complaining about the perceived ease of getting funding by FTX attracting the insincere.
(Other negative EA stereotypes contribute to putting well-intentioned people off, but I’m not sure they actually attract the wrong people)
I’m not sure what you’re imagining here. If you give people a trolley problem (only via text) and say on one track, there’s a dog and on the other one, there’s a computer program Eliza and they can chat to either, most would choose to save the dog, even if its only text output were “whoof whoof”.
What I’m imagining, which I evidently didn’t make clear enough, is not a trolley problem but simply trying to discern whether something else is conscious without knowing whether it has “faces, limbs, fur and a squishy body”, such as reading its output [if any] over a remote computer terminal.[1] In these circumstances, not only will humans be unable to find any grounds for empathy with almost all sentient beings, but they will find plenty of grounds to empathise with or at least attribute motivation and intent to software programs.[2] So in the absence of context there’s definitely a bias in mind attribution towards symbol manipulators; even trivially simple ones that merely mimic or perform arithmetic.
On the other hand it seems like the “faces, limbs, fur and a squishy body” are actually a relatively useful heuristic, especially since adults are seldom deceived by taxidermy or cuddly toys in comparison to how easily they’re impressed by “cheap parlour trick” level AI
Are people more likely to empathise with entities with “faces, limbs, fur and a squishy body” than disembodied entities with apparent facility with symbol manipulation? Possibly, though I think this varies,[3] but the relevant question is: are people more likely misplace their empathy in imputing basic consciousness to other mammals or imputing heightened consciousness to anything that can beat them at chess
The level of misattribution matters too. Our sense of empathy anthropomorphises dogs by overestimating their grasp of language and underestimating the extent to which they are motivated by smell, and anthropomorphises irritating repetitive hardcoded chatbots by assigning meaning and motivation which simply doesn’t exist.
Most non-dualists would say consciousness is a feature of information processing (functionalists, illusionists, non-reductive materialists) or something as fundamental as physics (Russelian monism, pan(proto)psychism)...The phrase “rooted in [biochemical processes]” is the least controversial but it still connotes something most might not endorse—i.e. that biology and chemistry is the correct category or level of description
The rooted in biochemical processes is the bit I’m aiming for here; I am not aware of a non-dualist theory which roots human cognition in something other than the biochemical processes of the body (I don’t think the biochemical processes of the body themselves particularly care whether philosophers of mind label them as the consequence of evolutionary imperatives, physics, function, or illusion.)[4] Perhaps I can only be fully confident of my own consciousness, but its relationship with my biochemistry and physiology does at least comes with a bunch of hypotheses originally tested on similar organisms (albeit many of those hypotheses I’d rather not test …).
Like Turing’s eponymous test, only not explicitly a test. Eliza might convince a human not primed to look for evidence it’s just a shoddy computer program that its outputs represent a stream of conscious thought; nobody’s going to try to follow the strains of thought in a dog’s typing...
including those [near]-universally agreed to be too simple to have any sort of motivation, intent or consciousness
No shortage of people who have developed feelings for ChatGPT, and I bet most of them eat cute farm animals
or indeed if between 93% and 99.9% of them live lifestyles too chaste or avant garde to acknowledge the possibility that changes in their hormone balance and neurological state associated with [the prospect of] sex might be linked to evolutionary imperatives to reproduce ;-)
I can experimentally verify claims made about how certain changes to my biochemistry or physiology would affect my consciousness, though in most cases I’d rather not :)
We are evolved to feel empathy—that is, modelling “what it is like to be them”—towards entities that have faces, limbs, fur and a squishy body. We feel a lot of empathy for pets and babies—entities that don’t control language
People empathise with ChatGPT transcripts too; the key distinction is between empathising with a wide range of physiological and behavioural cues as opposed to a narrow one. For all its flaws, observing whether they are like us seems like a more plausible way of establishing likely consciousness than aptitude for symbolic manipulation. Disintermediated by a computer to replace biases introduced by cuteness and non-verbal expressiveness with biases introduced by symbolic manipulation, humans would without exception rate a pocket calculator or Eliza- style toy script as more likely to be conscious than a dog or a two year old child. I don’t think anyone sincerely believes this to actually be the case. A corollary of this is that facility with human language and mathematics—something most entities considered to be conscious do not possess—is not a particularly good standalone proxy for consciousness, even if the entity under examination is much much better at it than Eliza.
Unless we’re positing dualism, what we perceive at consciousness is an emergent property of complex chemical processes rooted in our biology (and the imperatives of our biology to survive and self replicate.. That’s the case whether we empathise with other entities that share this biology, dispassionately analyse the likelihood we evolved from a common ancestor or torture them into demonstrating similar stress hormone responses to humans. That doesn’t necessarily mean warm blooded DNA-replicating machines with limbs and fur and cute eyes are the only possible form of consciousness, but it is something we have and current “AI” doesn’t even have a loose analogue of. And yes, obsession with symbolic manipulation is doing an awful lot of work to explain why people are concerned about the consciousness of assemblies of silicon chips running specific software whilst disregarding the possibility of the sentience of more complex and interesting long running processes like forests, rivers or planetary systems, or indeed larger assemblies of silicon chips running software that doesn’t transform human text into cute replies.
Feels like tractability is the key point here. It doesn’t matter a huge amount if 7 billion is or isn’t the total amount of animals that would counterfactually be saved if all pets were fed vegan diets[1]
What matters is what change can feasibly be achieved by a marginal campaign or food innovation, given that vegan pet food is already a thing which I suspect most vegans are aware of, and most pet owners are not vegans. Also, many vegans are comfortable feeding their pets (or in the case of one person I know, an entire zoo) with omnivorous or carnivorous diets.
I suspect the returns to campaigning would look like marginal returns to vegan advocacy and meat alternatives research for humans, but it feels like this is where the evidence would be most interesting.
the order of magnitude seems plausible when considering how many more animals free ranging domestic cats alone are estimated to kill...
However, I think it misses a big contribution of HIA: demonstrating the absence of a need to risk everything on AGI.
I don’t think this is a real contribution. I don’t think people are trying to make AGI because they are concerned that there will be an insufficient number of high IQ humans alive in the next few decades. I think they’re trying to make it because they think they can.
And also because they [rightly or wrongly] believe that AGI will be more cost effective, more controllable, need less sleep and have higher problem solving potential than even the smartest possible humans. And be here a lot sooner. (And in some of the AGI fantasies, a route to making humans genetically smarter anyway!)
-
Even if one assumes near term “AGI” has a fairly low ceiling,[1] it seems like “intelligence augmentation” is unpromising as an EA intervention.[2] The necessary research is complex, expensive, long term and dependent not just on germline engineering, but on academic research to understand what intelligence is in less shallow terms than we currently do. It’s not clear that there are individual tractable interventions. The quantifiable impact—if it actually worked—would presumably be a tiny proportion of people sufficiently rich and focused on maximising their offspring’s intelligence paying to select a few genes somewhat correlated with intelligence for “designer babies”, with the possibility this might translate enough into real world outcomes to turn a handful of children with already above average prospects into particularly capable and influential individuals. It is not obvious these children will grow up to use their greater talent (real or perceived) for mitigating existential risk or any other sort of greater good[3] Humans with rich, driven parents who’ve been taught about their superiority to ordinary humans from birth don’t sound immune to “alignment problems” either....
As far as germline engineering goes, the more obviously positive quantifiable impacts would be addressing debilitating genetic conditions, where at least we can be confident that the expensive and risky process could alleviate some suffering.
In this case Anthropic chose to supply the DoW via a partnership with a company deeply embedded in the administration’s part of the political spectrum, and even pointedly denied any objections to being used to support the administration’s little expedition in Venezuela, and the administration decided that wasn’t enough. There are many criticisms that can be made of Anthropic’s stance on those issues; reluctance to engage with the current US administration isn’t one of them.
If declining to actively support MAGA’s demands they support development of AI with the explicit purpose of being an autonomous killing device is “virtue signalling”, what’s left of “AI alignment” to pursue?
A useful post and interesting starting point for further discussion
Few more that spring to mind
is there an intuitively plausible alternative/complementary causal factor which might explain the results? Has the study made some sort of attempt to control for this or estimate the effect size?
Does the paper include a large number of hypotheses and find some of them to be statistically significant at the 5 or 10% level.? You would expect that to happen by random chance and it smells of p-hacking, though including them all in the paper is at least more intellectually honest than the alternative approach of testing a lot of hypotheses and only acknowledging the ones that were [incidentally] statistically significant. This is why preregistration is valuable. Note that testing so many hypotheses one of them is bound to be a “finding” is not the same thing as testing whether an association between x and y persists across a large number of alternative regressions including other variables that might plausibly have a relationship; that’s good practice.
Is there a standard methodology for conducting research of this nature in the field? has it been used, and if not, has a plausible rationale for taking a different approach been provided?
would be interesting to hear some of the more specialized ones that organizations like GiveWell and Rethink Priorities that evaluate a lot of research papers in particular fields use.
n.b. on the SMC vs ITN example, I’m fairly confident the answer is that the ITNs are a baseline directly comparable to “no treatment” as they shouldn’t affect the progression from bite to symptomatic malaria infection targeted by SMC at all; they simply reduce the frequency of bites (but not to zero if the sample size is sufficiently large). Prevalance of malarial bites varies between cohorts in “no treatment studies” already. Access to some level of treatment after the fact (HMM) isn’t a problem of study construction either; it complicates comparing severe malaria or death statistics with papers where sufferers may have had no treatment at all, but if anything would probably reduce the reported effect size for SMC. Medical ethics means the appropriate baseline/comparator for lifesaving treatment usually isn’t “do absolutely nothing”, it’s “do the [next] best alternative”
I suspect that any resolution to this dispute is likely to be a lot less public than the OpenAI one
It’s fairly obvious though that Amodei is signalling the company didn’t object to the use of Claude to support the Venezuela operation, and that the company freely choose to be a defense contractor with a formal partnership with Palantir when they had plenty of other revenue/capital sources...
It seems like most of the additional coefficients you’ve added are impossible to estimate with any degree of confidence, particularly when it is plausible the impact may be negative. Whether it was the intention or not, that is the main message I get from your formulation
As someone who is not a strong longtermist, I note that an advantage for using non longtermist heuristics to evaluate impact is that identifying whether an action appears robustly positive for aggregate utility [for humans] on earth up to time t is much easier than anticipating the effect on the Virgo supercluster after time t
(A more sophisticated approach might use discounting for temporal and extreme spatial distance rather than time bounding, but amounts to the same thing; attaching zero weight to the estimated impact of my actions on the Virgo supercluster a thousand years from now)
EAs were also warning, for a long time, about the importance of health aid sent overseas. In contrast, non-EA leftists were more likely to call these institutions colonialist and call EAs racist for neglecting domestic political issues. But when Trump got elected, we were vindicated in the worst of ways: he destroyed much of USAID, and this was the single act in his presidency that led to the most deaths.
This feels like a strange argument to make, and one which seems to be trying way too hard to find evidence of vindication even in failures, which ironically is the opposite of what people with good epistemics should be doing. EAs were criticised [by critics whose arguments greatly varied in quality] for tending to treat international aid as primarily an optimization problem best addressed by small specialist charities and individuals maximising their donations, and largely ignoring the political dimension.
Then domestic political issues killed government programmes funding traditional Big Aid multinational aid agencies[1] with the stroke of a pen[2] and did far more damage than EA philanthropy is able to repair.
Directionally, thats the opposite of a validation of EA orthodoxy on aid.
I don’t think neglecting the politics of whether aid actually gets disbursed is a strong argument against EA either—not least because I don’t think EAs would have been able to dissuade people from voting Trump even if they’d made it their leading cause area, or convince Trump/Musk that foreigners lives mattered—but it’s definitely not one where the “don’t neglect politics” and “actually big programs that aren’t quite as good as AMF are still really good” critics can be said to have lost the argument.
(programs not run by EAs, but admired by some of them for their results)
For added irony, the person who gleefully signed those death warrants was at least superficially EA-adjacent enough to have enthusiastically endorsed MacAskill’s writing and funded a couple of longtermist organizations in the past.
It would probably be worthwhile to encourage legally binding versions of the Giving Pledge in general.
Donations before death are optimal, but it’s particularly easy to ensure that the pledge is met at that stage with a will which can be updated at the time of signing it. (I presume most of the 64% did have a will, but chose to leave their fortune to others. I guess it’s possible some fortunes inherited by widow[er]s will be donated to pledged causes in the fullness of time).
I don’t think this should replace the Giving Pledge; some people’s intentions and financial situations are too complex to write into a binding contract, but such pledges should be taken more seriously (even though in practice they are still likely to be reversible).
Meta is paying billions of dollars to recruit people with proven experience at developing relevant AI models.
Does the set of “people with proven experience in building AI models” overlap with “people who defer to Eliezer on whether AI is safe” at all? I doubt it.
Indeed given that Yudkowsky’s arguments on AI are not universally admired and people who have chosen building the thing he says will make everybody die as their career are particularly likely to be sceptical about his convictions on that issue, an endorsement might even be net negative.
The opportunity cost only exists for those with a high chance of securing comparable level roles in AI companies, or very senior roles at non-AI companies in the near future. Clearly this applies to some people working in AI capabilities research,[1] but if you wish to imply this applies to everyone working at MIRI and similar AI research organizations, I think the burden of proof actually rests on you. As for Eliezer, I don’t think his motivation for dooming is profit, but it’s beyond dispute that dooming is profitable for him. Could he earn orders of magnitude more money from building benevolent superintelligence based on his decision theory as he once hoped to? Well yes, but it’d have to actually work.[2]
Anyway, my point was less to question MIRI’s motivations or Thomas’ observation Nate could earn at least as much if he decided to work for a pro-AI organization and more to point out that (i) no, really, those industry norm salaries are very high compared with pretty much any quasi-academic research job not related to treating superintelligence as imminent and especially to roles typically considered “altruistic” and (ii) if we’re worried that money gives AI company founders the wrong incentives, we should worry about the whole EA-AI ecosystem and talent pipeline EA is backing. Especially since that pipeline incubated those founders.
$235K is not very much money. I made close to Nate’s salary as basically an unproductive intern at MIRI.
I understand the point being made (Nate plausibly could get a pay rise from an accelerationist AI company in Silicon Valley, even if the work involved was pure safetywashing, because those companies have even deeper pockets), but I would stress that these two sentences underline just how lucrative peddling doom has become for MIRI[1] as well as how uniquely positioned all sides of the AI safety movement are.
There are not many organizations whose messaging has resonated with deep pocketed donors to the extent that they can afford to pay their [unproductive] interns north of $200k pro rata to brainstorm with them.[2] Or indeed up to $450k to someone with interesting ideas for experiments to test AI threats, communication skills and at least enough knowledge of software to write basic Python data processing scripts. So the financial motivations to believe that AI is really important are there on either side of the debate; the real asymmetry is between the earning potential of having really strong views on AI vs really strong views on the need to eliminate malaria or factory farming.
tbf to Eliezer, he appears to have been prophesizing imminent tech-enabled doom/salvation since he was a teenager on quirky extropian mailing lists, so one thing he cannot be accused of is bandwagon jumping.
Outside the Valley bubble, plenty of people at profitable or well-backed companies with specialist STEM skillsets or leadership roles are not earning that for shipping product under pressure, never mind junior research hires for nonprofits with nominally altruistic missions
Easier to persuade commercial entities of the merits of making more money (by incidentally doing the right thing) than persuade a reviewer of multiple competitive funding bids scoped for habitat preservation to fund a study into lab grown meat. At the end of the day, the proposals written by biodiversity enthusiasts with biodiversity rationales and very specific biodiversity metrics are just going to be more plausible,[1] even if they turn out to be ineffective.
For similar reasons, I don’t expect EA animal welfare funds to award funding to an economic think tank proposing to research how to grow the economy, even if the economic think tank insists its true goal is animal welfare and provides a lot of evidence that investment in meat alternatives and enforcement of animal welfare legislation is linked to overall economic growth.
Biobanks and biodiversity charity effectiveness research might stand a chance, obviously
Also failures trying to do really outlandish things like bribing Congresspeople to endorse Jim Mattis as a centrist candidate in the 2024 US Presidential Election are likely to backfire in more spectacular ways than (say) providing malaria nets for a region with falling malaria or losing a court case against a factory farming conglomerate. That said, this criticism does apply to some other things EAs are interested in, particularly actions purportedly addressing x-risks.
Think this is a good post. But the specific example below feels like so much of a mischaracterization of typical debate it might actually illustrate part of the problem.
Whilst you may have experienced specific counterexamples, I think relatively few objectors to EA are questioning the principle that restoring the sight of 1000 people might plausibly be agreed to be a preferable outcome to moderately aiding one blind person[1] as opposed to generalising from this to what EA actually is, which is a much bolder set of claims and priorities than “sometimes you can help more people achieve the same ends with the same amount of money”
tbh even if they are explicitly asking an EA to state something as fundamental as how they can treat helping 1000 blind people > 1 blind person as established fact, if they’re someone who’s ever had a conversation with an EA before there’s a decent chance they’re doing so to ward off the EA promptly substituting 2000 chickens or
priorsconvenient imputed probability claims into the equation! And, in general, I think people asking why EA is so convinced it’s so much better at prioritising and insisting philanthropic choices are arbitrary aren’t doing so because they don’t think cost benefit analysis can be done in any circumstances[2] they’re asking because of how many other assumptions are necessarily smuggled in to even attempt cause neutrality (or because they think the assumptions smuggled in are wrong,[3] or just not actually that neutral).Obviously this isn’t because EAs refuse to debate their assumptions at all. Indeed they love nothing more than respectful debate with people who like to state their own priors and utility ranges, and even someone questioning the validity of the whole EA utilitarian consequentialist framework might get entertained so long as they establish themselves as being suitably intellectual if they contextualise it by explaining their own ethical framework first. But there’s a strong tendency to assume that if people don’t come armed with the right jargon and specific alternatives or at least have the decency to state that their question is axiological they must just not understand anything, or actually reject the whole edifice of science.[4] I think even when delivered without a hint of condescension that sort of talking past people is more annoying than the bluntness[5]
-
Similarly I don’t think advising people midway through medical school to retrain as AI researchers is a form of fanaticism which is just unrealistic enough to be annoying (though it’s an excellent example of unrealistic enough to be annoying fanaticism). It’s also a fundamental error based on evaluating outsider professions differently from the recommended ones: completely forgetting that if we’re dismissing the impact of choosing to become a doctor by comparing them with the counterfactual of the next best applicant like 80k hours says, we should we should treat the impact of the alternative career the same way. Which means the impact of a couple of medical students career changing to the even more competitive profession of AI safety should be measured by how much more likely they are to solve AI alignment than the multiple CompSci graduates who’ve been obsessing over it for five years applying for the same positions. Even if they somehow find a job in that field they’re unlikely to tangibly change it[6]. It’s not just the fanaticism producing recommendations that are annoying or too late, it’s also that cargo culting the arguments produces recommendations that are plain bad.[7]
There are obviously also explicit arguments for preferring to help one person over 1000s far away based on axiological assumptions about duty to particular communities or individuals, but yeah, those arguments don’t tend to be made implicitly in the form of a question....
again, people (including EAs) sometimes have arguments against the applying it to particular fields and evidence bases which are really good but they tend to explicitly make them
there are lots of epistemological questions about recommendations that have a better answer than “don’t you know the difference between opinion and fact?” too: the better EA analyses are explicit about their evidence base and the extent to which it represents a knowledge claim and their level of confidence X actually does deliver Z more cost effectively than Y
I’m not saying bad questions don’t exist, I’m saying that reasonable ones often get framed as unreasonable ones because lecturing people about how cost benefit analysis is just maths and they surely aren’t arguing with maths is easier than justifying the use of “moral weights”
tbh online at least I think EA’s style of debate is more likely to be accused—fairly and otherwise—of annoyingly polite persistence
nor will the modal AI researcher...
tbf the other default EA recommendation of doing medical research instead probably isn’t a bad suggestion in this instance. Or maybe they should study how to use AI to do medical research to make everyone happy ;)