I agree, and also immediately thought of pure mathematics as a counterexample. E.g., if one’s most important goal was to prove the Riemann hypothesis, then I claim (based on my personal experience of doing maths, though e.g. Terence Tao seems to agree) that it’d be a very bad strategy to only do things where one has an end-to-end story for how they might contribute to a proof of the Riemann hypothesis. This is true especially if one is junior, but I claim it would be true even for a hypothetical person eventually proving the Riemann conjecture, except maybe in some of the very last stages of them actually figuring out the proof.
I think the history of maths also provides some suggestive examples of the dangers of requiring end-to-end stories. E.g., consider some famous open questions in Ancient mathematics that were phrased in the language of geometric constructions with ruler and compass, such as whether it’s possible to ‘square the circle’. It was solved 2,000 years after it was posed using modern number theory. But if you had insisted that everyone working on it has an end-to-end story for how what they’re doing contributes to solving that problem, I think there would have been a real risk that people continue thinking purely in ruler-and-compass terms and we never develop modern number theory in the first place.
The Planners vs. Hayekians distinction seems related. The way I’m understanding Buck is that he thinks that, at least within AI alignment, a Planning strategy is superior to a Hayekian one (i.e. roughly one based on optimizing robust heuristics rather than an end-to-end story).
One of the strongest defenses of Buck’s original claim I can think of would appeal specifically to the “preparadigmatic” stage of AI alignment. I.e. roughly the argument would be: sure, perhaps in areas where we know of heuristics that are robustly good to pursue it can sometimes be best to do so; however, the challenge with AI alignment precisely is that we do not know of such heuristics, hence there simply is no good alternative to having an end-to-end story.
Thanks for sharing your perspective. I find it really helpful to hear reactions from practitioners.
This discussion (incl. child comments) was one of the most interesting things I read in the last weeks, maybe months. - Thank you for having it publicly. :)
(FWIW, when reading the above discussion I independently had almost exactly the same reaction as the following before reading it in Richard’s latest comment:
This argument feels to me like saying “We shouldn’t keep building bigger and bigger bombs because in the limit of size they’ll form a black hole and destroy the Earth.”
Minor: The in-text links to endnotes link to the page for making a new forum post, which presumably is not intended.
Minor: The following sentence right before section 4 starts seems jumbled?
This is true for when the magnitude of warming was comparable to what we are in for in the next 200 years, and when, on a regional basis, the rate of warming was comparable to what we are in for in the next 200 years.
Excellent post. I highly value such summaries of known and relevant factual information, and views on lessons we can learn. Thank you for putting this together.
I’m also curious what prompted you to look into this topic?
[Epistemic status: info from the WHO website and Wikipedia, but I overall invested only ~10 min, so might be missing something.]
It seems my remarks do apply for “public health emergency of international concern (PHEIC)” instead of “pandemic”. For example, from Wikipedia:
Under the 2005 International Health Regulations (IHR), states have a legal duty to respond promptly to a PHEIC.
[Note by me: The International Health Regulations include multiple instances of “public health emergency of international concern”. By contrast, they include only one instance of “pandemic”, and this is in the term “pandemic influenza” in a formal statement by China rather than the main text of the regulation.]
The WHO declared a PHEIC due to COVID-19 on January 30th.
The OP was prompted by a claim that the timing of the WHO using the term “pandemic” provides an argument against epistemic modesty. (Though I appreciate this was less clear in the OP than it could have been, and maybe it was a bad idea to copy my Facebook comment here anyway.) From the Facebook comment I was responding to:
For example, to me, the WHO taking until ~March 12 to call this a pandemic*, when the informed amateurs I listen to were all pretty convinced that this will be pretty bad since at least early March, is at least some evidence that trusting informed amateurs has some value over entirely trusting people usually perceived as experts.
Since the WHO declaring a PHEIC seems much more consequential than them using the term “pandemic”, the timing of the PHEIC declaration seems more relevant for assessing the merits of the WHO response, and thus for any argument regarding epistemic modesty.
Since the PHEIC declaration happened significantly earlier, any argument based on the premise that it happened too late is significantly weaker. And whatever the apparent initial force of this weaker argument, my undermining response from the OP still applies.
So overall, while the OP’s premise appealing to major legal/institutional consequences of the WHO using the term “pandemic” seems false, I’m now even more convinced of the key claim I wanted to argue for: that the WHO response does not provide an argument against epistemic modesty in general, nor for the epistemic superiority of “informed amateurs” over experts on COVID-19.
Thank you for pointing this out! It sounds like my guess was probably just wrong.
My guess was based on a crude prior on international organizations, not anything I know about the WHO specifically. I clarified the epistemic status in the OP.
[Epistemic status: speculation based on priors about international organizations. I know next to nothing about the WHO specifically.]
[On the WHO declaring COVID-19 a pandemic only (?) on March 12th. Prompted by this Facebook discussion on epistemic modesty on COVID-19.]
- [ETA: this point is likely wrong, cf. Khorton’s comment below. However, I believe the conclusion that the timing of WHO declarations by itself doesn’t provide a significant argument against epistemic modesty still stands, as I explain in a follow-up comment below.] The WHO declaring a pandemic has a bunch of major legal and institutional consequences. E.g. my guess is that among other things it affects the amounts of resources the WHO and other actors can utilize, the kind of work the WHO and others are allowed to do, and the kind of recommendations the WHO can make.
- The optimal time for the WHO to declare a pandemic is primarily determined by these legal and institutional consequences. Whether COVID-19 is or will in fact be a pandemic in the everyday or epidemiological sense is an important input into the decision, but not a decisive one.
- Without familiarity with the WHO and the legal and institutional system it is a part of, it is very difficult to accurately assess the consequences of the WHO declaring a pandemic. Therefore, it is very hard to evaluate the timing of the WHO’s declaration without such familiarity. And being even maximally well-informed about COVID-19 itself isn’t even remotely sufficient for an accurate evaluation.
- The bottom line is that the WHO officially declaring that COVID-19 is a pandemic is a totally different thing from any individual persuasively arguing that COVID-19 is or will be a pandemic. In a language that would accurately reflect differences in meaning, me saying that COVID-19 is a pandemic and the WHO declaring COVID-19 is a pandemic would be done using different words. It is simply not the primary purpose of this WHO speech act to be an early, accurate, reliable, or whatever indicator of whether “COVID-19 is a pandemic”, to predict its impact, or any other similar thing. It isn’t primarily epistemic in any sense.
- If just based on information about COVID-19 itself someone confidently thinks that the WHO ought to have declared a pandemic earlier, they are making a mistake akin to the mistake reflected by answering “yes” to the question “could you pass me the salt?” without doing anything.
So did the WHO make a mistake by not declaring COVID-19 to be a pandemic earlier, and if so how consequential was it? Well, I think the timing was probably suboptimal just because my prior is that most complex institutions aren’t optimized for getting the timing of such things exactly right. But I have no idea how consequential a potential mistake was. In fact, I’m about 50-50 on whether the optimal time would have been slightly earlier or slightly later. (Though substantially earlier seems significantly more likely optimal than substantially later.)
I’ve also heard that 40-70% figure (e.g. from German public health officials like the director of Germany’s equivalent of the CDC). But I’m confused for the reason you stated. So I’d also appreciate an answer.
Some hypotheses (other than the 40-70% being just wrong) I can think of, though my guess is none of them is right:
(a) The 40-70% are a very long-term figure like risk of life-time infection assuming that the virus becomes permanently endemic.
(b) There being many more undetected than confirmed cases.
(c) The slowdown in new cases in Hubei only being temporary, i.e. expecting it to accelerate again and reaching 40-70% there.
(d) Thinking that the virus will spread more widely outside of Hubei, e.g. because one expects less drastic prevention/mitigation measures. [ETA: This comment seems to point to (d).]
I don’t think that peculiarities of what kinds of EA work we’re most enthusiastic about lead to much of the disagreement. When I imagine myself taking on various different people’s views about what work would be most helpful, most of the time I end up thinking that valuable contributions could be made to that work by sufficiently talented undergrads.
I agree we have important disagreements other than what kinds of EA work we’re most enthusiastic about. While not of major relevance for the original issue, I’d still note that I’m surprised by what you say about various other people’s view on EA, and I suspect it might not be true for me: while I agree there are some highly-valuable tasks that could be done by recent undergrads, I’d guess that if I made a list of the most valuable possible contributions then a majority of the entries would require someone to have a lot of AI-weighted generic influence/power (e.g. the kind of influence over AI a senior government member responsible for tech policy has, or a senior manager in a lab that could plausibly develop AGI), and that because of the way relevant existing institutions are structured this would usually require a significant amount of seniority. (It’s possible for some smart undergrads to embark on a path culminating in such a position, but my guess this is not the kind of thing you had in mind.)
I am pretty skeptical of this. Eg I suspect that people like Evan (sorry Evan if you’re reading this for using you as a running example) are extremely unlikely to remain unidentified, because one of the things that they do is think about things in their own time and put the results online. [...]
I am not intending to include beliefs and preferences in my definition of “great person”, except for preferences/beliefs like being not very altruistic, which I do count.
I don’t think these two claims are plausibly consistent, at least if “people like Evan” is also meant to exclude beliefs and preferences: For instance, if someone with Evan-level abilities doesn’t believe that thinking in their own time and putting results online is a worthwhile thing to do, then the identification mechanism you appeal to will fail. More broadly, someone’s actions will generally depend on all kinds of beliefs and preferences (e.g. on what they are able to do, on what people around them expect, on other incentives, …) that are much more dependent on the environment than relatively “innate” traits like fluid intelligence. The boundary between beliefs/preferences and abilities is fuzzy, but as I suggested at the end of my previous comment, I think for the purpose of this discussion it’s most useful to distinguish changes in value we can achieve (a) by changing the “environment” of existing people vs. (b) by adding more people to the pool.
Could you name a profile of such a person, and which of the types of work I named you think they’d maybe be as good at as the people I named?
What do you mean by “profile”? Saying what properties they have, but without identifying them? Or naming names or at least usernames? If the latter, I’d want to ask the people if they’re OK with me naming them publicly. But in principle happy to do either of these things, as I agree it’s a good way to check if my claim is plausible.
I think my definition of great might be a higher bar than yours, based on the proportion of people who I think meet it?
Maybe. When I said “they might be great”, I meant something roughly like: if it was my main goal to find people great at task X, I’d want to invest at least 1-10 hours per person finding out more about how good they’d be at X (this might mean talking to them, giving them some sort of trial tasks etc.) I’d guess that for between 5 and 50% of these people I’d eventually end up concluding they should work full-time doing X or similar.
Also note that originally I meant to exclude practice/experience from the relevant notion of “greatness” (i.e. it just includes talent/potential). So for some of these people my view might be something like “if they did 2 years of deliberate practice, they then would have a 5% to 50% chance of meeting the bar for X”. But I know think that probably the “marginal value from changing the environment vs. marginal value from adding more people” operationalization is more useful, which would require “greatness” to include practice/experience to be consistent with it.
If we disagree about the bar, I suspect that me having bad models about some of the examples you gave explains more of the disagreement than me generally dismissing high bars. “Functional programming” just doesn’t sound like the kind of task to me with high returns to super-high ability levels, and similar for community building; but it’t plausible that there are bundles of tasks involving these things where it matters a lot if you have someone whose ability is 6 instead of 5 standard deviations above the mean (not always well-defined, but you get the idea). E.g. if your “task” is “make a painting that will be held in similar regards as the Mona Lisa” or “prove P != NP” or “be as prolific as Ramanujan at finding weird infinite series for pi”, then, sure, I agree we need an extremely high bar.
For what it’s worth, I think that you’re not credulous enough of the possibility that the person you talked to actually disagreed with you—I think you might doing that thing whose name I forget where you steelman someone into saying the thing you think instead of the thing they think.
Thanks for pointing this out. FWIW, I think there likely is both substantial disagreement between me and that person and that I misunderstood their view in some ways.
You might also be interested in John Halstead’s and Johannes Ackva’s recent Climate & Lifestyle Report for Founders Pledge. They point out that taking into account policy effects can dramatically change the estimated climate impact of lifestyle choices, and on children specifically they say that:
The biggest discrepancy here concerns the climate effect of having children. For the reasons given, we think our estimate of the effect of having children is more accurate for people living in the EU or US states with strong climate policy, such as California, New York, as well as other states in the Northeast. Indeed, even outside the US states with strong climate policy, we think the estimate accounting for policy is much closer to the truth, since emissions per head are also declining at the national level, and climate policy is likely to strengthen across the US in the next few decades.
After taking into account policy effects, they find that the climate impact of having children is comparable to some other lifestyle choices such as living car-free. (I.e. it’s not the case that the climate impact of having children is orders of magnitude larger, as one might naively think w/o considering policy effects.)
For more detail, see their section 3.
I agree that blanket endorsements of anti-natalism (whether for climate or other reasons) in EA social media spaces are concerning, and I appreciate you taking the time to write down why you think they are misguided.
FWIW, my reaction to this post is: you present a valid argument (i.e. if I believed all your factual premises, then I’d think your conclusion follows), but this post by itself doesn’t convince me that the following factual premise is true:
The magnitude of [your kids’] impact on the climate is likely to be much, much smaller than any of the three other factors I have raised.
At first glance, this seems highly non-obvious to me. I’d probably at least want to see a back-of-the-envelope calculation before believing this is right.
(And I’m not sure it is: I agree that your kids’ impact on the climate would be more causally distant than their impact on your own well-being, your career, etc. However, conversely, there is a massive scale difference: impacts on climate affect the well-being of many people in many generations, not just your own. Notably, this is also true for impacts on your career, in particular if you try to improve the long-term future. So my first-pass guess is that the expected impact will be dominated by the non-obvious comparison of these two “distant” effects.)
Thanks, very interesting!
I agree the examples you gave could be done by a recent graduate. (Though my guess is the community building stuff would benefit from some kinds of additional experience that has trained relevant project management and people skills.)
I suspect our impressions differ in two ways:
1. My guess is I consider the activities you mentioned less valuable than you do. Probably the difference is largest for programming at MIRI and smallest for Hubinger-style AI safety research. (This would probably be a bigger discussion.)
2. Independent of this, my guess would be that EA does have a decent number of unidentified people who would be about as good as people you’ve identified. E.g., I can think of ~5 people off the top of my head of whom I think they might be great at one of the things you listed, and if I had your view on their value I’d probably think they should stop doing what they’re doing now and switch to trying one of these things. And I suspect if I thought hard about it, I could come up with 5-10 more people—and then there is the large number of people neither of us has any information about.
Two other thoughts I had in response:
It might be quite relevant if “great people” refers only to talent or also to beliefs and values/preferences. E.g. my guess is that there are several people who could be great at functional programming who either don’t want to work for MIRI, or don’t believe that this would be valuable. (This includes e.g. myself.) If to count as “great person” you need to have the right beliefs and preferences, I think your claim that “EA needs more great people” becomes stronger. But I think the practical implications would differ from the “greatness is only about talent” version, which is the one I had in mind in the OP.
One way to make the question more precise: At the margin, is it more valuable (a) to try to add high-potential people to the pool of EAs or (b) change the environment (e.g. coordination, incentives, …) to increase the expected value of activities by people in the current pool. With this operationalization, I might actually agree that the highest-value activities of type (a) are better than the ones of type (b), at least if the goal is finding programmers for MIRI and maybe for community building. (I’d still think that this would be because, while there are sufficiently talented people in EA, they don’t want to do this, and it’s hard to change beliefs/preferences and easier to get new smart people excited about EA. - Not because the community literally doesn’t have anyone with a sufficient level of innate talent. Of course, this probably wasn’t the claim the person I originally talked to was making.)
(The following summary [not by me] might be helpful to some readers not familiar with the book:
I almost never read the EA Facebook group. But I tend to generally dislike Facebook, and there simply is no Facebook group I regularly use. I think I joined the EA Facebook group in early 2016, though it’s possible that it was a few months earlier or later. (In fact, I didn’t have a Facebook account previously. I only created one because a lot of EA communication seemed to happen via Facebook, which I found somewhat annoying.) Based on my very infrequent visits, I don’t have a sense that it changed significantly. But I’m not sure if I would have noticed.
[On https://www.technologyreview.com/s/615181/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/ ]
[ETA: After having talked to more people, it now seems to me that disagreeing on this point more often explains different reactions than I thought it would. I’m also now less confident that my impression that there wasn’t bad faith from the start is correct, though I think I still somewhat disagree with many EAs on this. In particular, I’ve also seen plenty of non-EA people who don’t plausibly have a “protect my family” reaction say the piece felt like a failed attempt to justify a negative bottom line that was determined in advance.] (Most of the following doesn’t apply in cases where someone is acting in bad faith and is determined to screw you over. And in fact I’ve seen the opposing failure mode of people assuming good faith for too long. But I don’t think this is a case of bad faith.)
I’ve seen some EAs react pretty negatively or angrily to that piece. (Tbc, I’ve also seen different reactions.) Some have described the article as a “hit piece”.
I don’t think it qualifies as a hit piece. More like a piece that’s independent/pseudo-neutral/ambiguous and tried to stick to dry facts/observations but in some places provides a distorted picture by failing to be charitable / arguably missing the point / being one-sided and selective in the observation it reports.
I still think that reporting like this is net good, and that the world would be better if there was more of it at the margin, even if it has flaws similarly severe to that one. (Tbc, I think there would have been a plausibly realistic/achievable version of that article that would have been better, and that there is fair criticism one can direct at it.)
To put it bluntly, I don’t believe that having even maximally well-intentioned and intelligent people at key institutions is sufficient for achieving a good outcome for the world. I find it extremely hard to have faith in a setup that doesn’t involve a legible system/structure with things like division of labor, checks and balances, procedural guarantees, healthy competition, and independent scrutiny of key actors. I don’t know if the ideal system for providing such outside scrutiny will look even remotely like today’s press, but currently it’s one of the few things in this vein that we have for nonprofits, and Karen Hao’s article is an (albeit flawed) example of it.
Whether this specific article was net good or not seems pretty debatable. I definitely see reasons to think it’ll have bad consequences, e.g. it might crowd out better reporting, might provide bad incentives by punishing orgs for trying to do good things, … I’m less wedded to a prediction of this specific article’s impact than to the broader frame for interpreting and reacting to it.
I find something about the very negative reactions I’ve seen worrying. I of course cannot know what they were motivated by, but some seemed like I would expect someone to react who’s personally hurt because they judge a situation as being misunderstood, feels like they need to defend themself, or like they need to rally to protect their family. I can relate to misunderstandings being a painful experience, and have sympathy for it. But I also think that if you’re OpenAI, or “the EA community”, or anyone aiming to change the world, then misunderstandings are part of the game, and that any misunderstanding involves at least two sides. The reactions I’d like to see would try to understand what has happened and engage constructively with how to productively manage the many communication and other challenges involved in trying to do something that’s good for everyone without being able to fully explain your plans to most people. (An operationalization: If you think this article was bad, I think that ideally the hypothesis “it would be good it we had better reporting” would enter your mind as readily as the hypothesis “it would be good if OpenAI’s comms team and leadership had done a better job”.)
[Is longtermism bottlenecked by “great people”?]
Someone very influential in EA recently claimed in conversation with me that there are many tasks X such that (i) we currently don’t have anyone in the EA community who can do X, (ii) the bottleneck for this isn’t credentials or experience or knowledge but person-internal talent, and (iii) it would be very valuable (specifically from a longtermist point of view) if we could do X. And that therefore what we most need in EA are more “great people”.
I find this extremely dubious. (In fact, it seems so crazy to me that it seems more likely than not that I significantly misunderstood the person who I think made these claims.) The first claim is of course vacuously true if, for X, we choose some ~impossible task such as “experience a utility-monster amount of pleasure” or “come up with a blueprint for how to build safe AGI that is convincing to benign actors able to execute it”. But of course more great people don’t help with solving impossible tasks.
Given the size and talent distribution of the EA community my guess is that for most apparent X, the issue either is that (a) X is ~impossible, or (b) there are people in EA who could do X, but the relevant actors cannot identify them, or (c) acquiring the ability to do X is costly (e.g. perhaps you need time to acquire domain-specific expertise), even for maximally talented “great people”, and the relevant actors either are unable to help pay that cost (e.g. by training people themselves, or giving them the resources to allow them to get training elsewhere) or make a mistake by not doing so.
My best guess for the genesis of the “we need more great people” perspective: Suppose I talk a lot to people at an organization that thinks there’s a decent chance we’ll develop transformative AI soon but it will go badly, and that as a consequence tries to grow as fast as possible to pursue various ambitious activities which they think reduces that risk. If these activities are scalable projects with short feedback loops on some intermediate metrics (e.g. running some super-large-scale machine learning experiments), then I expect I would hear a lot of claims like “we really need someone who can do X”. I think it’s just a general property of a certain kind of fast-growing organization that’s doing practical things in the world that everything constantly seems like it’s on fire. But I would also expect that, if I poked a bit at these claims, it would usually turn out that X is something like “contribute to this software project at the pace and quality level of our best engineers, w/o requiring any management time” or “convince some investors to give us much more money, but w/o anyone spending any time transferring relevant knowledge”. If you see that things break because X isn’t done, even though something like X seems doable in principle (perhaps you see others do it), it’s tempting to think that what you need is more “great people” who can do X. After all, people generally are the sort of stuff that does things, and maybe you’ve actually seen some people do X. But it still doesn’t follow that in your situation “great people” are the bottleneck …
Curious if anyone has examples of tasks X for which the original claims seem in fact true. That’s probably the easiest way to convince me that I’m wrong.
Thank you for sharing your reaction!
Would be interested to hear if the authors have though through this.
I haven’t, but it’s possible that my coauthors have. I generally agree that it might be worthwhile to think along the lines you suggested.