Studying behaviour and interactions of boundedly rational agents, AI alignment and complex systems.
Research fellow at Future of Humanity Institute, Oxford. Other projects: European Summer Program on Rationality. Human-aligned AI Summer School. Epistea Lab.
Seems a case of “How x-risk projects are different from startups”
(i.e. most people who are likely to update downwards on Yudkowsky on the basis of this post, seem to me to be generically too trusting, and I am confident I can write a more compelling post about any other central figure in Effective Altruism that would likely cause you to update downwards even more)
My impression is the post is somewhat unfortunate attempt to “patch” the situation in which many generically too trusting people updated a lot on AGI Ruin: A List of Lethalities and Death with Dignity and subsequent deference/update cascades.
In my view the deeper problem here is instead of disagreements about model internals, many of these people do some sort of “averaging conclusions” move, based on signals like seniority, karma, vibes, etc.
Many of these signals are currently wildly off from truth-tracking, so you get attempts to push the conclusion-updates directly.
I. It might be worth reflecting upon how large part of this seem tied to something like “climbing the EA social ladder”.
E.g. just from the first part, emphasis mine
Coming to Berkeley and, e.g., running into someone impressive at an office space already establishes a certain level of trust since they know you aren’t some random person (you’ve come through all the filters from being a random EA to being at the office space).
If you’re in Berkeley for a while you can also build up more signals that you are worth people’s time. E.g., be involved in EA projects, hang around cool EAs.
Replace “EA” by some other environment with prestige gradients, and you have something like a highly generic social climbing guide. Seek cool kids, hang around them, go to exclusive parties, get good at signalling.
II. This isn’t to say this is bad . Climbing the ladder to some extent could be instrumentally useful, or even necessary, for an ability to do some interesting things, sometimes.
III. But note the hidden costs. Climbing the social ladder can trade of against building things. Learning all the Berkeley vibes can trade of against, eg., learning the math actually useful for understanding agency.
I don’t think this has any clear bottom line—I do agree for many people caring about EA topics it’s useful to come to the Bay from time to time. Compared to the original post I would probably mainly suggest to also consult virtue ethics and think about what sort of person you are changing yourself to, and if you, for example, most want to become “a highly cool and well networked EA” or e.g. “do things which need to be done”, which are different goals.
Suggested variation, which I’d expect to lead to better results: use raw “completion probabilities” for different answers.
E.g. with prompt “Will Russia invade Ukrainian territory in 2022?” extract completion likelihoods of the next few tokes “Yes” and “No”. Normalize
Also the direction of ALERT is generally more on “doing”. Doing seems often very different from forecasting, often needs different people—part of the relevant skills is plausibly even anticorrelated.
Crisis response is a broader topic. I would probably suggest creating additional tag for Crises response (most of our recent sequence would fit there)
I don’t have a strong preference. There a some aspects in which longerism can be better framing, at least sometimes.
I. In a “longetermist” framework, x-risk reduction is the most important thing to work on for many orders of magnitude of uncertainty about the probability of x-risk in the next e.g. 30 years. (due to the weight of the long term future). Even if AI related x-risk is only 10ˆ-3 in next 30 years, it is still an extremely important problem or the most important one. In a “short-termist” view with, say, a discount rate of 5%, it is not nearly so clear.
The short-termist urgency of x-risk (“you and everyone you know will die”) depends on the x-risk probability being actually high, like of order 1 percent, or tens of percents . Arguments why this probability is actually so high are usually brittle pieces of mathematical philosophy (eg many specific individual claims by Eliezer Yudkowsky) or brittle use of proxies with lot of variables obviously missing from the reasoning (eg the report by Ajeya Cotra). Actual disagreements about probabilities are often in fact grounded in black-box intuitions about esoteric mathematical concepts. It is relatively easy to come with brittle pieces of philosophy arguing in the opposite direction: why this number is low. In fact my actual, action guiding estimate is not based on an argument conveyable by a few paragraphs, but more on something like “feeling you get after working on this over years”. What I can offer other is something like “an argument from testimony”, and I don’t think it’s that great.
II. Longermism is a positive word, pointing toward the fact that future could be large and nice. X-risk is the opposite.
Similar: AI safety vs AI alignment. My guess is the “AI safety” framing is by default more controversial and gets more of a pushback (eg “safety department” is usually not the most loved part of an organisation, with connotations like “safety people want to prevent us from doing what we want”)
Title EA Dropouts seems a bit confusing, because it can be naturally interpreted as people who dropped out of EA
I had little influence over the 1st wave, credit goes elsewhere.
What happened in subsequent waves is complicated. One sentence version is Czechia changed minister of health 4 times, only some of them were reasonably oriented, and how much they were interested in external advice differed a lot in time.
Note that the “death tolls per capita in the world” stats are misleading, due to differences in reporting. Czechia had average or even slightly lower than average mortality compared to “Eastern Europe” reference class, but much better reporting. For more reliable data, see https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(21)02796-3/fulltext
Both “EA translation service” and “EA spaces everywhere” seem like ideas which can take good, but also many bad or outright harmful forms.
A few years ago, I tried to describe how to to establish a robustly good “local effective altruism” in a new country or culture (other than Anglosphere).
The super brief summary is
1. it’s not about translating a book, but about “transmission of a tradition of knowledge”
2. what is needed is a highly competent group of people
3. who can, apart from other things, figure out what the generative question of EA means in the given context (which may be quite different from Oxford or Bay)
Point #2 is the bottleneck. Without it, efforts to have “country/language X EA” will be often unsuccessful or worse. Doing Good Better was translated to e.g. japanese. It is doing a bit worse on Amazon.co.jp than the english version on Amazon.co.uk, but not by many orders. Yet you aren’t seeing a lot of content and projects from EA Japan.
So: One good version of “translation service” seems to be basically giving functional national EA groups money to get as good translations as needed.
One bad version is a “centralized project” trying to translate centrally selected content to centrally selected languages by hiring professional contractors to do it.
Similarly EA spaces:
Hiring offices in 100 cities is easy, and you can do that centrally. Running a space with a good culture and gatekeeping is harder, and bottlenecked on “#2″.
Good version of the project seems to be basically giving functional city EA groups money to get good coworking spaces, which is probably doable via CEA group support.
Many bad versions are somewhere close to “centralized megaproject to run EA spaces everywhere”.
In other words: it isn’t easy to get around the bottleneck ‘you need people on the ground’.
Footnote: there are many variations on this theme. The basic pattern roughly is:
“notice that successful groups are doing activity X” (e.g. they have a coworking space, or materials in the local language, or they organize events,...).
the next step, in which this goes wrong, is, “let’s start a project P(X) that will try to make activity X happen everywhere”.
Also: The chances of bad or explicitly harmful outcomes increase roughly in proportion to the combination of cultural, network and geographic distance from the “centre” from which such activity should be directed. Eg project which will try to run spaces in Stanford from Berkeley seem fine
Mass-reach posts came later, but sooner than the US mainstream updates
https://www.lesswrong.com/posts/h4vWsBBjASgiQ2pn6/credibility-of-the-cdc-on-sars-cov-2
https://slatestarcodex.com/2020/03/23/face-masks-much-more-than-you-wanted-to-know/
The practical tradeoff was between what, where and when to publish. The first version of the preprint which is on medrxive contains those estimates. Some version with them could probably be published in a much worse journal than Science, and would have much less impact.
We could have published them separately, but a paper is a lot of work, and it’s not clear to me whether, for example, to sacrifice some of the”What we tried”and get this done would have been a good call.
It is possible to escape from the game in specific cases—in the case of covid, for example, the advisory body we created in the Czech Republic was able to take into account analyses based on “internal quality”, especially if it was clear peer review game will take months. If such bodies existed in more/more countries, it would be possible.
Similarly, it could be done with the help of an ECDC or WHO type institution.
In general, it’s an “inadequate equilibrium” type of problem, I have some thoughts on typical solutions to them, but not in easily shareable written form, at the moment.
NPI & IFR: thanks, it’s now explained in the text.
Re: Rigour
I think much of the problem is due not to our methods being “unrigourous” in any objective sense, but to interdisciplinarity. For example, in the survey case, we used mostly standard methods from a field called “discrete choice modelling” (btw, some EAs should learn it—it’s a pretty significant body of knowledge on “how to determine people’s utility functions”).
Unfortunately, it’s not something commonly found in the field of, for example, “mathematical modeling of infectious diseases”. It makes it more difficult for journals to review such a paper, because ideally they would need several different reviewers for different parts of the paper. This is unlikely to happen in practice, so usually the reviewers tend to either evaluate everything according to the conventions of their field, or to be critical and dismissive of things they don’t understand.
Similar thing is going on with use of “forecasting”-based methods. There is published scientific literature on their use, their track record is good, but before the pandemic there was almost no “published literature” on the subject of their use in combination with epidemic modelling (there is now!).
The second part of the problem is that we were ultimately more interested in “what is actually true” than what “looks rigorous”. A paper that contains few pages of equations, lots of complex modeling, and many simulations can look “rigorous” (in the sense of the stylized dialogue). If at the same time, for example, it contains completely and obviously wrong assumptions about the IFR of covid it will still pass many tests of “rigorousness” because it only shows that “under assumptions that do not hold in our world we reach conclusions that are irrelevant to our world” (the implication is true). At the same time, it can have disastrous consequences, if used by policymakers, who assume something like “research tracks reality”.
Ex post, we can demonstrate that some of our methods (relying on forecasters) were much closer to reality (e.g. based on serological studies) than a lot of published stuff.
Ex ante, it was clear this will be the case to many people who understand both academic research and forecasting.
Re: Funding
For the record, EpiFor is a project that has ended, and is not seeking any funding. Also, as noted in the post, we were actually able to get some funding offered: just not in a form which the university was able to accept, etc.
It’s not like there is one funder evaluating whether to fund IHME, or EpidemicForecasting. In my view the problems pointed to here are almost completely unrelated, and I don’t want them to get conflated in some way
Hi, as the next post in the sequencd is about ‘failures’ I think it would be more useful after that is published.
If the main problem you want to solve is “scaling up grantmaking”, there are probably many other ways how to do it other than “impact markets”.
(Roughly, you can amplify any “expert panel of judges” evaluations with judgemental forecasting.)