I do research at Longview Philanthropy. Previously I was a Research scholar at FHI and assistant to Toby Ord. Philosophy at Cambridge before that.
I also do a podcast about EA called Hear This Idea.
finm
Karma: 2,589
For what it’s worth I think I basically endorse that comment.
I definitely think an investigation that starts with a questioning attitude, and ends up less negative than the author’s initial priors, should count.
That said, some people probably do already just have useful, considered critiques in their heads that they just need to write out. It’d be good to hear them.
Also, presumably (convincing) negative conclusions for key claims are more informationally valuable than confirmatory ones, so it makes sense to explicitly encourage the kind of investigations that have the best chance of yielding those conclusions (because the claims they address look under-scrutinised).
Thanks for writing this — in general I am pro thinking more about what MWI could entail!
But I think it’s worth being clear about what this kind of intervention would achieve. Importantly (as I’m sure you’re aware), no amount of world slicing is going to increase the expected value of the future (roughly all the branches from here), or decrease the overall (subjective) chance of existential catastrophe.
But it could increase the chance of something like “at least [some small fraction]% of’branches’ survive catastrophe”, or at the extreme “at least one ‘branch’ survives catastrophy”. If you have some special reason to care about this, then this could be good.
For instance, suppose you thought whether or not to accelerate AI capabilities research in the US is likely to have a very large impact on the chance of existential catastrophe, but you’re unsure about the sign. To use some ridiculous play numbers: maybe you’re split 50-50 between thinking investing in AI raises p(catastrophe) to 98% and 0 otherwise, or investing in AI lowers p(catastrophe) to 0 and 98% otherwise. If you flip a ‘classical’ coin, the expected chance of catastrophe is 49%, but you can’t be sure we’ll end up in a world where we survive. If you flip a ‘quantum’ coin and split into two ‘branches’ with equal measure, you can be sure that one world will survive (and another will encounter catastrophe with 98% likelihood). So you’ve increased the chance that ‘at least 40% of the future worlds will survive’ from 50% to 100%.[1]
In general you’ve moving from more overall uncertainty about whether things will turn out good or bad, to more certainty that things will turn out in some mixture of good and bad.
Maybe that sounds good, if for instance you think the mere fact that something exists is good in itself (you might have in mind that if someone perfectly duplicated the Mona Lisa, the duplicate would be worth less than the original, and that the analogy carries).[2]
But I also think it is astronomically unlikely that a world splitting exercise like this would make the difference[3] between ‘at least one branch survives’ and ‘no branches survive’. The reason is just that there are so, so many branches, such that —
It just seems very likely that at least some branches survive anyway;
Even if you thought there was a decent chance that no branches survive without doing the world splitting, then you should have such a wide uncertainty over the number of branches you expect to survive that (I claim) your odds on something like [at least one branch will survive if we do split worlds, and no branches will survive if we don’t] should be very very low.[4] And I think this still goes through even if you split the world many times.
- ^
It’s like choosing between [putting $50 on black and 50% on red] at a roulette table, and [putting $100 on red].
- ^
But also note that by splitting worlds you’re also increasing the chance that ‘at least 40% of the future worlds will encounter catastrophe’ from 48% to 99%. And maybe there’s a symmetry, where if you think there’s something intrinsically good about the fact that a good thing occurs at all, then you should think there’s something intrinsically bad about the fact that a bad thing occurs at all, and I count existential catastrophe as bad!
- ^
Note this is not the same as claining it’s highly unlikely that this intervention will increase the chance of surviving in at least one world.
- ^
Because you are making at most a factor-of-two difference by ‘splitting’ the world once.
Really appreciate your taking the time to do this AMA — I badly want to see this cause area succeed, largely thanks to your writing (Michael), advocacy (Tim), and research (Matthew). Thanks for leading the way.
In a recent conversation with Tim, Hamilton Morris worries out loud about how the psychadelic pendulum has recently swung towards hype, with lots of trendy op-eds and new capital:
Well, what happens when that gets a little bit old, and what happens when, I don’t know, someone has a bad experience? Maybe a celebrity has a bad experience and they decide that mushrooms caused their psychosis. And then what?
How do you think about setting expectations here? Are you concerned about this resurgence in interest repeating some of the mistakes of the 60s? And, this time, what can we be doing to establish things for the much longer term?
Most these ideas sound interesting to me. However —
- OpenPhil making a statement to fund high quality work they disagree with
I’m not quite sure what this means? I’m reading it as “funding work which looks set to make good progress on a goal OP don’t believe is especially important, or even net bad”. And that doesn’t seem right to me.
Similar ideas that could be good —
OP/other grantmakers clarifying that they will consider funding you on equal terms even if you’ve publicly criticised OP/that grantmaker
More funding for thoughtful criticisms of effective altruism and longtermism (theory and practice)
I’m especially keen on the latter!
Thanks so much for writing this Ben! I think it’s great that strong longtermism is being properly scrutinised, and I loved your recent podcast episode on this (as well as Vaden’s piece).
I don’t have a view of my own yet; but I do have some questions about a few of your points, and I think I can guess at how a proponent of strong longtermism might respond to others.
For clarity, I’m understanding part of your argument as saying something like the following. First, “[E]xpected value calculations, Bayes theorem, and mathematical models” are tools — often useful, often totally innapropriate or inapplicable. Second, ‘Bayesian epistemology’ (BE) makes inviolable laws out of these tools, running into all kinds of paradoxes and failing to represent how scientific knowledge advances. This makes BE silly at best and downright ‘refuted’ at worst. Third, the case for strong longtermism relies essentially on BE, which is bad news for strong longtermism.
I can imagine that a fan of BE would just object that Bayesianism in particular is just not a tool which can be swapped out for something else when it’s convenient . This feels like an important but tangential argument — this LW post might be relevant. Also, briefly, I’m not 100% convinced by Popper’s argument against Bayesianism which you’re indirectly referencing, and I haven’t read the paper Vaden wrote but it looks interesting. In any case: declaring that BE “has been refuted” seems unfairly rash.
You suggest at a few points that longtermists are just pulling numbers out of nowhere in order to take an expectation over, for instance, the number of people who will live in the long-run future. In other words, I’m reading you as saying that these numbers are totally arbitrary. You also mention that they’re problematically unfalsifiable.
On the first point, it feels more accurate to say that these numbers are highly uncertain rather than totally arbitrary. I can imagine someone saying “I wouldn’t be surprised if my estimate were off by several orders of magnitude”; but not “I have literally no reason to believe that this estimate is any better than a wildly different one”. That’s because it is possible to begin reasoning about these numbers. For instance, I was reminded of Nick Beckstead’s preliminary review of the feasibility of space colonisation. If it turned out that space colonisation was practically impossible, the ceiling would fall down on estimates for the size of humanity’s future. So there’s some information to go on — just very little.
You make the same point in the context of estimating existential risks:
My credence could be that working on AI safety will reduce existential risk by 5% and yours could be , and there’s no way to discriminate between them.
Really? If you’re a rationalist (in the broad Popperian sense and the internet-cult sense), and we share common knowledge of each other’s beliefs, then shouldn’t we be able to argue towards closer agreement? Not if our estimates were totally arbitrary — but clearly they’re not. Again, they’re just especially uncertain.
[I]t abolishes the means by which one can disagree with its conclusion, because it can always simply use bigger numbers.
You can use bigger numbers in the sense that you can type extra zeroes on your keyboard, but you can’t use bigger numbers if you care about making sure your numbers fall reasonably in line with the available facts, right? I could try turning “donating to Fin’s retirement fund” into an EA cause area by just lying about its impact, but there are norms of honesty and criticism (and common sense) which would prevent the plot succeeding. Because I don’t think you’re suggesting that proponents of strong longtermism are being dishonest in this way, I’m confused about what you are suggesting.
Plus, as James Aung mentioned, I don’t think it works to criticise subjective probabilities (and estimates derived from them) as too precise. The response is presumably: “sure, this guess is hugely uncertain. But better to give some number rather than none, and any number I pick is going to seem too precise to you. Crucially, I’m trying to represent something about my own beliefs — not that I know something precise about the actual world.”
On the falsifiability point, estimates about the size of humanity’s future clearly are falsifiable — it’s just going to take a long time to find out. But plenty of sensible scientific claims are like this — e.g. predictions about the future of stars including our Sun. So the criticism can’t be that predictions about the size of humanity’s future are somehow unscientific because not immediately falsifiable.
I think this paragraph is key:
Thus, subjective credences tend to be compared side-by-side with statistics derived from actual data, and treated as if they were equivalent. But prophecies about when AGI will take over the world — even when cloaked in advanced mathematics — are of an entirely different nature than, say, impact evaluations from randomized controlled trials. They should not be treated as equivalent.
My reaction is something like this: even if other interpretations of probability are available, it seems at least harmless to form subjective credences about the effectiveness of, say, global health interventions backed by a bunch of RCTs. Where there’s lots of empirical evidence, there should be little daylight between your subjective credences and the probabilities that fall straight out of the ‘actual data’. In fact, using subjective credences begins to look positively useful when you venture into otherwise comparable but more speculative interventions. That’s because whether you might want to fund that intervention is going to depend on your best guess about its likely effects and what you might learn from them, and that guess should be sensitive to all kinds of information — a job Bayesian methods were built for. However, if you agree that subjective credences are applicable to innocuous ‘short-term’ situations with plenty of ‘data’, then you can imagine gradually pushing the time horizon (or some other source of uncertainty) all the way to questions about the very long-run future. At this extreme, you’ve said that there’s something qualitatively wrong with subjective credences about such murky questions. But I want to say: given that you can join up the two kinds of subject matter by a series of intermediate questions, and there wasn’t originally anything wrong with using credences and no qualitative or step-change, why think that the two ends of the scale end up being “of an entirely different nature”? I think this applies to Vaden’s point that the maths of taking an expectation over the long-run future is somehow literally unworkable, because you can’t have a measure over infinite possibilities (or something). Does that mean we can’t take an expectation over what happens next year? The next decade?
I hope that makes sense! Happy to say more.
My last worry is that you’re painting an unrealistically grim picture of what strong longtermism practically entails. For starters, you say “[l]ongtermism asks us to ignore problems now”, and Hilary and Will say we can “often” ignore short-term effects . Two points here: first, in situations where we can have a large effect on the present / immediate future without risking something comparably bad in the future, it’s presumably still just as good to do that thing. Second, it seems reasonable to expect considerable overlap between solving present problems and making the long-run future go best, for obvious reasons. For example, investing in renewables or clean meat R&D just seem robustly good from short-term and long-term perspectives.
I’m interested in the comparison to totalitarian regimes, and it reminded me of something Isaiah Berlin wrote:
[T]o make mankind just and happy and creative and harmonious forever—what could be too high a price to pay for that? To make such an omelette, there is surely no limit to the number of eggs that should be broken[.]
However, my guess is that there are too few similarities for the comparison to be instructive. I would want to say that the totalitarian regimes of the past failed so horrendously not because they used expected utility theory or Bayesian epistemology correctly but innapropriately, but because they were just wrong — wrong that revolutionary violence and totalitarianism make the world remotely better in the short or long term. Also, note that a vein of longtermist thinking discusses reducing the likelihood of a great power conflict, improving instutional decision-making, and spreading good (viz. liberal) political norms in general — in other words, how to secure an open society for our descendants.
Longtermism asks us to ignore problems now, and focus on what we believe will be the biggest problems many generations from now. Abiding by this logic would result in the stagnation of knowledge creation and progress.
Isn’t it the case that strong longtermism makes knowledge creation and accelerating progress seem more valuable, if anything? And would the world really generate less knowledge, or progress at a slower rate, if the EA community shifted priorities in a longtermist direction?
Finally, a minor point: my impression is that ‘longtermism’ is generally taken to mean something a little less controversial than ‘strong longtermism’. I appreciate you make the distinction early on, but using the ‘longtermism’ shorthand seems borderline misleading when some of your arguments only apply to a specific version.
For what it’s worth, I’m most convinced by the practical problems with strong longtermism. I especially liked your point about longtermism being less permeable to error correction, and generally I’m curious to know more about reasons for thinking that influencing the long-run future is really tractable. Thanks again for starting this conversation along with Vaden!
Copying a comment from Substack:
If offence and defence both get faster, but all the relative speeds stay the same, I don’t see how that in itself favours offence (we get ICBMs, but the same rocketry + guidance etc tech means missile defence gets faster at the same rate). But ideas like this make sense, e.g. if there are any fixed lags in defence (like humans don’t get much faster at responding but need to be involved in defensive moves) then speed favours offence in that respect.That is to say there could be a ‘faster is different’ effect, where in the AI case things might move too chaotically fast — faster than the human-friendly timescales of previous tech — to effectively defend. For instance, your model of cybersecurity might be a kind of cat-and-mouse game, where defenders are always on the back foot looking for exploits, but they patch them with a small (fixed) time lag. The lag might be insignificant historically, until the absolute lag begins to matter. Not sure I buy this though.
A related vague theme is that more powerful tech in some sense ‘turns up the volatility/variance’. And then maybe there’s some ‘risk of ruin’ asymmetry if you could dip below a point that’s irrecoverable, but can’t rise irrecoverably above a point. Going all in on such risky bets can still be good on expected value grounds, while also making it much more likely that you get wiped out, which is the thing at stake.
Also, embarassingly, I realise I don’t have a very good sense of how exactly people operationalise the ‘offence-defence balance’. One way could be something like ‘cost to attacker of doing $1M of damage in equilibrium’, or in terms of relative spending like Garfinkel and Dafoe do (“if investments into cybersecurity and into cyberattacks both double, should we expect successful attacks to become more or less feasible”). Or maybe something about the cost-per-attacker spending to hold on to some resource (or cost-per-defender spending to sieze it).
This is important because I don’t currently know how to say that some technology is more or less defence-dominant than another, other than in a hand-wavery intuitive way. But in hand-wavey terms it sure seems like bioweapons are more offence-dominant than, say, fighter planes. Because it’s already the case that you need to spend a lot of money to prevent most the damage someone could cause with not much money at all.
I see the AI stories — at least the ones I find most compelling — as being kinda openly idiosyncratic and unprecedented. The prior from previous new tech very much points against them, as you show. But the claim is just: yes, but we have stories about why things are different this time ¯\_(ツ)_/¯
Great post.
To make this list easier to navigate, I’ve compiled it into a spreadsheet, and organised as many as the PDFs as I can into one place:
Here’s a link to the Google Sheets version;
Here’s a link to the Notion version;
And here’s a link to a Google Drive folder with as many of the PDFs as I could find.
I’m not totally sure about whether it’s ok to re-upload a bunch of published academic writing like this — would be interested in people’s thoughts. For what it’s worth, I haven’t uploaded anything that was behind a paywall or not publicly available.
Anyway, hope some people find this useful!
The time seems right for more competent+ambitious EA entrepreneurship, and this seems like an excellent list. Thanks for putting it together!
I like the idea of setting up a home for criticisms of EA/longtermism. Although I guess the EA Forum already exists as a natural place for anyone to post criticisms, even anonymously. So I guess the question is — what is the forum lacking? My tentative answer might be prestige / funding. Journals offer the first. The tricky question on the second is: who decides which criticisms get awarded? If it’s just EAs, this would be disingenuous.
Thanks very much for writing this — I’m inclined to agree that results from the happiness literature are often surprising and underrated for finding promising neartermist interventions and thinking about the value of economic growth. I also enjoyed hearing this talk in person!
The “aren’t people’s scales adjusting over time?” story (‘scale norming’) is most compelling to me, and I think I’m less sure that we can rule it out. For instance — if I’m reading you right, you suggest that one reason to be skeptical that people are adjusting their scales over time is that people mostly agree on which adjectives like “good” correspond with which numerical scores of wellbeing. This doesn’t strike me as strong evidence that people are not scale norming, since I wouldn’t be surprised if people adjust the rough meaning of adjectives roughly in line with numbers.
If people thought this task was meaningless, they’d answer at random, and the lines would be flat.
I don’t see a dichotomy between “people use the same scales across time and context for both words and adjectives” and “people view this task as meaningless”.
You also suggest a story about what people are doing when they come up SWB scores, which if true leaves little room for scale norming/adjustment. And since (again, if I’m reading you right) this story seems independently plausible, we have an independently plausible reason to be skeptical that scale norming is occurring. Here’s the story:
the way we intuitively use 0 to 10 scales is by taking 10 to be the highest realistic level (i.e. the happiest a person can realistically be) and 0 as the lowest (i.e. the least happy a person could realistically be) (Plant 2020). We do this, I claim, so that [...] we can use the same scales as other people and over time. If we didn’t do this, it would make it very difficult for our answers to be understood.
I think I don’t find this line of argument super compelling, and not even because I strongly disagree with that excerpt. Rather: the excerpt underdetermines what function you use to project from an extremely wide space onto a bounded scale, and there is no obvious such ‘Schelling’ function (i.e. I don’t even know what it would mean for your function to be linear). And indeed people could change functions over time while keeping those 0 and 10 pegs fixed. Another thing that could be going on is that people might be considering how to make their score informationally valuable, which might involve imagining what kind of function would give a relatively even spread across 0–10 when used population-wide. I don’t think this is primarily what is going on, but to the extent that it is, such a consideration would make a person’s scale more relative to the population they understand themselves to be part of[1], and as such to re-adjust over time.
Two extra things: (i) in general I strongly agree that this question (about how people’s SWB scales adjust across time or contexts) is important and understudied, and (ii) having spoken with you and read your stuff I’ve become relatively less confident in scale-norming as a primary explanation of all this stuff.
I would change my mind more fully that scale norming is not occuring if I saw evidence that experience-sampling type measures of affect also did not change over the course of decades as countries become/became wealthier (and earned more leisure time etc). I’d also change my mind if I saw some experiment where people were asked to rate how their lives were going in relation to some shared reference point(s), such as other people’s lives descibed in a good amount of detail, and where people’s ratings of how their lives were going relative to those reference points also didn’t change as countries became significantly wealthier.
(Caveat to all of above that I’m writing in a hurry!)
- ^
If almost everyone falls between 6–7 on the widest scale I can imagine, maybe the scale I actually use should significantly zoom in on that region.
- ^
Broadly agreed with this, but I’m a bit worried that contests with large prizes can have distortionary effects. That is, they might pull EAs towards using their time in ways which are not altruistically/impartially best. This would happen when an EA switches her marginal time to some contest with a big prize, where she otherwise would have been doing something expected to be more impactful (e.g. because she’s a better fit for it), but which doesn’t stand to win her as much money or acclaim.
For instance, I think the creative writing prize was a really great idea, but my own experience was one of feeling like I really ought to at least ‘buy a raffle ticket’, sinking a few hours into trying to write something halfway decent, then missing the deadling and feeling a bit deflated. I don’t mean this in an arrogant way, but in retrospect I think it’s likely I could have done something better with my time, impact-wise.
One fix could be leaning towards more and narrower contests, which require specialised skills, rather than fewer, highly general contests with large prizes. That way, the prize doesn’t have such a big ‘gravity well’ that it sucks in a lot of ‘might as well try’ folks who were already doing useful stuff.
Another fix, which I like most, is more retroactive prizes/funding. The Forum Prize was a good start, although it’s been retired with a promise of finding better (e.g. more democratic) alternatives. Note that some people have recently been writing about and trialling retroactive funding. There is likely more than a post’s worth to discuss here, but I think one key idea is that there are potential EA projects where (i) only a couple people would really be suited to doing, such that (ii) a competitive prize wouldn’t make sense; (iii) it’s really hard to track all those ideas and promise rewards for them in advance; but (iv) it would be great if someone did them and were rewarded. I’d be interested to see more discussion of what scalable retroactive funding could look like in EA. In general, retroactive funding avoids this distortionary worry — as long as you trust the evaluators to judge what was most impactful, you can’t beat just trying to do actually impactful things.
Anyway, bottom line is to notice that incentives can go wrong a bit more easily in an altruistic context, and you need to consider the kind of work that a well-meaning contest is getting people to replace. In some circumstances, the impact of a contest with a very large prize but questionable impact might be therefore be unclear, even imagining that money is free.
Enormous +1
Thanks, this is a good post. A half-baked thought about a related but (I think) distinct reason for this phenomenon: I wonder if we tend to (re)define the scale of problems such that they are mostly unsolved at present (but also not so vast that we obviously couldn’t make a dent). For instance, it’s not natural to think that the problem of ‘eradicating global undernourishment’ is more than 90% solved, because fewer than 10% of people in the world are undernourished. As long as problems are (re)defined in this way to be smaller in absolute terms, then tractability is going to (appear to) proportionally increase, as a countervailing factor to diminishing returns from extra investment of resources. A nice feature of ITN is that (re)defining the scale of a problem such that it is always mostly unsolved at present doesn’t affect the bottom line of utility per marginal dollar, because (utility / % of problem solved) increases as (% of problem solved / marginal dollar) decreases. To the extent this is a real phenomenon, it could emphasise the importance of not reading too much into direct comparisons between tractability across causes.
Replying in personal capacity:
I hope the contest will consider lower effort but insightful or impactful submissions to account for this?
Yes, very short submissions count. And so should “low effort” posts, in the sense of “I have a criticism I’ve thought through, but I don’t have time to put together a meticulous writeup, so I can either write something short/scrappy, or nothing at all.” I’d much rather see unpolished ideas than nothing at all.
Secondly, I’d expect people with the most valuable critiques to be more outside EA since I would expect to find blindspots in the particular way of thinking, arguing and knowing EA uses. What will the panelists do to ensure they can access pieces using a very different style of argument? Have you considered having non-EA panelists to aid with this?
Thanks, I think this is important.
We (co-posters) are proactively sharing this contest with non-EA circles (e.g.), and others should feel welcome and encouraged to do the same.
Note the incentives for referring posts from outside the Forum. This can and should include writing that was not written with this contest in mind. It could also include writing aimed at some idea associated with EA that doesn’t itself mention “effective altruism”.
It obviously shouldn’t be a requirement that submissions use EA jargon.
I do think writing a post roughly in line with the Forum guidelines (e.g. trying to be clear and transparent in your reasoning) means the post will be more likely to get understood and acted on. As such, I do think it makes sense to encourage this manner of writing where possible, but it’s not a hard requirement.
To this end, one idea might be to speak to someone who is more ‘fluent’ in modes of thinking associated with effective altruism, and to frame the submission as a dialogue or collaboration.
But that shouldn’t be a requirement either. In cases where the style of argument is unfamiliar, but the argument itself seems potentially really good, we’ll make the effort — such as by reaching out to the author for clarifications or a call. I hope there are few really important points that cannot be communicated through just having a conversation!
I’m curious which non-EA judges you would have liked to see! We went with EA judges (i) to credibly show that representatives for big EA stakeholders are invested in this, and (ii) because people with a lot of context on specific parts of EA seem best placed to spot which critiques are most underrated. I’m also not confident that every member of the panel would strongly identify as an “effective altruist”, though I appreciate connection to EA comes in degrees.
Thirdly, criticisms from outside of EA might also contain mistakes about the movement but nonetheless make valid arguments. I hope this can be taken into account and such pieces not just dismissed.
Yes. We’ll try to be charitable in looking for important insights, and and forgiving of innacuracies from missing context where they don’t affect the main argument.
That said, it does seem straightforwardly useful to avoid factual errors that can easily be resolved with public information, because that’s good practice in general.
What plans do you have in place to help prevent and mitigate backlash[?]
My guess is that the best plan is going to be very context specific. If you have concerns in this direction, you can email criticism-contest@effectivealtruism.com, and we will consider steps to help, such as by liaising with the community health team at CEA. I can also imagine cases where you just want to communicate a criticism privately and directly to someone. Let us know, and we can arrange for that to happen also (“we” meaning myself, Lizka, or Joshua).
Agreed. I don’t think this video got anything badly wrong, but do be aware that there are plenty of EA types on this forum and elsewhere who would be happy to read over and comment on scripts.
I notice that I’m getting confused when I try to make the market analogy especially well, but I do think there’s something valuable to it.
Caveat that I skim-read up to “To what extent is EA functioning differently from this right now?”, so may have missed important points, and also I’m writing quickly.
Claims inspired by the analogy which I agree with:
Various kinds of competition between EA-oriented orgs is good: competition for hires, competition for funding, and competition for kinds of reputation
And I think this is true roughly for the same reason that competition between for-profit firms is good: it imposes a pressure on orgs/firms to innovate to get some edge over their competitors, which causes the sector as a whole to innovate
I think it is also good to have some pressures to exist for orgs to fold, or at least to fold specific projects, when they’re not having the impact they hoped for. When a firm folds, that’s bad for its employees in the short run; but having an environment where the least productive firms can go bust can raise the average productivity of a firm
If you don’t allow many projects to fail, that could mean (i) that the ecosystem is insufficiently risk-tolerant; or (ii) the ecosystem is inefficiently sustaining failed projects on life-support, in a way which wouldn’t happen in a free market
Here’s a commendable example of an org wrapping up a program because of disappointing empirical results. Seems good to celebrate stuff like this and make sure the incentives are there for such decisions to be made when best
More concretely: I don’t think we need to always assume that it’s not worth starting an org working on X if an org already exists to work on X (e.g. I think it’s cool that Probably Good exists as well as 80k)
Many things that make standard markets inefficient are also bad for the EA ecosystem. You list “corruption, nepotism, arbitrariness, dishonesty” and those do all sound like things which shouldn’t exist within EA
It would be good if there were more large donors of EA (largely because this would mean more money going to important causes)
It’s often good for EA orgs which provide a service to other EA orgs to charge those orgs directly, rather than rely on grant money themselves to provide the service for free. And perhaps this should be more common
For roughly the same reason that centrally planned economies are worse than free markets at naturally scaling down services which aren’t providing much value, and scaling up the opposite
However, there are aspects of the analogy which still feel a bit confusing to me (after ~10 mins of thinking), such that I’d want to resist claims that in some sense this “market for ways to do the most good” analogy should be a or the central way to conceptualise what EA is about. In particular:
As Ben West points out, the consumers in this analogy are not the beneficiaries. The Hayekian story about what makes markets indispensable involves a story about how they’re indispensably good at aggregating preferences across many buyers and sellers, more effectively than any planner. But these stories don’t go through in the analogous case, because the buyers (donors) are buying on behalf of others
Indeed, this is a major reason to expect that ‘markets’ for charitable interventions are inefficient with respect to actual impact, and thus a major insight behind EA!
Another complication is that in commissioning research rather than on-the-ground interventions, the donors are doing something like buying information to better inform their own preferences. I don’t know how this maps onto the standard market case (maybe it does)
Seems to me that the EA case might be more analogous to a labour market than a product market (since donors are more like employers than people shopping at a farmers market). Much of the analogy goes through with this change but not all (e.g. labour supply curves are often kind of funky)
I’m less clear on why monopsony is bad specifically for reasons inspired by the market analogy. My impression of the major reason why monopsonies are bad is a bit different from yours —
Imagine there’s one employer facing an upward-sloping labour supply curve and paying the same wage to everyone. Then the profit maximising wage for a monopsonist can be lower than the competitive equilibrium, leading to a deadweight loss (e.g. more unemployment and lower wages). And it’s the deadweight loss that is the bad thing
But EA employers aren’t maximising profit for themselves — they’re mostly nonprofits!
You could make the analogy work better by treating profits for the donor as impact. I’m confused on exactly how you’d model this, and would be interested if someone who knew economics had thoughts. But it just seems intuitive to me that the analogous deadweight loss reason to avoid monopsony doesn’t straightforwardly carry over (minimally, the impartial donor could just choose to pay the competitive wage)
CompetitiveMarkets can involve some behaviour which is not directly productive, but does help companies get a leg-up on one another (such that many or all companies involved would prefer if that behaviour weren’t an option for anyone). One example is advertising (advertising is useful for other reasons, I mostly have in mind “Pepsi vs Coke” style advertising). I don’t like the idea of more of this kind of competitive advertising-type behaviour in EAEdit: this is an example of imperfect competition, thanks to yefreitor for pointing out
Companies in competition won’t share valuable proprietary information with one another for obvious reasons. But I think it’s often really good that EA orgs share research insights and other kinds of advice, even when not sharing that information could have given the org that generated it a leg-up on other orgs
Indeed, I think this mutual supportiveness is a good feature of the EA community on the whole, and could account for some of its successes
More generally, if the claim is that this market analogy should be a or the central way to conceptualise what EA is about, then I just feel like the analogy misses most of what’s important. It captures how transactions work between donors and orgs, and how orgs compete for funding. But it seems to me that it matters at least as much to understand what people are doing inside those orgs — what they are working on, how they are reasoning about them, why they are working on them, how the donors choose what to fund, and so on. Makes me think of this Herbert Simon quote.
Hopefully some of that makes sense. I think it’s likely I got some economics-y points wrong and look forward to being corrected on them.
Sorry if I missed this in other comments, but one question I have is if there are ways for small donors to support projects or individuals in the short term who have been thrown into uncertainty by the FTX collapse (such as people who were planning on the assumption that they would be receiving a regrant). I suppose it would be possible to donate to Nonlinear’s emergency funding pot, or just to something like the EAIF / LTFF / SFF.
But I’m imagining that a major bottleneck on supporting these affected projects is just having capacity to evaluate them all. So I wonder about some kind of initiative where affected projects can choose to put some details on a public register/spreadsheet (e.g. a description of the project, how they’ve been affected, what amount of funding they’re looking for, contact details). Then small donors can look through the register and evaluate projects which fit their areas of interest / experience, and reach out to them individually. It could be a living spreadsheet where entries are updated if their plans change or they receive funding. And maybe there could be some way for donors to coordinate around funding particular projects that they individually each donor couldn’t afford to fund, and which wouldn’t run without some threshold amount. E.g. donors themselves could flag that they’d consider pitching in on some project if others were also interested.
A more sophisticated version of this could involve small donors putting donations into some kind of escrow managed by a trusted party that donates on people’s behalf, and that trusted party shares donors on information about projects affected by FTX. That would help maintain some privacy / anonymity if some projects would prefer that, but at administrative cost. I’d guess this idea is too much work given the time-sensitivity of everything.
An 80-20 version is just to set up a form similar to Nonlinear’s, but which feeds into a database which everyone can see, for projects happy to publicly share that they are seeking shortish-term funding to stay afloat / make good on their plans. Then small donors can reach out at their discretion. If this worked, then it might be a way to help ‘funge’ not just the money but also the time of grant evaluators at grantmaking orgs (and similar) which is spent evaluating small projects. It could also be a chance to support projects that you feel especially strongly about (and suspect that major grant evaluators won’t share your level of interest).
I’m not sure how to feel about this idea overall. In particular, I feel misgivings about the public and uncoordinated nature of the whole thing, and also about the fact that typically it’s a better division of labour for small donors to follow the recommendations of experienced grant investigators/evaluators. Decisions about who to fund, especially in times like these, are often very difficult and sensitive, and I worry about weird dynamics if they’re made public.
Curious about people’s thoughts, and I’d be happy to make this a shortform or post in the effective giving sub-forum if that seems useful.
Sounds good. At the more granular and practical end, this sounds like red-teaming, which is often just good practice.
I think it is worth appreciating the number and depth of insights that FHI can claim significant credit for. In no particular order:
The concept of existential risk, and arguments for treating x-risk reduction as a global priority (see: The Precipice)
Arguments for x-risk from AI, and other philosophical considerations around superintelligent AI (see: Superintelligence)
Arguments for the scope and importance of humanity’s long-term future (since called longtermism)
Information hazards
Observer selection effects and ‘anthropic shadow’
Bounding natural extinction rates with statistical methods
The vulnerable world hypothesis
Moral trade
Crucial considerations
The unilteralist’s curse
Dissolving the Fermi paradox
The reversal test in applied ethics
‘Comprehensive AI services’ as an alternative to unipolar outcomes
The concept of existential hope
Note especially how much of the literal terminology was coined on (one imagines) a whiteboard in FHI. “Existential risk” isn’t a neologism, but I understand it was Nick who first suggested it be used in a principled way to point to the “loss of potential” thing. “Existential hope”, “vulnerable world”, “unilateralist’s curse”, “information hazard”, all (as far as I know) tracing back to an FHI publication.
It’s also worth remarking on the areas of study that FHI effectively incubated, and which are now full-blown fields of research:
The ‘Governance of AI Program’ was launched in 2017, to study questions around policy and advanced AI, beyond the narrowly technical questions. That project was spun out of FHI to become the Centre for the Governance of AI. As far as I understand, it was the first serious research effort on what’s now called ”AI governance”.
From roughly 2019 onwards, the working group on biological risks seems to have been fairly instrumental in making the case for biological risk reduction as a global priority, specifically because of engineered pandemics.
If research on digital minds (and their implications) grows to become something resembling a ‘field’, then the small team and working groups on digital minds can make a claim to precedence, as well as early and more recent published work.
FHI was staggeringly influential; more than many realise.
Edit: I wrote some longer reflections on FHI here.