I do research at Longview Philanthropy. Previously I was a Research scholar at FHI and assistant to Toby Ord. Philosophy at Cambridge before that.
I also do a podcast about EA called Hear This Idea.
finm
Karma: 2,563
The singer-songwriter José González has mentioned being inspired by The Precipice and apparently other EA-related ideas. Take the charmingly scout mindset ‘Head On’:
Speak up
Stand down
Pick your battles
Look around
Reflect
Update
Pause your intuitions and deal with it
Head on
[Copied from an email exchange with Vasco, slightly embellished]
I think the probability of a flat universe is ~0 because the distribution describing our knowledge about the curvature of the universe is continuous, whereas a flat universe corresponds to a discrete curvature of 0.
Sure, if you put infinitesimal weight on a flat universe in your prior (true if your distribution is continuous over a measure of spatial curvature and you think it’s infinite only if spatial curvature = 0), then no observation of (local) curvature is going to be enough. On your framing, I think the question is just why the distribution needs to be continuous? Consider: “the falloff of light intensity / gravity etc is very close to being proportional to , but presumably the exponent isn’t exactly 2 since our distribution over for is continuous”.
all the evidence for infinity is coming from having some weight on infinity in our prior.
‘All’ in the sense that you need nonzero non-infinitesimal weight on infinity in your prior, but not in the sense that your prior is the only thing influencing your credence in infinity. Presumably observations of local flatness do actually upweight hypotheses about the universe being infinite, or at least keep them open if you are open to the possibility in the first place. And I could imagine other things counting as more indirect evidence, such as how well or poorly our best physical theories fit with infinity.
[Added] I think this speaks to something interesting about a picture of theoretical science suggested by a subjective Bayesian attitude to belief-forming in general, on which we start with some prior distribution(s) over some big (continuous?) hypothesis space(s), and observations tell us how to update our priors. But you might think that’s a weird way to figure out which theories to believe, because e.g. (i) the hypothesis space is indefinitely large such that you should have infinitesimal or very small credence in any given theory; (ii) the hypothesis space is unknown in some important way, in which case you can’t assign credences at all, or (iii) theorists value various kinds of simplicity or elegance which are hard to cash out in Bayesian terms in a non-arbitrary way. I don’t know where I come down on this but this is a case where I’m unusually sympathetic to such critiques (which I associate with Popper/Deutsch[1]).
[Continuing email] I do agree that “the universe is infinite in extent” (made precise) is different from “for any size, we can’t rule out the universe being at least that big”, and that the first claim is of a different kind. For instance, your distribution over the size of the universe could have an infinite mean while implying certainty that the universe has some finite size (e.g. if that distribution over the size of the universe is where ).
That does put us in a weird spot though, where all the action seems to be in your choice of prior.
I don’t know how relevant it is that the axiom of infinity is independent of ZFC, unless you think that all true mathematical claims are made true by actual physical things in the world (JS Mill believed something like this I think). Then you might have thought you have independent reason to believe (i) the axioms, and if so believing that (ii) you’d be forced to believe in an actual physical infinity. But that has the same suspect “synthetic a priori” character as ontological arguments for God’s existence, and is moot in any case because (ii) is false!
For what it’s worth, as a complete outsider I feel a surprised by how little serious discussion there is in e.g. astrophysics / philosophy of physics etc around whether the universe is infinite in some way. It seems like such a big deal; indeed an infinitely big deal!
- ^
Though I don’t think these views would have much constructive to say about how much credence to put on the universe being infinite, since they’d probably reject the suggestion that you can or should be trying to figure out what credence to put on it. Paging @ben_chugg since I think he could say if I’m misrepresenting the view.
- ^
Very cool! Feel free to share your paper if you’re able, I’d be curious to see.
I don’t know how to interpret the image, but the this makes sense:
With a [small] attack surface (grid) for each actor, the budget multiplication should have no effect on loss rates, because all vulnerabilities are found and it’s just a matter of who found them first, which is not affected by budget multiplication. However, with a [large attack surface], the multiplication of budgets strictly benefits the attacker, because the defenders will ~never check the same squares that the attacker checks.
Copying a comment from Substack:
If offence and defence both get faster, but all the relative speeds stay the same, I don’t see how that in itself favours offence (we get ICBMs, but the same rocketry + guidance etc tech means missile defence gets faster at the same rate). But ideas like this make sense, e.g. if there are any fixed lags in defence (like humans don’t get much faster at responding but need to be involved in defensive moves) then speed favours offence in that respect.That is to say there could be a ‘faster is different’ effect, where in the AI case things might move too chaotically fast — faster than the human-friendly timescales of previous tech — to effectively defend. For instance, your model of cybersecurity might be a kind of cat-and-mouse game, where defenders are always on the back foot looking for exploits, but they patch them with a small (fixed) time lag. The lag might be insignificant historically, until the absolute lag begins to matter. Not sure I buy this though.
A related vague theme is that more powerful tech in some sense ‘turns up the volatility/variance’. And then maybe there’s some ‘risk of ruin’ asymmetry if you could dip below a point that’s irrecoverable, but can’t rise irrecoverably above a point. Going all in on such risky bets can still be good on expected value grounds, while also making it much more likely that you get wiped out, which is the thing at stake.
Also, embarassingly, I realise I don’t have a very good sense of how exactly people operationalise the ‘offence-defence balance’. One way could be something like ‘cost to attacker of doing $1M of damage in equilibrium’, or in terms of relative spending like Garfinkel and Dafoe do (“if investments into cybersecurity and into cyberattacks both double, should we expect successful attacks to become more or less feasible”). Or maybe something about the cost-per-attacker spending to hold on to some resource (or cost-per-defender spending to sieze it).
This is important because I don’t currently know how to say that some technology is more or less defence-dominant than another, other than in a hand-wavery intuitive way. But in hand-wavey terms it sure seems like bioweapons are more offence-dominant than, say, fighter planes. Because it’s already the case that you need to spend a lot of money to prevent most the damage someone could cause with not much money at all.
I see the AI stories — at least the ones I find most compelling — as being kinda openly idiosyncratic and unprecedented. The prior from previous new tech very much points against them, as you show. But the claim is just: yes, but we have stories about why things are different this time ¯\_(ツ)_/¯
Great post.
What a great resource, thanks for putting it together!
Opinionated lists like this feel significantly more useful than comprehensive but unordered lists of relevant resources, because: (i) for most literatures, you’re likely to get most of all the good insights from reading a small standout minority of everything written; and (ii) it’s typically often not obvious to an outsider which resources are best in this respect. I hadn’t heard of many of the books you rate highly.
Incidentally: consider reformatting the papers to not be headers? It makes the navigation bar feel cluttered to me.
Congrats Toby, excited to see what you get up to in the new role! And thanks for all your work on Amplify.
(I’d guess the different titles mostly just reflect the difference in seniority? cf. “program officer” vs “program associate”)
Thanks for these details! I updated the relevant paragraph to include them.
Chuck Feeney (1931–2023)
I got a lot of value out of Guesstimate, and this (plus Squiggle itself) looks like a big step up. So thanks, and kudos!
(Also — both this new site and the Squiggle lang seem generally useful far beyond EA / x-risk contexts; e.g. for consultancies / policy planning / finance. I’d be interested to see if it catches on more widely.)
New DeepMind report on institutions for global AI governance
I also found no negative effects on my productivity.
This makes it sound to me like you think most the value comes from the health/fitness benefits of generally being less sedentary during a working day; and less to no value comes from potential benefits to focus or productivity (except insofar as they’re downstream of being healthier). Is that a fair summary?
Thanks for the response.
You point out that both in markets and in EA (at least its idealised version), people are deliberately seeking out the most value for themselves or others, contrasted to much of the charity world, where people don’t tend to think of what they’re doing as seeking out the most value for themselves or others. That sounds roughly right, but I don’t think it follows that EA is best imagined or idealised as a kind of market. Though I’m not suggesting you claim that it does follow.
It also seems worth pointing out that in some sense there are literal markets for ‘normal charity’ interventions — like the different options I can choose from to sponsor a cute animal as a Christmas gift for someone. And these are markets where people are in some sense choosing the best or most ‘valuable’ deal (insofar as I might compare charities, and those charities will do various things to vie for my donation). I think this shows that the “is this a market” test does not necessarily delineate your idealised version of EA from ‘normal charity’ alone. Again, not suggesting you make that exact claim, but I think it’s worth getting clear on.
Instead, as you suggest, it’s what the market is in that matters — in the case of EA we want a market for “things that do the most good”. You could construe this as a difference in the preferences of the buyers, where the preferences of EA donors are typically more explicitly consequentialist / welfarist / cosmopolitan than donors to other kinds of charity. So I guess your claim is not that being a market in charitable interventions would make EA distinctive, but rather that it is or should be a particular kind of market where the buyers want to do the most good. Is that a fair summary of your view?
If so, I think I’m emphasising that descriptively the ”...doing the most good” part may be more distinctive of the EA project than “EA is a market for...” Normatively I take you to want EA to be more like a competitive market, and there I think there are certainly features of competitive markets that seem good to move towards, but I’m also hesitant to make the market analogy, like, the central guide to how EA should change.
Couple other points:
I still don’t think the Hayekian motivation for markets carries over to the EA case, at least not as you’ve make the pitch. My (possibly poorly remembered) understanding was that markets are a useful way to aggregate information about individuals preferences and affordances via the price discovery mechanism. It’s true that the EA system as a whole (hopefully) discovers things about what are the best ways to help people, but not through the mechanism of price discovery! In fact, I’d say the way it uncovers information is just as similar to how a planner could uncover information — by commissioning research etc. Maybe I’m missing something here.[1]
I agree that the fact people are aiming at value for others doesn’t invalidate the analogy. Indeed, people buy things for other people in normal markets very often.
On your point about intervention, I guess I’m confused about what it means to ‘intervene’ in the market for doing the most good, and who is the ‘we’ doing the intervening (who presumably are neither funder nor org). Like, what is the analogy to imposing taxes or subsidies, and what is the entity imposing them?
You characterise my view as being indifferent on whether EA should be more like a market, and in favour of advocating for particular causes. I’d say my view is more that I’m just kinda confused about exactly what the market analogy prescribes, and as such I’m wary of using the market metaphor as a guide. I’d probably endorse some of the things you say it recommends.
However I strongly agree that if EA just became a vehicle for advocating a fixed set of causes from now on, then it would lose a very major part of what makes it distinctive. Part of what makes EA distinctive are all the features that identifies those causes — a culture of open discussion and curiosity, norms around good epistemic practice, a relatively meritocratic job market, and a willingness on the part of orgs, funders, and individuals to radically reassess their priorities on the grounds of new evidence. Those things have much in common with free markets, but I don’t think we need the market analogy to see their merit.
- ^
Another disanalogy might be that price discovery works through an adversarial relationship where (speaking loosely) buyers care about output for money and sellers care about money for input. But in the EA case, buyers care about altruistic value per dollar, but sellers (e.g. orgs) don’t care about profit — they often also care about altruistic value per dollar. So what is the analogous price discovery mechanism?
- ^
I notice that I’m getting confused when I try to make the market analogy especially well, but I do think there’s something valuable to it.
Caveat that I skim-read up to “To what extent is EA functioning differently from this right now?”, so may have missed important points, and also I’m writing quickly.
Claims inspired by the analogy which I agree with:
Various kinds of competition between EA-oriented orgs is good: competition for hires, competition for funding, and competition for kinds of reputation
And I think this is true roughly for the same reason that competition between for-profit firms is good: it imposes a pressure on orgs/firms to innovate to get some edge over their competitors, which causes the sector as a whole to innovate
I think it is also good to have some pressures to exist for orgs to fold, or at least to fold specific projects, when they’re not having the impact they hoped for. When a firm folds, that’s bad for its employees in the short run; but having an environment where the least productive firms can go bust can raise the average productivity of a firm
If you don’t allow many projects to fail, that could mean (i) that the ecosystem is insufficiently risk-tolerant; or (ii) the ecosystem is inefficiently sustaining failed projects on life-support, in a way which wouldn’t happen in a free market
Here’s a commendable example of an org wrapping up a program because of disappointing empirical results. Seems good to celebrate stuff like this and make sure the incentives are there for such decisions to be made when best
More concretely: I don’t think we need to always assume that it’s not worth starting an org working on X if an org already exists to work on X (e.g. I think it’s cool that Probably Good exists as well as 80k)
Many things that make standard markets inefficient are also bad for the EA ecosystem. You list “corruption, nepotism, arbitrariness, dishonesty” and those do all sound like things which shouldn’t exist within EA
It would be good if there were more large donors of EA (largely because this would mean more money going to important causes)
It’s often good for EA orgs which provide a service to other EA orgs to charge those orgs directly, rather than rely on grant money themselves to provide the service for free. And perhaps this should be more common
For roughly the same reason that centrally planned economies are worse than free markets at naturally scaling down services which aren’t providing much value, and scaling up the opposite
However, there are aspects of the analogy which still feel a bit confusing to me (after ~10 mins of thinking), such that I’d want to resist claims that in some sense this “market for ways to do the most good” analogy should be a or the central way to conceptualise what EA is about. In particular:
As Ben West points out, the consumers in this analogy are not the beneficiaries. The Hayekian story about what makes markets indispensable involves a story about how they’re indispensably good at aggregating preferences across many buyers and sellers, more effectively than any planner. But these stories don’t go through in the analogous case, because the buyers (donors) are buying on behalf of others
Indeed, this is a major reason to expect that ‘markets’ for charitable interventions are inefficient with respect to actual impact, and thus a major insight behind EA!
Another complication is that in commissioning research rather than on-the-ground interventions, the donors are doing something like buying information to better inform their own preferences. I don’t know how this maps onto the standard market case (maybe it does)
Seems to me that the EA case might be more analogous to a labour market than a product market (since donors are more like employers than people shopping at a farmers market). Much of the analogy goes through with this change but not all (e.g. labour supply curves are often kind of funky)
I’m less clear on why monopsony is bad specifically for reasons inspired by the market analogy. My impression of the major reason why monopsonies are bad is a bit different from yours —
Imagine there’s one employer facing an upward-sloping labour supply curve and paying the same wage to everyone. Then the profit maximising wage for a monopsonist can be lower than the competitive equilibrium, leading to a deadweight loss (e.g. more unemployment and lower wages). And it’s the deadweight loss that is the bad thing
But EA employers aren’t maximising profit for themselves — they’re mostly nonprofits!
You could make the analogy work better by treating profits for the donor as impact. I’m confused on exactly how you’d model this, and would be interested if someone who knew economics had thoughts. But it just seems intuitive to me that the analogous deadweight loss reason to avoid monopsony doesn’t straightforwardly carry over (minimally, the impartial donor could just choose to pay the competitive wage)
CompetitiveMarkets can involve some behaviour which is not directly productive, but does help companies get a leg-up on one another (such that many or all companies involved would prefer if that behaviour weren’t an option for anyone). One example is advertising (advertising is useful for other reasons, I mostly have in mind “Pepsi vs Coke” style advertising). I don’t like the idea of more of this kind of competitive advertising-type behaviour in EAEdit: this is an example of imperfect competition, thanks to yefreitor for pointing out
Companies in competition won’t share valuable proprietary information with one another for obvious reasons. But I think it’s often really good that EA orgs share research insights and other kinds of advice, even when not sharing that information could have given the org that generated it a leg-up on other orgs
Indeed, I think this mutual supportiveness is a good feature of the EA community on the whole, and could account for some of its successes
More generally, if the claim is that this market analogy should be a or the central way to conceptualise what EA is about, then I just feel like the analogy misses most of what’s important. It captures how transactions work between donors and orgs, and how orgs compete for funding. But it seems to me that it matters at least as much to understand what people are doing inside those orgs — what they are working on, how they are reasoning about them, why they are working on them, how the donors choose what to fund, and so on. Makes me think of this Herbert Simon quote.
Hopefully some of that makes sense. I think it’s likely I got some economics-y points wrong and look forward to being corrected on them.
First clean water, now clean air
Thanks for that link! Since writing this post I have become aware of a bunch of exciting BOTEC-adjacent projects, especially from speaking with Adam Binks from Sage / Quantified Intuitions.
I’m curious about how far you and Tom have come in working on this.
I’m not actively working with Tom on this project, but I could put you in touch.
Some more projects I’d like to see
I think this is a good and important question. I also agree that humanity’s predicament in 500 years is wildly unpredictable.
But there are some considerations that can guide our guess:
Almost everyone wants to improve their own lives; few people want to make their own lives worse for the sake of it
Some people want to improve the lives of others for the sake of it; few people want to harm others for the sake of it
Technological progress tends to enable people to get more of what they want; in this case the things that improve their lives
If humans are still around in 500 years, we should expect them to be more technologically advanced — since it seems easier to learn new capabilities than to entirely forget old ones
If you begin totally unsure whether the future is good or bad in expectation, then considerations like these might break the symmetry (while remaining entirely open to the possibility that the future is bad).
This post might also be useful; it recomplicates things by giving some considerations on the other side.
I think it is worth appreciating the number and depth of insights that FHI can claim significant credit for. In no particular order:
The concept of existential risk, and arguments for treating x-risk reduction as a global priority (see: The Precipice)
Arguments for x-risk from AI, and other philosophical considerations around superintelligent AI (see: Superintelligence)
Arguments for the scope and importance of humanity’s long-term future (since called longtermism)
Information hazards
Observer selection effects and ‘anthropic shadow’
Bounding natural extinction rates with statistical methods
The vulnerable world hypothesis
Moral trade
Crucial considerations
The unilteralist’s curse
Dissolving the Fermi paradox
The reversal test in applied ethics
‘Comprehensive AI services’ as an alternative to unipolar outcomes
The concept of existential hope
Note especially how much of the literal terminology was coined on (one imagines) a whiteboard in FHI. “Existential risk” isn’t a neologism, but I understand it was Nick who first suggested it be used in a principled way to point to the “loss of potential” thing. “Existential hope”, “vulnerable world”, “unilateralist’s curse”, “information hazard”, all (as far as I know) tracing back to an FHI publication.
It’s also worth remarking on the areas of study that FHI effectively incubated, and which are now full-blown fields of research:
The ‘Governance of AI Program’ was launched in 2017, to study questions around policy and advanced AI, beyond the narrowly technical questions. That project was spun out of FHI to become the Centre for the Governance of AI. As far as I understand, it was the first serious research effort on what’s now called ”AI governance”.
From roughly 2019 onwards, the working group on biological risks seems to have been fairly instrumental in making the case for biological risk reduction as a global priority, specifically because of engineered pandemics.
If research on digital minds (and their implications) grows to become something resembling a ‘field’, then the small team and working groups on digital minds can make a claim to precedence, as well as early and more recent published work.
FHI was staggeringly influential; more than many realise.
Edit: I wrote some longer reflections on FHI here.