Generally I think we should expect tails to be a little thicker than log-normal
What’s your reasoning for this?
I don’t off-hand see how to follow this through quantitatively.
In general it depends on the structure of the distribution, or more precisely the copula between the ground truth and estimate. If you assume a Gaussian copula then you can easily reduce to the bivariate-normal case that Gregory describes here, but model uncertainty in the estimate probably fattens the copula tails a bit.
Generally I think we should expect tails to be a little thicker than log-normal
What’s your reasoning for this?
That there are various mechanisms (of which I only feel like I understand a few) in complex systems which produce power-law type tails. These can enter as factors, and the convergence back to log-normal we’d expect from the central limit theorem is slowish in the tails.
On the other hand we’ll see additive effects too, which could pull the tails in more tightly than log-normal. I maintain a reasonable amount of uncertainty over what kind of tail we’ll eventually think is most appropriate, but while I have that uncertainty I don’t want to say that the tails have to be thin.
Of course this is all qualitative reasoning which really only affects the behaviour quite far down the tail. I think for practical purposes log-normal is often a decent assumption (leaving you with the not-insignificant problem of how to pick parameters).
That there are various mechanisms (of which I only feel like I understand a few) in complex systems which produce power-law type tails. These can enter as factors, and the convergence back to log-normal we’d expect from the central limit theorem is slowish in the tails.
It seems like this probably depends a lot on what type of intervention you’re studying. I guess I would expect x-risks to have power-law-ish distributions, but I can’t think of very many power-law factors that would influence e.g. scaling up a proven global health intervention.
I agree that the distribution will depend on the kind of intervention. When you take into account indirect effects you may get some power-law type behaviour even in interventions where it looks unlikely, though—for instance coalescing broader societal support around an intervention so that it gets implemented far more than your direct funding provides for.
Our distribution of beliefs about the cost-effectiveness of scaling up something which is “proven” is likely to have particularly thin tails compared to dealing with “unproven” things, as by proof we tend to mean high-quality evidence that substantially tightens the possibilities. I’m not sure whether it changes the eventual tail to a qualitatively different kind of behaviour, or if they’re just quantitatively narrower distributions, though.
Carl has argued convincingly that the [edit: normal and] log-normal priors are too thin-tailed here:
I think it’s worth pointing out some of the strange implications of a normal prior for charity cost-effectiveness.
For instance, it appears that one can save lives hundreds of times more cheaply through vaccinations in the developing world than through typical charity expenditures aimed at saving lives in rich countries, according to experiments, government statistics, etc.
But a normal distribution (assigns) a probability of one in tens of thousands that a sample will be more than 4 standard deviations above the median, and one in hundreds of billions that a charity will be more than 7 standard deviations from the median. The odds get tremendously worse as one goes on. If your prior was that charity cost-effectiveness levels were normally distributed, then no conceivable evidence could convince you that a charity could be 100x as good as the 90th percentile charity. The probability of systematic error or hoax would always be ludicrously larger than the chance of such an effective charity. One could not believe, even in hindsight, that paying for Norman Borlaug’s team to work on the Green Revolution, or administering smallpox vaccines (with all the knowledge of hindsight) actually did much more good than typical. The gains from resources like GiveWell would be small compared to acting like an index fund and distributing charitable dollars widely.
Such denial seems unreasonable to me, and I think to Holden. However, if one does believe that there have in fact been multiple interventions that turned out 100x as effective as the 90th percentile charity, then one should reject a normal prior. When a model predicts that the chance of something happening is less than 10^-100, and that thing goes on to happen repeatedly in the 20th century, the model is broken, and one should try to understand how it could be so wrong.
Another problem with the normal prior (and, to a lesser but still problematic extent, a log-normal prior) is that it would imply overconfident conclusions about the physical world.
For instance, consider threats of human extinction. Using measures like “lives saved” or “happy life-years produced,” counting future generations the gain of averting a human extinction scales with the expected future population of humanity. There are pretty well understood extinction risks with well understood interventions, where substantial progress has been made: with a trickle of a few million dollars per year (for a couple decades) in funding 90% of dinosaur-killer size asteroids were tracked and checked for future impacts on Earth. So, if future populations are large then using measures like happy life-years there will be at least some ultra-effective interventions.
If humanity can set up a sustainable civilization and harness a good chunk of the energy of the Sun, or colonize other stars, then really enormous prosperous populations could be created: see Nick Bostrom’s (paper) on astronomical waste for figures.
From this we can get something of a reductio ad absurdum for the normal prior on charity effectiveness. If we believed a normal prior then we could reason as follows:
If humanity has a reasonable chance of surviving to build a lasting advanced civilization, then some charity interventions are immensely cost-effective, e.g. the historically successful efforts in asteroid tracking.
By the normal (or log-normal) prior on charity cost-effectiveness, no charity can be immensely cost-effective (with overwhelming probability).
Therefore, 3. Humanity is doomed to premature extinction, stagnation, or an otherwise cramped future.
I find this “Charity Doomsday Argument” pretty implausible. Why should intuitions about charity effectiveness let us predict near-certain doom for humanity from our armchairs? Long-term survival of civilization on Earth and/or space colonization are plausible scenarios, not to be ruled out in this a priori fashion.
To really flesh out the strangely strong conclusion of these priors, suppose that we lived to see spacecraft intensively colonize the galaxy. There would be a detailed history leading up to this outcome, technical blueprints and experiments supporting the existence of the relevant technologies, radio communication and travelers from other star systems, etc. This would be a lot of evidence by normal standards, but the normal (or log-normal) priors would never let us believe our own eyes: someone who really held a prior like that would conclude they had gone insane or that some conspiracy was faking the evidence.
Yet if I lived through an era of space colonization, I think I could be convinced that it was real. I think Holden could be convinced that it was real. So a prior which says that space colonization is essentially impossible does not accurately characterize our beliefs.
What’s your reasoning for this?
In general it depends on the structure of the distribution, or more precisely the copula between the ground truth and estimate. If you assume a Gaussian copula then you can easily reduce to the bivariate-normal case that Gregory describes here, but model uncertainty in the estimate probably fattens the copula tails a bit.
That there are various mechanisms (of which I only feel like I understand a few) in complex systems which produce power-law type tails. These can enter as factors, and the convergence back to log-normal we’d expect from the central limit theorem is slowish in the tails.
On the other hand we’ll see additive effects too, which could pull the tails in more tightly than log-normal. I maintain a reasonable amount of uncertainty over what kind of tail we’ll eventually think is most appropriate, but while I have that uncertainty I don’t want to say that the tails have to be thin.
Of course this is all qualitative reasoning which really only affects the behaviour quite far down the tail. I think for practical purposes log-normal is often a decent assumption (leaving you with the not-insignificant problem of how to pick parameters).
It seems like this probably depends a lot on what type of intervention you’re studying. I guess I would expect x-risks to have power-law-ish distributions, but I can’t think of very many power-law factors that would influence e.g. scaling up a proven global health intervention.
I agree that the distribution will depend on the kind of intervention. When you take into account indirect effects you may get some power-law type behaviour even in interventions where it looks unlikely, though—for instance coalescing broader societal support around an intervention so that it gets implemented far more than your direct funding provides for.
Our distribution of beliefs about the cost-effectiveness of scaling up something which is “proven” is likely to have particularly thin tails compared to dealing with “unproven” things, as by proof we tend to mean high-quality evidence that substantially tightens the possibilities. I’m not sure whether it changes the eventual tail to a qualitatively different kind of behaviour, or if they’re just quantitatively narrower distributions, though.
Carl has argued convincingly that the [edit: normal and] log-normal priors are too thin-tailed here:
Ryan, unless I’m dramatically misreading this post, it is about a normal, not log-normal distribution. Their behaviors are very different.
Carl starts off objecting to a normal prior then goes on to explain why normal and log-normal priors both look too thin-tailed.