Yes, if you assume your errors are log normal distributed, you expect to see big errors.
Your simulation says that the range of actual threats is tightly bounded between 10e-5 and 10e-7 (IMO much too small a range). In contrast, your error estimates span 8 orders of magnitude (IMO likely too large a range).
I really think your choice of parameters fully explains your results.
Makes sense that range of threats should be wider (arbitrarily wide I guess, but a function of what scenarios you consider and differentiate between). I don’t see why error estimates should be thin though—there are certainly people guessing close to 100% for some risks and various mechanisms that we might not even have considered that we would consider high risk if we knew more about them, which lead to us underestimating the risks of innocuous actions by a huge amount (c.f. the recent upsurge in concern about mirror bacteria)
I agree my claim for tighter error estimates is very weak.
I could say that looking at the estimate you get by aggregating many different folks’ together reduces variance (assuming you believe the estimates have some amount of uncorrelated signal). Individual estimates are noisy, but aggregate estimates are less noisy. This is basically the point discussed in the ‘Why do different groups have the same rankings’ section of OP’s post.
But frankly I’m largely making a vibes claim (ie, model gives silly results → model probably wrong).
the OOM of variation in “ground truth” come from alpha and n, not xmin
alpha, we could talk all day, but the model is not extremely sensitive to it
on the other hand, if you say let’s have more OOMs in the possible values of ground truth, following the power law, that means jacking n up
and when you jack n up you have even more opportunities for errors to be crazy big, and this effect dominates (at least that’s what I read from the OP) and the curse becomes worse
now if we change alpha and n at the same time, idk
my honest opinion is that numbers are just one way to process information, and using them for this is so out of distribution that it’s essentially meaningless (as it is when discussing p(doom) and stuff like that)
Sure, I understand where the values come from. I’m saying the distribution created leads to (IMO) clearly wacky results. The difference between the most vs least spooky X-risks is way more than a 100X difference.
I personally get some value out of numbers & sanity checking them like this, but your mileage may vary.
Here is an example run I did where I tuned down the alpha to 1.5 and tuned the lognormal standard deviation down to 1.5:
Here, the top actual threat really is 7 orders of magnitude more dangerous than the bottom evaluated threat. However, the top apparent threat is overestimated by a factor of 10,000 or so.
If I do a bunch of runs with these settings, the median overestimate is over 100x:
So even if I trust your vibes here (which do not seem to be based on anything), the curse can still hit quite badly. I personally believe that the spread of numbers that people make up is going to be higher than the spread of actual threats: when we look at actual surveys you get a highest-lowest estimate spread of 11 orders of magnitude for some questions.
One thing that might be confusing you is that the power law model assumes that only threats above a certain threshold of actual danger are considered (this is the xmin factor). Obviously nuclear risk is a much greater risk than stubbing your toe, but it’s not going to show up the model.
The difference between the most vs least spooky X-risks is way more than a 100X difference.
I think I would agree with this, if I had to put a number.
What I mean in my comment is, with this model, if you say okay let’s pick a bigger n so that we see bigger differences in OOMs, then you are also introducing more points of failure in the estimation, and that effect dominates.
Do you have an a priori reason to discard this? Besides the conclusion being wacky, which is a good reason to discard a model anyways.
Yes, if you assume your errors are log normal distributed, you expect to see big errors.
Your simulation says that the range of actual threats is tightly bounded between 10e-5 and 10e-7 (IMO much too small a range). In contrast, your error estimates span 8 orders of magnitude (IMO likely too large a range).
I really think your choice of parameters fully explains your results.
Makes sense that range of threats should be wider (arbitrarily wide I guess, but a function of what scenarios you consider and differentiate between). I don’t see why error estimates should be thin though—there are certainly people guessing close to 100% for some risks and various mechanisms that we might not even have considered that we would consider high risk if we knew more about them, which lead to us underestimating the risks of innocuous actions by a huge amount (c.f. the recent upsurge in concern about mirror bacteria)
I agree my claim for tighter error estimates is very weak.
I could say that looking at the estimate you get by aggregating many different folks’ together reduces variance (assuming you believe the estimates have some amount of uncorrelated signal). Individual estimates are noisy, but aggregate estimates are less noisy. This is basically the point discussed in the ‘Why do different groups have the same rankings’ section of OP’s post.
But frankly I’m largely making a vibes claim (ie, model gives silly results → model probably wrong).
the OOM of variation in “ground truth” come from alpha and n, not xmin
alpha, we could talk all day, but the model is not extremely sensitive to it
on the other hand, if you say let’s have more OOMs in the possible values of ground truth, following the power law, that means jacking n up
and when you jack n up you have even more opportunities for errors to be crazy big, and this effect dominates (at least that’s what I read from the OP) and the curse becomes worse
now if we change alpha and n at the same time, idk
my honest opinion is that numbers are just one way to process information, and using them for this is so out of distribution that it’s essentially meaningless (as it is when discussing p(doom) and stuff like that)
Sure, I understand where the values come from. I’m saying the distribution created leads to (IMO) clearly wacky results. The difference between the most vs least spooky X-risks is way more than a 100X difference.
I personally get some value out of numbers & sanity checking them like this, but your mileage may vary.
Here is an example run I did where I tuned down the alpha to 1.5 and tuned the lognormal standard deviation down to 1.5:
Here, the top actual threat really is 7 orders of magnitude more dangerous than the bottom evaluated threat. However, the top apparent threat is overestimated by a factor of 10,000 or so.
If I do a bunch of runs with these settings, the median overestimate is over 100x:
So even if I trust your vibes here (which do not seem to be based on anything), the curse can still hit quite badly. I personally believe that the spread of numbers that people make up is going to be higher than the spread of actual threats: when we look at actual surveys you get a highest-lowest estimate spread of 11 orders of magnitude for some questions.
One thing that might be confusing you is that the power law model assumes that only threats above a certain threshold of actual danger are considered (this is the xmin factor). Obviously nuclear risk is a much greater risk than stubbing your toe, but it’s not going to show up the model.
The difference between the most vs least spooky X-risks is way more than a 100X difference.
I think I would agree with this, if I had to put a number.
What I mean in my comment is, with this model, if you say okay let’s pick a bigger n so that we see bigger differences in OOMs, then you are also introducing more points of failure in the estimation, and that effect dominates.
Do you have an a priori reason to discard this? Besides the conclusion being wacky, which is a good reason to discard a model anyways.
I agree that merely increasing n would not change the OP’s conclusion that errors dominate.
My point is more that they picked too big of parameters for error variance and too small of parameters for risk-size variance.