I haven’t read the meta-analysis, but I’d tentatively bet that much like biological properties these jobs actually follow log-normal distributions and they just couldn’t tell (and weren’t trying to tell) the difference.
I kind of agree with this (and this is why I deliberately said that “they report a Gaussian distribution” rather than e.g. “performance is normally distributed”). In particular, yes, they just assumed a normal distribution and then ran with this in all cases in which it didn’t lead to obvious problems/bad fits no matter the parameters. They did not compare Gaussian with other models.
I still think it’s accurate and useful to say that they were using (and didn’t reject) a normal distribution as model for low- and medium-complexity jobs as this does tell you something about how the data looks like. (Since there is a lot of possible data where no normal distribution is a reasonable fit.)
I also agree that probably a log-normal model is “closer to the truth” than a normal one. But on the other hand I think it’s pretty clear that actually neither a normal nor a log-normal model is fully correct. Indeed, what would it mean that “jobs actually follow a certain type of distribution”? If we’re just talking about fitting a distribution to data, we will never get a perfect fit, and all we can do is providing goodness-of-fit statistics for different models (which usually won’t conclusively identify any single one). This kind of brute/naive empiricism just won’t and can’t get us to “how things actually work”. On the other hand, if we try to build a model of the causal generating mechanism of job performance it seems clear that the ‘truth’ will be much more complex and messy—we will only have finitely many contributing things (and a log-normal distribution is something we’d get at best “in the limit”), the contributing factors won’t all be independent etc. etc. Indeed, “probability distribution” to me basically seems like the wrong type to talk about when we’re in the business of understanding “how things actually work”—what we want then is really a richer and more complex model (in the sense that we could have several different models that would yield the same approximate data distribution but that would paint a fairly different picture of “how things actually work”; basically I’m saying that things like ‘quantum mechanics’ or ‘the Solow growth model’ or whatever have much more structure and are not a single probability distribution).
Briefly on this, I think my issue becomes clearer if you look at the full section.
If we agree that log-normal is more likely than normal, and log-normal distributions are heavy-tailed, then saying ‘By contrast, [performance in these jobs] is thin-tailed’ is just incorrect? Assuming you meant the mathematical senses of heavy-tailed and thin-tailed here, which I guess I’m not sure if you did.
This uncertainty and resulting inability to assess whether this section is true or false obviously loops back to why I would prefer not to use the term ‘heavy-tailed’ at all, which I will address in more detail in my reply to your other comment.
Ex-post performance appears ‘heavy-tailed’ in many relevant domains, but with very large differences in how heavy-tailed: the top 1% account for between 4% to over 80% of the total. For instance, we find ‘heavy-tailed’ distributions (e.g. log-normal, power law) of scientific citations, startup valuations, income, and media sales. By contrast, a large meta-analysis reports ‘thin-tailed’ (Gaussian) distributions for ex-post performance in less complex jobs such as cook or mail carrier
I think the main takeaway here is that you find that section confusing, and that’s not something one can “argue away”, and does point to room for improvement in my writing. :)
With that being said, note that we in fact don’t say anywhere that anything ‘is thin-tailed’. We just say that some paper ‘reports’ a thin-tailed distribution, which seems uncontroversially true. (OTOH I can totally see that the “by contrast” is confusing on some readings. And I also agree that it basically doesn’t matter what we say literally—if people read what we say as claiming that something is thin-tailed, then that’s a problem.)
FWIW, from my perspective the key observations (which I apparently failed to convey in a clear way at least for you) here are:
The top 1% share of ex-post “performance” [though see elsewhere that maybe that’s not the ideal term] data reported in the literature varies a lot, at least between 3% and 80%. So usually you’ll want to know roughly where on the spectrum you are for the job/task/situation relevant to you rather than just whether or not some binary property holds.
The range of top 1% shares is almost as large for data for which the sources used a mathematically ‘heavy-tailed’ type of distribution as model. In particular, there are some cases where we some source reports a mathematically ‘heavy-tailed’ distribution but where the top 1% share is barely larger than for other data based on a mathematically ‘thin-tailed’ distribution.
(As discussed elsewhere, it’s of course mathematically possible to have a mathematically ‘thin-tailed’ distribution with a larger top 1% share than a mathematically ‘heavy-tailed’ distribution. But the above observation is about what we in fact find in the literature rather than about what’s mathematically possible. I think the key point here is not so much that we haven’t found a ‘thin-tailed’ distribution with larger top 1% share than some ‘heavy-tailed’ distribution. but that the mathematical ‘heavy-tailed’ property doesn’t cleanly distinguish data/distributions by their top 1% share even in practice.)
So don’t look at whether the type of distribution used is ‘thin-tailed’ or ‘heavy-tailed’ in the mathematical sense, ask how heavy-tailed in the everyday sense (as operationalized by top 1% share or whatever you care about) your data/distribution is.
So basically what I tried to do is mentioning that we find both mathematically thin-tailed and mathematically heavy-tailed distributions reported in the literature in order to point out that this arguably isn’t the key thing to pay attention to. (But yeah I can totally see that this is not coming across in the summary as currently worded.)
As I tried to explain in my previous comment, I think the question whether performance in some domain is actually ‘thin-tailed’ or ‘heavy-tailed’ in the mathematical sense is closer to ill-posed or meaningless than true or false. Hence why I set aside the issue of whether a normal distribution or similar-looking log-normal distribution is the better model.
I kind of agree with this (and this is why I deliberately said that “they report a Gaussian distribution” rather than e.g. “performance is normally distributed”). In particular, yes, they just assumed a normal distribution and then ran with this in all cases in which it didn’t lead to obvious problems/bad fits no matter the parameters. They did not compare Gaussian with other models.
I still think it’s accurate and useful to say that they were using (and didn’t reject) a normal distribution as model for low- and medium-complexity jobs as this does tell you something about how the data looks like. (Since there is a lot of possible data where no normal distribution is a reasonable fit.)
I also agree that probably a log-normal model is “closer to the truth” than a normal one. But on the other hand I think it’s pretty clear that actually neither a normal nor a log-normal model is fully correct. Indeed, what would it mean that “jobs actually follow a certain type of distribution”? If we’re just talking about fitting a distribution to data, we will never get a perfect fit, and all we can do is providing goodness-of-fit statistics for different models (which usually won’t conclusively identify any single one). This kind of brute/naive empiricism just won’t and can’t get us to “how things actually work”. On the other hand, if we try to build a model of the causal generating mechanism of job performance it seems clear that the ‘truth’ will be much more complex and messy—we will only have finitely many contributing things (and a log-normal distribution is something we’d get at best “in the limit”), the contributing factors won’t all be independent etc. etc. Indeed, “probability distribution” to me basically seems like the wrong type to talk about when we’re in the business of understanding “how things actually work”—what we want then is really a richer and more complex model (in the sense that we could have several different models that would yield the same approximate data distribution but that would paint a fairly different picture of “how things actually work”; basically I’m saying that things like ‘quantum mechanics’ or ‘the Solow growth model’ or whatever have much more structure and are not a single probability distribution).
Briefly on this, I think my issue becomes clearer if you look at the full section.
If we agree that log-normal is more likely than normal, and log-normal distributions are heavy-tailed, then saying ‘By contrast, [performance in these jobs] is thin-tailed’ is just incorrect? Assuming you meant the mathematical senses of heavy-tailed and thin-tailed here, which I guess I’m not sure if you did.
This uncertainty and resulting inability to assess whether this section is true or false obviously loops back to why I would prefer not to use the term ‘heavy-tailed’ at all, which I will address in more detail in my reply to your other comment.
I think the main takeaway here is that you find that section confusing, and that’s not something one can “argue away”, and does point to room for improvement in my writing. :)
With that being said, note that we in fact don’t say anywhere that anything ‘is thin-tailed’. We just say that some paper ‘reports’ a thin-tailed distribution, which seems uncontroversially true. (OTOH I can totally see that the “by contrast” is confusing on some readings. And I also agree that it basically doesn’t matter what we say literally—if people read what we say as claiming that something is thin-tailed, then that’s a problem.)
FWIW, from my perspective the key observations (which I apparently failed to convey in a clear way at least for you) here are:
The top 1% share of ex-post “performance” [though see elsewhere that maybe that’s not the ideal term] data reported in the literature varies a lot, at least between 3% and 80%. So usually you’ll want to know roughly where on the spectrum you are for the job/task/situation relevant to you rather than just whether or not some binary property holds.
The range of top 1% shares is almost as large for data for which the sources used a mathematically ‘heavy-tailed’ type of distribution as model. In particular, there are some cases where we some source reports a mathematically ‘heavy-tailed’ distribution but where the top 1% share is barely larger than for other data based on a mathematically ‘thin-tailed’ distribution.
(As discussed elsewhere, it’s of course mathematically possible to have a mathematically ‘thin-tailed’ distribution with a larger top 1% share than a mathematically ‘heavy-tailed’ distribution. But the above observation is about what we in fact find in the literature rather than about what’s mathematically possible. I think the key point here is not so much that we haven’t found a ‘thin-tailed’ distribution with larger top 1% share than some ‘heavy-tailed’ distribution. but that the mathematical ‘heavy-tailed’ property doesn’t cleanly distinguish data/distributions by their top 1% share even in practice.)
So don’t look at whether the type of distribution used is ‘thin-tailed’ or ‘heavy-tailed’ in the mathematical sense, ask how heavy-tailed in the everyday sense (as operationalized by top 1% share or whatever you care about) your data/distribution is.
So basically what I tried to do is mentioning that we find both mathematically thin-tailed and mathematically heavy-tailed distributions reported in the literature in order to point out that this arguably isn’t the key thing to pay attention to. (But yeah I can totally see that this is not coming across in the summary as currently worded.)
As I tried to explain in my previous comment, I think the question whether performance in some domain is actually ‘thin-tailed’ or ‘heavy-tailed’ in the mathematical sense is closer to ill-posed or meaningless than true or false. Hence why I set aside the issue of whether a normal distribution or similar-looking log-normal distribution is the better model.