I think it’s quite misleading to present p-values and claim that results are or aren’t ‘statistically significant’, without also presenting the disclaimer that this is very far from a random sample, and therefore these statistical results should be interpreted with significant skepticism.
This is covered in detail in the methodology section. We try not to talk about statistical significance much, we try to belabor that these are EAs “in our sample” and not necessarily EAs overall, and we try to meticulously benchmark how representative our sample is to the best of our abilities.
I agree some skepticism is warranted, but not sure if the skepticism should be so significant as to be “quite misleading”… I think you’d have to back up your claim on that. Could be a good conversation to take to the methodology section.
I haven’t looked there yet, so I’m flagging that my comment was not considering the full context.
(I think that the end links didn’t come up on mobile for me, but it could also have been an oversight on my part that there was supporting documentation, specifically labelled methodology.)
I think it’s quite misleading to present p-values and claim that results are or aren’t ‘statistically significant’, without also presenting the disclaimer that this is very far from a random sample, and therefore these statistical results should be interpreted with significant skepticism.
This is covered in detail in the methodology section. We try not to talk about statistical significance much, we try to belabor that these are EAs “in our sample” and not necessarily EAs overall, and we try to meticulously benchmark how representative our sample is to the best of our abilities.
I agree some skepticism is warranted, but not sure if the skepticism should be so significant as to be “quite misleading”… I think you’d have to back up your claim on that. Could be a good conversation to take to the methodology section.
I haven’t looked there yet, so I’m flagging that my comment was not considering the full context.
(I think that the end links didn’t come up on mobile for me, but it could also have been an oversight on my part that there was supporting documentation, specifically labelled methodology.)