Thanks for these very helpful insights! I thought the mosaic charts were particularly creative and visually insightful.
I have one minor statistical nit and one related question.
In cases where ‘only one significant difference was found’ (at a 95% c.i.), it could be worth noting that you have around 20 categories… so on average one spurious significant difference is to be expected! (If the difference is small.)
Also a question about how the significance test was carried out.σ(a−b)=√σ(a)2+σ(b)2≤σ(a)+σ(b) so for calling a difference significant at 95% it matters whether you a) check if the individual 95% confidence intervals overlap or b) check if the diff’d confidence interval noted above contains 0 (the usual approach). Which approach was used here? I ask because to my eye there might be a few more (weakly) significant results than were mentioned in the text.
We actually just performed the same analyses as we did last year, so any references to significance are after applying the Bonferroni adjustment. We just decided to show the confidence intervals rather than just the binary significant/not significant markers this year, but of course different people have different views about which is better.
Thanks for these very helpful insights! I thought the mosaic charts were particularly creative and visually insightful.
I have one minor statistical nit and one related question.
In cases where ‘only one significant difference was found’ (at a 95% c.i.), it could be worth noting that you have around 20 categories… so on average one spurious significant difference is to be expected! (If the difference is small.)
Also a question about how the significance test was carried out.σ(a−b)=√σ(a)2+σ(b)2≤σ(a)+σ(b) so for calling a difference significant at 95% it matters whether you a) check if the individual 95% confidence intervals overlap or b) check if the diff’d confidence interval noted above contains 0 (the usual approach). Which approach was used here? I ask because to my eye there might be a few more (weakly) significant results than were mentioned in the text.
Hi Oliver. Thanks for your question!
We actually just performed the same analyses as we did last year, so any references to significance are after applying the Bonferroni adjustment. We just decided to show the confidence intervals rather than just the binary significant/not significant markers this year, but of course different people have different views about which is better.