I will probably have longer comments later, but just on the fixed effects point, I feel it’s important to clarify that they are sometimes used in this kind of situation (when one fears publication bias or small study-type effects). For example, here is a slide deck from a paper presentation with three *highly* qualified co-authors. Slide 8 reads:
To be conservative, we use ‘fixed-effect’ MA or our new unrestricted WLS—Stanley and Doucouliagos (2015)
Not random-effects or the simple average: both are much more biased if there is publication bias (PB).
Fixed-effect (WLS-FE) is also biased with PB, but less so; thus will over-estimate the power of economic estimates.
This is basically also my take away. In the presence of publication bias or these small-study type effects, random effects “are much more biased” while fixed effects are “also biased [...] but less so.” Perhaps there are some disciplinary differences going on here, but what I’m saying is a reasonable position in political science, and Stanley and Doucouliagos are economists, and Ioannidis is in medicine, so using fixed effects in this context is not some weird fringe position.
--
(disclosure: I have a paper under review where Stanley and Doucouliagos are co-authors)
I may respond later after I’ve read more into this, but briefly — thank you! This is interesting and something I’m willing to change my mind about it. Also didn’t know about WAAP, but it sounds like a sensible alternative.
I will probably have longer comments later, but just on the fixed effects point, I feel it’s important to clarify that they are sometimes used in this kind of situation (when one fears publication bias or small study-type effects). For example, here is a slide deck from a paper presentation with three *highly* qualified co-authors. Slide 8 reads:
To be conservative, we use ‘fixed-effect’ MA or our new unrestricted WLS—Stanley and Doucouliagos (2015)
Not random-effects or the simple average: both are much more biased if there is publication bias (PB).
Fixed-effect (WLS-FE) is also biased with PB, but less so; thus will over-estimate the power of economic estimates.
This is basically also my take away. In the presence of publication bias or these small-study type effects, random effects “are much more biased” while fixed effects are “also biased [...] but less so.” Perhaps there are some disciplinary differences going on here, but what I’m saying is a reasonable position in political science, and Stanley and Doucouliagos are economists, and Ioannidis is in medicine, so using fixed effects in this context is not some weird fringe position.
--
(disclosure: I have a paper under review where Stanley and Doucouliagos are co-authors)
I may respond later after I’ve read more into this, but briefly — thank you! This is interesting and something I’m willing to change my mind about it. Also didn’t know about WAAP, but it sounds like a sensible alternative.