I’ve always wondered about the “first N Google results” strategy. Even in the absence of a file-drawer effect, isn’t this more likely to turn up papers making positive claims (on the assumption that e.g. rejections of the null are more likely to be cited than inconclusive results)?
I’m not sure how google scholar judges relevance (e.g. I can imagine eye-catching negative results also being boosted up the rankings) but I agree it is a source of distortion—I’d definitely offer it as ‘better than nothing’ rather than good. (Perhaps one tweak would be sample by a manageable date range rather than relevance, although one could worry about time trends).
A better option (although it has some learning curve and onerousness) is query a relevant repository, export all the results, and take a random sample from these.
I’ve always wondered about the “first N Google results” strategy. Even in the absence of a file-drawer effect, isn’t this more likely to turn up papers making positive claims (on the assumption that e.g. rejections of the null are more likely to be cited than inconclusive results)?
I’m not sure how google scholar judges relevance (e.g. I can imagine eye-catching negative results also being boosted up the rankings) but I agree it is a source of distortion—I’d definitely offer it as ‘better than nothing’ rather than good. (Perhaps one tweak would be sample by a manageable date range rather than relevance, although one could worry about time trends).
A better option (although it has some learning curve and onerousness) is query a relevant repository, export all the results, and take a random sample from these.