Regarding what effect sizes we can observe using “common sense”: this may vary with domain. Historically, people seem to have had better common sense-intuitions about human psychology than about medicine. That made the value of scientific studies greater in the latter domain. (Though I’m less sure of the relative accuracy of contemporary expert intuitions in psychology and medicine.)
Thus the benefits of additional studies in a domain depends both on whether that domain is underpowered, and on how reliable our common sense-intuitions are in that domain.
“Common sense” is obviously infused with insights from past scientific studies. Some findings, like the “System 1-System 2″-distinction, become part of common sense (at least in some social circles). (Obviously this is still more true of experts’ intuitions.) Since this tendency should tend to increase the accuracy of common sense, it further reduces what we should expect to learn from additional data on a specific question.
Great post; thanks for this, Greg.
Regarding what effect sizes we can observe using “common sense”: this may vary with domain. Historically, people seem to have had better common sense-intuitions about human psychology than about medicine. That made the value of scientific studies greater in the latter domain. (Though I’m less sure of the relative accuracy of contemporary expert intuitions in psychology and medicine.)
Thus the benefits of additional studies in a domain depends both on whether that domain is underpowered, and on how reliable our common sense-intuitions are in that domain.
“Common sense” is obviously infused with insights from past scientific studies. Some findings, like the “System 1-System 2″-distinction, become part of common sense (at least in some social circles). (Obviously this is still more true of experts’ intuitions.) Since this tendency should tend to increase the accuracy of common sense, it further reduces what we should expect to learn from additional data on a specific question.