Epistemic Institutions Some fields of science are uncontroversially more reliable than others. Physics is more reliable than theoretical sociology, for example. But other fields aren’t that easy to score. Should you believe the claims of a random sleep research paper? Or a paper from personality psychology? Efficacy is just as important, as a scientific field with low efficacy is probably not worth engaging with at all.
A scientific field can be evaluated by giving it a score along one or more dimensions, where a lower score indicates the field might not be worth taking seriously. Right now, people score fields of science informally. For instance, it is common to be skeptical of results from social psychology due to the replication crisis. Claims of nutrition scientists are often ignored due to their over-reliance on observational studies. If the field hasn’t been well investigated, the consumer of the scientific literature is on his own.
Scoring can be based on measurable factors such as
community norms in the field,
degree of p-hacking and publication bias,
reliance on observational studies over experimentation,
amount of “skin in the game”,
open data and open code,
how prestige-driven it is.
Scoring of the overall quality of a field serves multiple purposes.
A low score can dissuade people from taking the field seriously, potentially saving lots of time and money.
The scores can be used informally when forming an opinion. More formally, they can be used as input into other methods, e.g. to correct p-values when reading a paper.
If successful, the scores can incite reform in the poorly performing subfields.
Can be used as input to other EA organizations such as 80,000 hours.
Scoring scientific fields
Epistemic Institutions
Some fields of science are uncontroversially more reliable than others. Physics is more reliable than theoretical sociology, for example. But other fields aren’t that easy to score. Should you believe the claims of a random sleep research paper? Or a paper from personality psychology? Efficacy is just as important, as a scientific field with low efficacy is probably not worth engaging with at all.
A scientific field can be evaluated by giving it a score along one or more dimensions, where a lower score indicates the field might not be worth taking seriously. Right now, people score fields of science informally. For instance, it is common to be skeptical of results from social psychology due to the replication crisis. Claims of nutrition scientists are often ignored due to their over-reliance on observational studies. If the field hasn’t been well investigated, the consumer of the scientific literature is on his own.
Scoring can be based on measurable factors such as
community norms in the field,
degree of p-hacking and publication bias,
reliance on observational studies over experimentation,
amount of “skin in the game”,
open data and open code,
how prestige-driven it is.
Scoring of the overall quality of a field serves multiple purposes.
A low score can dissuade people from taking the field seriously, potentially saving lots of time and money.
The scores can be used informally when forming an opinion. More formally, they can be used as input into other methods, e.g. to correct p-values when reading a paper.
If successful, the scores can incite reform in the poorly performing subfields.
Can be used as input to other EA organizations such as 80,000 hours.