This article would be worth considering by EAs using animal cognition studies to assess their sentience; it suggests a lot of the literature is biased towards overly positive conclusions about animal cognition.
A prominent vein of comparative cognition research asks which cognitive abilities may be ascribed to different species. Here, we argue that the current structure of comparative cognition makes it near impossible to evaluate the accuracy of many of the claims produced by the field’s empirical research. … This argument is based on the following observations:
1) Phenomenon-based comparative cognition uses confirmatory research methods that are directionally biased
2) In combination with a publication bias and a likely high rate of false discoveries, this bias suggests our literature contains many false positive findings
3) This directional bias persists even with strong methodological criticism, and when researchers explicitly consider alternative explanations for the phenomena studied
4) No formal method exists for generating and assessing theory-disconfirming evidence that could counter the biased positive evidence
5) Ambiguity in definitions allow us as researchers to flexibly adjust our substantive claims depending on whether we are refuting criticism or selling the results
6) The small size of comparative cognition as a research field perpetuates and reinforces points 1 to 5.
My own research training is in neuroethology, where Krogh’s Principle is commonly used as a guide:
for such a large number of problems there will be some animal of choice, or a few such animals, on which it can be most conveniently studied.
While these days the principle is helpful for encouraging people to work on animals other than model organisms, it probably reinforces a lot of the biases above (particularly #1). For instance, desert ants have been studied as a model of visual navigation for many years, but were originally chosen for this because they lived in an environment that was hard to navigate and so initial studies mostly confirmed researchers opinions of their exceptional navigation abilities. If cognition is usually tested in exceptional cases then it probably also reduces the generality of findings between species—most ant species don’t live in the Saharan Desert and will have had less selection pressure to evolve exceptional navigation abilities.
Further work from the authors of the original article:
Claims and statistical inference in animal physical cognition research.
I’m not sure how important Krogh’s Principle is in animal cognition research of the kind we’re interested in; my impression is that the animals that are studied are primarily animals that are well-studied, like fruit flies, bees, mice, rats, cats, dogs, farmed animals and the stereotypically smart ones (corvids, parrots, elephants, cetaceans, primates), and the animals EAs are interested in fall into these groups. When I want to know about chicken cognition, I just look for studies on chickens. It’s worth mentioning that Rethink Priorities stuck to relatively narrow taxons in their report.
I do agree that this research is likely to be biased overall to produce more positive than could be reproduced or generalized. However, I also think that the priors are already very skeptical (e.g. Morgan’s canon over Occam’s razor and despite common descent) so scientists are also likely to attribute fewer and less complex mental states to animals than I think best explains the evidence, and it’s pretty clear that we’ve systematically underestimated their capacities, so it’s likely the current state of research underestimates them overall, too.
Or, rather, researchers aren’t using Bayesian reasoning in the first place, so they aren’t really using priors at all in interpreting evidence; I think Morgan’s canon is more like a p-value threshold than a prior.
Of course, we can just use our own priors in interpreting the evidence, and in doing so, we should take into account biases towards positive results in research.
I think I remember reading a book or paper by Noam Chomsky which said that chimpanzees don’t have minds because they beg for food usually directly facing a human, but if that doesn’t work they will use the same strategy and beg behind the human. He says this proves chimpanzees dont know what a human is—they view humans as food dispensers. My interpretation was if you are going to beg, maybe display subserviance—ask from the back, nor face to face. (Chomsky if i recall also said something like this in 2 lectures on linguistics i attended (along with many other things like ‘language did not evolve for communication’—his view was language first evolved to talk to yourself (he has a ‘technical term’ for this called ‘I(nternal) language’, which was then later found to have a slight evolutionary use for communication (‘E-language’) --eg Amazon books found it valuable. ).
Chomsky’s view is not what is found in my limited study of social biology, sociobiology, ethology (R Hinde had a reasonable book on this if i recall), etc. (Chomsky has written a few reasonable things, but also many which seem ‘off the wall’).
While i tend to think animals of all sorts have various forms of cognition, as noted with the ‘ant’ example, these often will be quite different. (Chomsky has also said the only animal language even close to human language is ‘bee language’—none of the other animals like birds or non-human primates have language in his view. I guess this is how you become the most famous public intellectual—Chomksy said it, i beleive it, and that is all there is to it. Chomsky related language to recursion—which as many have noted, he basically did not define until the 2000′s. Animals use recursion all the time in my experience. Squirrels know exactly where their acorns are and can point to them. )
Many if not most studies of animal cognition (and human) are flawed or imperfect—not really well thought out, but there is a bias to publish something. Sometimes they seem to be on the right track, and sometimes not.