The illusion of science in comparative cognition

Link post

This ar­ti­cle would be worth con­sid­er­ing by EAs us­ing an­i­mal cog­ni­tion stud­ies to as­sess their sen­tience; it sug­gests a lot of the liter­a­ture is bi­ased to­wards overly pos­i­tive con­clu­sions about an­i­mal cog­ni­tion.

A promi­nent vein of com­par­a­tive cog­ni­tion re­search asks which cog­ni­tive abil­ities may be as­cribed to differ­ent species. Here, we ar­gue that the cur­rent struc­ture of com­par­a­tive cog­ni­tion makes it near im­pos­si­ble to eval­u­ate the ac­cu­racy of many of the claims pro­duced by the field’s em­piri­cal re­search. … This ar­gu­ment is based on the fol­low­ing ob­ser­va­tions:
1) Phenomenon-based com­par­a­tive cog­ni­tion uses con­fir­ma­tory re­search meth­ods that are di­rec­tion­ally biased
2) In com­bi­na­tion with a pub­li­ca­tion bias and a likely high rate of false dis­cov­er­ies, this bias sug­gests our liter­a­ture con­tains many false pos­i­tive findings
3) This di­rec­tional bias per­sists even with strong method­olog­i­cal crit­i­cism, and when re­searchers ex­plic­itly con­sider al­ter­na­tive ex­pla­na­tions for the phe­nom­ena studied
4) No for­mal method ex­ists for gen­er­at­ing and as­sess­ing the­ory-dis­con­firm­ing ev­i­dence that could counter the bi­ased pos­i­tive evidence
5) Am­bi­guity in defi­ni­tions al­low us as re­searchers to flex­ibly ad­just our sub­stan­tive claims de­pend­ing on whether we are re­fut­ing crit­i­cism or sel­l­ing the results
6) The small size of com­par­a­tive cog­ni­tion as a re­search field per­pet­u­ates and re­in­forces points 1 to 5.

My own re­search train­ing is in neu­roethol­ogy, where Krogh’s Prin­ci­ple is com­monly used as a guide:

for such a large num­ber of prob­lems there will be some an­i­mal of choice, or a few such an­i­mals, on which it can be most con­ve­niently stud­ied.

While these days the prin­ci­ple is helpful for en­courag­ing peo­ple to work on an­i­mals other than model or­ganisms, it prob­a­bly re­in­forces a lot of the bi­ases above (par­tic­u­larly #1). For in­stance, desert ants have been stud­ied as a model of vi­sual nav­i­ga­tion for many years, but were origi­nally cho­sen for this be­cause they lived in an en­vi­ron­ment that was hard to nav­i­gate and so ini­tial stud­ies mostly con­firmed re­searchers opinions of their ex­cep­tional nav­i­ga­tion abil­ities. If cog­ni­tion is usu­ally tested in ex­cep­tional cases then it prob­a­bly also re­duces the gen­er­al­ity of find­ings be­tween species—most ant species don’t live in the Sa­haran Desert and will have had less se­lec­tion pres­sure to evolve ex­cep­tional nav­i­ga­tion abil­ities.