Disadvantages of the measures

This ar­ti­cle warns, among oth­ers, of the risk of fal­ling in love with mea­sure­ment sys­tems that seem very pre­cise, for ex­am­ple, be­cause they offer re­sults with dec­i­mals, but have other draw­backs.

Some­times, not hav­ing a mea­sure can be bet­ter than hav­ing a bad mea­sure. And some­times, not mea­sur­ing can be bet­ter than mea­sur­ing and caus­ing a per­verse effect, just the op­po­site of the de­sired effect.

The ad­van­tages of measurements

The ad­van­tages of hav­ing mea­sure­ments are ev­i­dent, so this ar­ti­cle fo­cuses on the in­con­ve­niences. How­ever, it is fair to first rec­og­nize the ad­van­tages. Mea­sure­ments provide us with knowl­edge, which can be achieved to achieve our goals, for ex­am­ple, hap­piness.

The knowl­edge pro­vided by the mea­sure­ments has differ­ent as­pects:

· The qual­i­ta­tive as­pect, which al­lows us to know what is hap­pen­ing.

· But can also be quan­ti­ta­tive, pro­vid­ing a pre­cise, quan­tified idea.

· Since there is a mea­sure, this also al­lows us rel­a­tive or com­pa­rable knowl­edge. Even when we have er­rors in the mea­sure­ment, and as long as the er­rors main­tain a cer­tain co­her­ence, we can com­pare two er­ro­neous mea­sure­ments to have a cor­rect idea of the evolu­tion in time, in space, or in any other di­men­sion.

Mea­sure­ments can also cause vir­tu­ous, per­haps un­fore­seen effects.

· Sev­eral stud­ies on the in­fluence of light­ing on work­ers’ pro­duc­tivity showed that it in­creased in cases where light­ing lev­els were higher, but also in those cases where light­ing was re­duced, sim­ply be­cause it was con­duct­ing the study (Hawthorne effect).

The in­con­ve­niences of the measurements

Th­ese are differ­ent ways in which mak­ing or hav­ing a mea­sure­ment can be in­con­ve­nient: 1) Cost 2) Er­ror 3) Mod­ifi­ca­tion of the mea­sured ob­ject (and even of the mea­sure it­self) 4) Un­wanted side effects 5) Mis­in­ter­pre­ta­tion 6) In­visi­bi­liza­tion.

1. Cost. Ob­tain­ing a mea­sure­ment has a cost that we would pre­fer if it did not ex­ist. This always hap­pens, or is there a case where tak­ing a mea­sure is free?

2. Er­ror. The mea­sure­ment could be sig­nifi­cantly in­cor­rect. If the mea­sure­ment were very bad, it might have been bet­ter not to have it. Some­times this oc­curs be­cause of the con­text in which the mea­sure­ment is performed.

· A con­flict of in­ter­est should make us dis­trust a mea­sure, how­ever pre­cise it may seem. It may be rea­son­able to dis­trust the stud­ies of the good­ness of the veg­etable diet con­ducted by ve­g­ans, as well as stud­ies of the good­ness of dairy foods fi­nanced by live­stock com­pa­nies, stud­ies of the benefits of psy­chol­ogy con­ducted by psy­chol­o­gists, or stud­ies of the safety of mo­bile telephony con­ducted by mo­bile phone com­pa­nies.

· Some­times feed­back is col­lected in situ­a­tions where it would be dis­turb­ing to make an open crit­i­cism (for ex­am­ple, the boss asks us in pub­lic for our opinion on the last de­ci­sion he has made). The re­sults may be very differ­ent from those ob­tained in an anony­mous sur­vey.

3. Mod­ifi­ca­tion of the mea­sured ob­ject (and even of the mea­sure it­self). A mea­sure­ment usu­ally af­fects the mea­sured el­e­ment, even if it is min­i­mally. In ad­di­tion, mea­sur­ing can pre­cisely mod­ify the vari­able we in­tend to mea­sure. This is known as ob­server’s para­dox.

· Con­tact with a ther­mome­ter will mod­ify the tem­per­a­ture of the ob­ject we in­tend to mea­sure.

· A cy­tol­ogy ex­tracts part of what is in­tended to study.

· When shoot­ing pho­tons (light) to an ob­ject to vi­su­al­ize it, we will be mod­ify­ing it.

4. Un­wanted side effects. The pre­vi­ous ex­am­ples do not seem very se­ri­ous, but un­for­tu­nately some­times the mea­sure­ment has a per­verse effect, just the op­po­site of the de­sired one:

· Some pre­na­tal di­ag­nos­tic tests (such as am­nio­cen­te­sis and co­rial biopsy) are in­va­sive and have a risk. Although they are performed to de­tect and avoid cer­tain com­pli­ca­tions, they can also cause oth­ers.

· In­stal­ling an office ac­cess con­trol sys­tem can make em­ploy­ees more punc­tual, but it can also mo­ti­vate them to spend hours drink­ing coffee chat­ting, in­stead of work­ing or vis­it­ing cus­tomers.

· Speed tests de­signed to as­sess the perfor­mance of com­put­ers, and in par­tic­u­lar, the perfor­mance of CPUs (a fun­da­men­tal el­e­ment in a com­puter) can cause man­u­fac­tur­ers to de­sign com­put­ers not in­tended to be very fast when work­ing with them, but to be very fast when the speed test is ap­plied (Good­hart’s law).

· If you have de­cided that your chil­dren go to school on the bus, there are a num­ber of pos­si­ble in­di­ca­tors or KPIs that may be of in­ter­est to give an idea of the qual­ity of the school trans­porta­tion ser­vice, such as punc­tu­al­ity of the bus, clean­li­ness, and road safety. Punc­tu­al­ity of the bus can be one of the ob­jec­tives; and keep­ing track of the punc­tu­al­ity of the bus is ex­tremely cheap and ac­cu­rate, but it can have a per­verse effect on safety. No doubt you do not want to con­vey to the driver of the school bus a great mo­ti­va­tion to be punc­tual, as this could en­courage him to in­crease the speed and risk to com­pen­sate for the de­lay caused by in­ci­dents of any kind.

· Try­ing to mea­sure hap­piness seems to frus­trate it. Ask­ing some­one at a party (or in a love or sex­ual act) to as­sess how well they are hav­ing fun can de­crease satis­fac­tion (al­though in other cases rec­og­niz­ing or thank­ing pos­i­tive things can en­hance them). “Hap­piness is found only in lit­tle mo­ments of inat­ten­tion” —João Guimarães Rosa. “Ask your­self whether you are happy, and you cease to be so” –John Stu­art Mill”. https://​​en.m.wikipe­dia.org/​​wiki/​​Para­dox_of_hedonism

5. Mis­in­ter­pre­ta­tion. Mea­sure­ment can con­fuse and give the wrong idea of re­al­ity, since it mea­sures what it mea­sures, not what it seems to mea­sure:

· If in sex sur­veys peo­ple re­spond that they have sex four days a week, this does not mean that peo­ple have sex four days a week. What this sur­vey tells us for sure is that when you ask them about sex, peo­ple say they have sex about four days a week.

· Vi­tamin D tests are not re­li­able (https://​​www.youtube.com/​​watch?v=I1uoc8ZN0m4)

· Es­ti­mated lev­els of cal­cium in the body are not re­li­able. The only good way to make a mea­sure­ment of cal­cium in the body is an au­topsy.

· Vi­tamin B12 mea­sure­ments are con­tro­ver­sial.

· In the same way that a tele­phone sur­vey leaves those in­di­vi­d­u­als who do not have a tele­phone out of the study, a sur­vey on well-be­ing or suffer­ing leaves out those in­di­vi­d­u­als who can­not an­swer us.

6. In­visi­bi­liza­tion. We usu­ally start mea­sur­ing what we can mea­sure well, and we lack mo­ti­va­tion to try to mea­sure what we can­not mea­sure well. In this way, the mea­sure­ment makes in­visi­ble the el­e­ments that are more difficult to mea­sure al­though they could be much more rele­vant. This in­creases the risk of ig­nor­ing those other el­e­ments and even in some cases, pro­mot­ing the idea that they do not ex­ist.

· Many of the vic­tims of sex­ual as­sault (women and men, girls and boys; on the street, in homes, and in pris­ons) do not re­port and hide the event, which makes their ac­count­ing difficult. There could be to­tally un­der­val­ued types or con­texts of sex­ual as­sault due to the difficulty of mea­sur­ing them.

· The death toll (num­ber of deaths) in an ac­ci­dent or in a con­flict can make suffer­ing -much more difficult to mea­sure- in­visi­ble. For ex­am­ple, surely 10 in­di­vi­d­u­als who die burned al­ive in an avi­a­tion ac­ci­dent, as a whole, suffer much more than 100 in­di­vi­d­u­als who die from con­cus­sion. But suffer­ing is difficult to mea­sure, and at­ten­tion goes to the num­ber we can eas­ily get. This could cause the es­tab­lish­ment of wrong pri­ori­ties, es­pe­cially if we add to this the effect that cer­tain things have to re­pel the at­ten­tion.

Th­ese in­con­ve­niences should make us think twice be­fore dis­card­ing anec­do­tal ev­i­dence ver­sus other more for­mal ev­i­dence. In some cases, it is pos­si­ble that for­mal and well-struc­tured ev­i­dence has been ob­tained in a way that dis­cards the re­al­ity that anec­do­tal ev­i­dence is sug­gest­ing.

How to avoid mea­sure­ment inconvenience

The draw­backs of the mea­sure­ments that have been iden­ti­fied are: 1) Cost 2) Er­ror 3) Mod­ifi­ca­tion of the mea­sured ob­ject (and even of the mea­sure it­self) 4) Un­wanted side effects 5) Mis­in­ter­pre­ta­tion 6) In­visi­bi­liza­tion.

To avoid these in­con­ve­niences, we can ask our­selves the fol­low­ing ques­tions:

· How much does it cost to get that data? Is it worth it?

· Could the data be in­cor­rect? How? What is the mar­gin of er­ror?

· Is there any other source or ev­i­dence that we can con­sult to col­late the data? Without over­whelming, the more, the bet­ter. It is said that a sailor must sail with a com­pass or with three but never with two, but it is not true: it is bet­ter to have two com­passes than just one. What cred­i­bil­ity do these other ev­i­dences have? Should we to­tally dis­card them?

· Who has ob­tained or will get the data? Is there a con­flict of in­ter­est? Is there any ide­ol­ogy be­hind who is be­hind that data?

· Does ob­tain­ing the data mod­ify the ob­ject we are mea­sur­ing? How? Is it ac­cept­able?

· Can ob­tain­ing the data have other side effects? Which? Are they per­verse or vir­tu­ous effects? For whom?

· What ex­actly does that data rep­re­sent? How do I in­ter­pret it? And how will oth­ers in­ter­pret it? Are these in­ter­pre­ta­tions cor­rect? Is the mean­ing of the data likely to be mi­s­un­der­stood?

· What has been left out when con­duct­ing the study? What does the data omit? What can be the effect of that omis­sion?