The Important/​Neglected/​Tractable framework needs to be applied with care

The Im­por­tant/​Ne­glected/​Tractable (INT) frame­work has be­come quite pop­u­lar over the last two years. I be­lieve it was first used by GiveWell for their cause and in­ter­ven­tion in­ves­ti­ga­tions. Since then Giv­ing What We Can and 80,000 Hours have adopted it with the ad­di­tion of ‘per­sonal fit’ as a fur­ther crite­rion, and it has been fea­tured promi­nently in Will MacAskill’s book Do­ing Good Bet­ter.

This qual­i­ta­tive frame­work is an al­ter­na­tive to straight cost-effec­tive­ness calcu­la­tions when they are im­prac­ti­cal. Briefly the ar­gu­ments in favour of scor­ing a cause or char­ity us­ing a mul­ti­ple-crite­ria frame­work are:

  • It is more ro­bust to a ma­jor er­ror in one part of your re­search.

  • It forces you to look at some­thing from mul­ti­ple differ­ent an­gles and seek sup­port from ‘many weak ar­gu­ments’ rather than just ‘one strong ar­gu­ment’.

  • In prac­tice it leads to faster re­search progress than try­ing to come up with a per­sua­sive cost-effec­tive­ness es­ti­mate when raw in­for­ma­tion is scarce.

I favour con­tinued use of this frame­work. But I have also no­ticed some traps it’s easy to fall into when us­ing it.
Th­ese crite­ria are ‘heuris­tics’ that are de­signed to point in the di­rec­tion of some­thing be­ing cost-effec­tive. Un­for­tu­nately, us­ing the com­mon defi­ni­tions of these words they blur heav­ily into one an­other. For ex­am­ple:
  • A cause can re­main im­por­tant be­cause it’s ne­glected.

  • A cause can re­main tractable be­cause it’s ne­glected.

  • A cause can be tractable be­cause it’s im­por­tant.

Some­times I have seen pro­jects eval­u­ated us­ing this frame­work dra­mat­i­cally over­weight one ob­ser­va­tion, be­cause it can fit into mul­ti­ple crite­ria.
Nat­u­rally, a cause that’s highly ne­glected will score well on ne­glect­ed­ness. It can then go on to score very well on im­por­tance be­cause its ne­glect­ed­ness means the prob­lem re­mains se­ri­ous. Fi­nally, it can also score well on tractabil­ity, be­cause the most promis­ing ap­proaches are not yet be­ing taken, be­cause it’s ne­glected.
Most of the case in favour of the cause then boils down to it be­ing ne­glected, rather than just one third, as origi­nally in­tended.
A way around this prob­lem is to rate each crite­ria, while ‘con­trol­ling for’ the oth­ers.
For ex­am­ple, rather than ask ‘how im­por­tant is this cause?‘, in­stead ask ‘how im­por­tant would it be if no­body were do­ing any­thing about it?‘, or ‘how im­por­tant is this cause rel­a­tive to oth­ers that are equally as ne­glected?‘. Rather than ask ‘how tractable is this cause on the mar­gin?‘, in­stead ask ‘how tractable would it be if no­body were work­ing on it?‘, or ‘how tractable is it rel­a­tive to other causes that are equally ne­glected?’.
Those ques­tions would be in­ap­pro­pri­ate if we weren’t con­sid­er­ing ne­glect­ed­ness sep­a­rately, but for­tu­nately we are.
If you do make this ad­just­ment, make sure to do it con­sis­tently across all of the op­tions you are com­par­ing.

--

Another pit­fall is to look at the wrong thing when eval­u­at­ing ‘Im­por­tance’ (some­times also called ‘Scale’). Imag­ine you were look­ing into ‘un­con­di­tional cash trans­fers’ dis­tributed by GiveDirectly. What should you look at to see if it’s solv­ing an ‘Im­por­tant’ prob­lem? Two nat­u­ral op­tions are:

  • The to­tal num­ber of peo­ple liv­ing in ex­treme poverty who could benefit from cash trans­fers.

  • The sever­ity of the poverty of GiveDirectly’s re­cip­i­ents.

Which should you use? It de­pends what you think you are fund­ing.
When GiveDirectly was first de­vel­op­ing ‘un­con­di­tional cash trans­fers’, care­fully study­ing their im­pact, and pop­u­laris­ing the ap­proach, the most im­por­tant mea­sure was the first. If their cash trans­fers proved to be a huge suc­cess, then the num­ber of peo­ple who could benefit from the in­no­va­tion was the 1-2 billion peo­ple with ex­tremely low in­comes.
But to­day a lot of this re­search has been done, and many or most peo­ple in de­vel­op­ment are aware of the ar­gu­ments for cash trans­fers. Scal­ing up GiveDirectly fur­ther is only go­ing to have mod­est learn­ing and demon­stra­tion effects.
For that rea­son, I now think the most suit­able crite­ria for eval­u­at­ing the im­por­tance of the prob­lem GiveDirectly is solv­ing, rel­a­tive to oth­ers, is the sec­ond: how des­per­ately poor are their re­cip­i­ents rel­a­tive to other benefi­cia­ries we could help?
To gen­er­al­ise:
  • If some­one is in­vent­ing some­thing new, what you want to know is the size of the to­tal prob­lem that in­ven­tion could con­tribute to solv­ing.

  • If some­one is scal­ing up some­thing that already ex­ists, what you want to know is the sever­ity of the prob­lem is that it solves per re­cip­i­ent (or, al­ter­na­tively per dol­lar that is spent on it).

If you are try­ing to com­pare in­vent­ing new things against scal­ing up things that already ex­ists, be pre­pared to face se­ri­ous challenges with any frame­work I know of.