The Important/​Neglected/​Tractable framework needs to be applied with care

[Robert_Wiblin]

The Important/​Neglected/​Tractable (INT) framework has become quite popular over the last two years. I believe it was first used by GiveWell for their cause and intervention investigations. Since then Giving What We Can and 80,000 Hours have adopted it with the addition of ‘personal fit’ as a further criterion, and it has been featured prominently in Will MacAskill’s book Doing Good Better.

This qualitative framework is an alternative to straight cost-effectiveness calculations when they are impractical. Briefly the arguments in favour of scoring a cause or charity using a multiple-criteria framework are:

• It is more robust to a major error in one part of your research.

• It forces you to look at something from multiple different angles and seek support from ‘many weak arguments’ rather than just ‘one strong argument’.

• In practice it leads to faster research progress than trying to come up with a persuasive cost-effectiveness estimate when raw information is scarce.

I favour continued use of this framework. But I have also noticed some traps it’s easy to fall into when using it.

These criteria are ‘heuristics’ that are designed to point in the direction of something being cost-effective. Unfortunately, using the common definitions of these words they blur heavily into one another. For example:

• A cause can remain important because it’s neglected.

• A cause can remain tractable because it’s neglected.

• A cause can be tractable because it’s important.

Sometimes I have seen projects evaluated using this framework dramatically overweight one observation, because it can fit into multiple criteria.

Naturally, a cause that’s highly neglected will score well on neglectedness. It can then go on to score very well on importance because its neglectedness means the problem remains serious. Finally, it can also score well on tractability, because the most promising approaches are not yet being taken, because it’s neglected.

Most of the case in favour of the cause then boils down to it being neglected, rather than just one third, as originally intended.

A way around this problem is to rate each criteria, while ‘controlling for’ the others.

For example, rather than ask ‘how important is this cause?‘, instead ask ‘how important would it be if nobody were doing anything about it?‘, or ‘how important is this cause relative to others that are equally as neglected?‘. Rather than ask ‘how tractable is this cause on the margin?‘, instead ask ‘how tractable would it be if nobody were working on it?‘, or ‘how tractable is it relative to other causes that are equally neglected?’.

Those questions would be inappropriate if we weren’t considering neglectedness separately, but fortunately we are.

If you do make this adjustment, make sure to do it consistently across all of the options you are comparing.

--

Another pitfall is to look at the wrong thing when evaluating ‘Importance’ (sometimes also called ‘Scale’). Imagine you were looking into ‘unconditional cash transfers’ distributed by GiveDirectly. What should you look at to see if it’s solving an ‘Important’ problem? Two natural options are:

• The total number of people living in extreme poverty who could benefit from cash transfers.

• The severity of the poverty of GiveDirectly’s recipients.

Which should you use? It depends what you think you are funding.

When GiveDirectly was first developing ‘unconditional cash transfers’, carefully studying their impact, and popularising the approach, the most important measure was the first. If their cash transfers proved to be a huge success, then the number of people who could benefit from the innovation was the 1-2 billion people with extremely low incomes.

But today a lot of this research has been done, and many or most people in development are aware of the arguments for cash transfers. Scaling up GiveDirectly further is only going to have modest learning and demonstration effects.

For that reason, I now think the most suitable criteria for evaluating the importance of the problem GiveDirectly is solving, relative to others, is the second: how desperately poor are their recipients relative to other beneficiaries we could help?

To generalise:

• If someone is inventing something new, what you want to know is the size of the total problem that invention could contribute to solving.

• If someone is scaling up something that already exists, what you want to know is the severity of the problem is that it solves per recipient (or, alternatively per dollar that is spent on it).

If you are trying to compare inventing new things against scaling up things that already exists, be prepared to face serious challenges with any framework I know of.

[MichaelDickens]

A way around this problem is to rate each criteria, while ‘controlling for’ the others.

For example, rather than ask ‘how important is this cause?‘, instead ask ‘how important would it be if nobody were doing anything about it?‘, or ‘how important is this cause relative to others that are equally as neglected?‘. Rather than ask ‘how tractable is this cause on the margin?‘, instead ask ‘how tractable would it be if nobody were working on it?‘, or ‘how tractable is it relative to other causes that are equally neglected?’.

Those questions would be inappropriate if we weren’t considering neglectedness separately, but fortunately we are.

This strikes me as a weird way to go about it. You’re essentially taking your best estimate about how effective a cause is, breaking out its effectiveness into three separate factors, and then re-combining those three factors to estimate its effectiveness. You’re going to lose information at each step here, and end up back where you started.

It seems to me that the importance/​tractability/​neglectedness framework is more useful when you have a good sense of a cause’s overall importance but you don’t know about its importance on margin—factoring in tractability and neglectedness helps you understand the value of marginal investment.

[Paul_Lang]

I am wondering if 80k should publicly speak of the PINT instead of the INT framework (with P for personal fit). This is because I get the impression that the INT framework contributes to generating a reputation hierarchy of cause areas to work on; and many (young) EAs tend to over-emphasize reputation over personal fit, which basically sets them up for failure a couple of years down the line. Putting the “P” in there might help to avoid this.

[Stefan_Schubert]

Good post. I agree that this framework, though somewhat useful, sometimes seems to be taken as more precise than it is. I’ve also thought about how it could be improved, but without notable success so far (one problem is that in order to become more precise, you need more criteria, which threatens to make the framework too cumbersome). I also thought about the “controlling for other factors”-idea, which I think people should use.

[Pablo]

[posted in wrong thread]

[Austen_Forrester]

I completely agree that neglectedness is often the route cause of the size and tractability, but that’s the whole point. Neglectedness only matters inasmuch as it affects the other two criteria and is included on it’s own mostly to aid analysis of tractability – it has no value on it’s own. For instance, if a cause is low in importance/​scale and tractability, it wouldn’t matter what the neglectedness is at all. I think the neglectedness factor only comes into play if a problem is important and tractable (including on the margin), yet crowded. In that case, even though it would be a high priority for an organization to enter because the only two relevant factors (scale and tractability) are high, if it is a very popular cause, it may become less tractable/​important in the near future with more entrants into the space.

I think that the type of importance analysis depends on whether the intervention is scalable or systemic, not whether or not it is new. In systemic action, you are tackling the entire issue, so you want to judge the importance of the issue’s entirety. Scalable action works on a per unit basis, so the severity of the issue per person/​animal/​acre, etc is what matters, as you mentioned regarding GiveDirectly.

Lastly, I think that flow through effects should always be included among cause prioritization criteria, because (cheese warning) what matters at the end of the day, is how much the entire world benefits from an intervention, not just how much it benefits the targeted cause. For instance, research intended to progress one subject often ends up providing unforseen benefits in other areas.

[Robert_Wiblin]

An alternative option would be to scrap neglectedness as an independent criterion and consider it in importance and tractability.

I think we agree on the the systemic/​scalable distinction—perhaps I just expressed myself poorly.

[Austen_Forrester]

It may be more technically correct not to have neglectedness as a separate criterion, but I find that it is the single most important factor in cause prioritization, despite the fact that it’s only utility is its affect on the other two factors. Just my personal observation. For example, the causes that I think have the highest expected value: pesticide poisoning, self harm/​suicide, depression in the Third World, loneliness, are all great in magnitude and have huge potential for progress precisely because they have been severely neglected. That’s why I’ve come to see neglectedness as the starting point in cause selection, even though I now view it as a subfactor rather than an independent factor. So technical correctness may not necessarily lead to the most usefulness.

Perhaps someone will come up with a linear process for strategic cause prioritization one day, rather than the current practice of referring to static criteria. For instance, the process could start with brainstorming what you think the biggest sources of suffering (or impediments to flourishing) in the world are, followed by an examination of why those sources exist, then on to how they can be solved, and so on. It could be represented visually in a diagram. A process rather than set criteria could also be useful in unexpected ways. For example, I consider loneliness (lack of positive socialization or a romantic partner) to be one of the biggest social issues in the world because it is one of the largest sources of sorrow in the world. However, most people wouldn’t even think about taking action on it because they don’t even view it as a social issue. Thinking logically, however, it must be a social issue because it causes so much suffering, but cause prioritization using set criteria would be unlikely to lead you to that conclusion without using some sort of critical thinking process.

Similar benefits could probably be had without using a process by branching off each criterion into sub-components. For example, the importance bubble could have bubbles around it of “Why are people not happier?”, “What prevents flourishing?”, “Why is a society not better than it currently is?”, “What is an advancement in society that would lead to other improvements?”