Type Checking GiveWell’s GiveDirectly Cost Effective Analysis

SamNolan23 Jun 2021 6:35 UTC
109 points
2 comments6 min readEA link

tldr: You can check for the consistent use of units to find assumptions and/​or errors within calculations. I use this on GiveDirectly’s Cost Effective Analysis by GiveWell to find two implicit assumptions that impact the result. I think they should probably be included explicitly within the model, and that this technique could be used for other Cost Effectiveness Analysis. If you are not interested in the specifics of how the effectiveness of charities are calculated, you will probably find this read a waste of time.

Numerical models could describe the world, or it might not describe the world. However, there’s an interesting way that you can work out whether your model does describe the world without even looking at any data.

To do that, we turn to the field of Dimensional Analysis. Dimensional Analysis is based off really simple ideas. Basic Dimensional Analysis pretty much has two rules:

  1. You can only equate, add or subtract numbers that have the same units

  2. Multiplying/​dividing numbers means multiplying/​dividing their units

We use dimensional analysis all the time. If I have 4 apples per bag, and 5 bags, then we apples. In this case we multiplied the units together

This is the second rule above. We however, cannot add units that are different. It doesn’t make sense to say that I have 4 apples per bag + 5 bags.

We do not know what the units of the result would be. This is the first rule above. What’s great about physical equations is that you can vary the units of the equation and it still means the same thing. Say instead of talking about apples, you cut each apple in half and talk about apple halves. This changes the units from apples to apple halves. So we would have 8 apple halves per bag and 5 bags. How many apple halves do you have? Well, we have apple halves, which is exactly the same prediction as 20 apples. The fact that we can change our units without changing the meaning of a physical equation I’ll call “scale invariance”.

If you have equations that are not physical, then that law can be violated. For instance, if we assumed (wrongly) that we can add the apples per bag and bags to get apples.

Then scale invariance will be violated. Because in the first example, if you have 4 apples per bag, and 5 bags, this would predict you would have apples in total. However, if we start talking about apple halves, and say we have 8 apple halves per bag, then the equation would give apples halves. 13 apple halves is not the same as 9 apples. Therefore scale invariance has been violated and this equation therefore must be wrong.

If dimensional analysis is not followed, then simply changing the units of your equation would change it’s results! An equation that follows these rules is deemed “physical”.

In physics, this is such a powerful tool in finding incorrect equations, that it’s possible to derive many equations by looking at the units of the parameters alone. I was particularly blown away by an example of deriving the range equation from the units of it’s parameters on Wikipedia.

I’m from a software background, so I like to consider this as type checking math. Hence the title.

There are a lot of long calculations done in Cost Effectiveness Analysis (CEAs). As of such, I investigated as to whether these calculations were physical for GiveDirectly. That is, do these calculations abide by the rules of Dimensional Analysis.

Dimensional Analysis has two important but distinct concepts, “units” and “dimensions”. Examples of dimensions include “length”, “time”, “money”. Examples of units include “meters”, “feet”, “seconds”, “years” etc. Each unit belongs to a dimension.

A quick overview of GiveDirectly’s CEA

Before going further, if you’re not familiar with GiveWell’s CEAs, here’s a quick overview of the way GiveDirectly’s cost effectiveness is calculated. It’s deliberately only the parts that are needed to understand this post. If you want the full details, you can take a look at their spreadsheet.

The calculation is done by modeling giving funds to the recipient. The recipient then invests a portion of the funds and uses the other portion. The portion that they use increases their consumption by a certain amount. The investment, which could be things such as fixing a roof or investing in their business, allows them to increase their consumption over a longer period of time. After the benefits end from the investment, a certain amount of the investment’s worth is returned to the recipient, also increasing their consumption.

The total increased consumption is the increase consumption from the money not invested, plus the amount of consumption increased from the investment, plus the amount of consumption increased when the benefits end. The value of the donation is calculated by multiplying this total increased consumption by a moral weight, indicating the value of the increase of consumption.

Checking Scale Invariance

Thankfully, there’s a simple way of checking whether a numerical model is physical. All we need to do is change the units of the calculation and see if the result changes because of it. If it does, the calculation violates scale invariance, and for some reason the result depends on the units of measure that are used to calculate it.

GiveWell’s CEAs all have a “total units of value generated” as the result of their calculations. This result is in “Value” units. This means that if I change the definition of units in other dimensions (such as swapping out Time in years for Time in days), then it shouldn’t impact that result. If it does, we have a problem!

It should be noted that this does not guarantee that GiveDirectly’s model is correct if this check passes. This is a very rudimentary check, in fact Wikipedia calls it a “sanity check”. Just because your program compiles doesn’t mean it’s correct.

Results

So, is GiveDirectly’s CEA physical? Does it abide by the laws of Dimensional Analysis?

Well, actually no, but honestly not so much in a bad way. The resulting spreadsheet is here. The model is scale invariant for money, households and people but not time. Meaning if I set the time units to be anything that’s not years, the value calculated will change. The reason for this is because there’s two implied parameters that are not included in the model. Those parameters are:

  • Time for which the recipient consumption is increased from funds not invested

  • Time for which the recipient consumption is increased from the funds returned from the investment when benefits end

These two values were left unspecified, and therefore assumed whatever 1 was in
the current units for time (which in this case, is the reasonable 1 year). If I changed the units down to days, then that meant that these parameters would change to 1 day, changing the results.

The results depend on these two parameters because wellbeing is measured as a log increase in consumption. This means that if consumption is increased but only in a short period of time, the wellbeing isn’t as large as a small consumption increase over a long period of time. I’ve made a graphical and interactive explanation of the concept here.

Including those two parameters fixes scale invariance under time. Although it might seem pedantic, I think GiveWell should include these two parameters in their model explicitly, because they do change the result of the calculation when changed.

Assuming these parameters are one year, which I believe is a fair assumption, This doesn’t change the cost effectiveness of GiveDirectly at all. Although it might mean that these parameters need to be justified in some way.

Problems and caveats

Just a warning, this section is a bit technical, and requires a proper understanding of how GiveDirectly’s cost effectiveness is calculated.

To do this analysis, I had to change the definition of the Present Value slightly. Instead of using a discrete Present Value, calculating what would happen if you get discrete payouts at the end of the year, I chose a continuous option, which is like if I got millions of tiny payouts that slowly decreased over time. This gets very similar results, but bumps the total units of value for GiveDirectly up by a few points. A discrete version of the CEA is still valid under the rules of Dimensional Analysis, it just meant that the units of the discount rate became incredibly complicated, and something I couldn’t even solve for.

The calculation for the discrete Present Value, for payment , rate and duration is:

Whereas the continuous version of the same quantity is:

Although I did this for mathematical convenience alone, I think that this might model the benefits likely received better than a more discrete model. This is because the investment’s discussed when looking at GiveDirectly are not so much like receiving benefits at the end of every year, but increased consumption from not spending time and money repairing roofs. That being said, for someone with an economics background, the PV function might make more sense than the continuously equivalent integral. For a discount rate of 4%, the continuous version is 2% larger.

I’m fairly certain that my patched model still has problems. These problems are however minor considering the uncertainty of the parameters and the model itself.

One problem I can name is that the recipient is modeled to increase consumption from the investment and from money not invested in the first year. However the increase in consumption in the first year are both calculated as ratios from the baseline consumption, then multiplied (added under logs) together. Meaning that if the is the baseline consumption, is the consumption increased from investment and is the consumption increased from funds transferred (not invested). The consumption in the first year is modeled as:

instead of:

this issue however is remarkably pedantic, and is pretty much
lost considering the uncertainty of the parameters and model in the calculation.

The model probably also has other issues. If you see any more, please let me know.

I think that dimensional analysis is a useful tool for investigating the validity
and assumptions of other CEA. I’d love to continue this technique with the other
models done by GiveWell.

SamNolan23 Jun 2021 6:35 UTC
109 points
2 comments6 min readEA link