# steve.hind comments on How You Can Counterfactually Send Millions of Dollars to EA Charities

Sure:

### Cash balance methodology

You say:

Dividing \$26,260 by GiveWell’s end of year cash balance in 2019—the \$41,101,155 we calculated in the previous section—produces an upper bound interest rate of 0.0006389, or 0.064%.

The U.S. federal funds rate, an interest rate that represents the minimum bound of a good rate of interest on a savings account, averaged around 2.16% in 2019.

Multiplying GiveWell’s \$41,101,155 in cash at the end of 2018 by a 2.16% rate of interest yields a total of \$887,785 in potential interest earnings which is nearly 34 times higher than the actual investment income GiveWell earned.

Subtracting the \$26,260 upper bound on how much GiveWell actually earned on its cash in 2019 results in an annualized opportunity cost of \$861,525.

Incorrect and misleading statements in these four paragraphs:

• First paragraph: Dividing interest earned over the year by end of year cash is not an upper bound estimate interest. It’s actually more like a lower bound estimate, because it implicitly assumes that the cash level at end of year was the same as the average cash level throughout the year.

• Since GiveWell is a grant-making organization that’s a very dangerous assumption to make (and their 990 form shows them making ~\$30m in grants over the course of the year).

• You acknowledge this in the next section, but then propose an equally questionable workaround:

GiveWell earns interest on its average cash balance during the year, rather than its cash balance at one point in time like the end of the year. We can better approximate this figure by averaging GiveWell’s beginning of year and end of year zero-interest and non-zero-interest cash and then summing them. This produces an estimated average cash balance in 2019 of \$33,441,682.

You may not be understanding what the article means by “cash balance.” As I make very clear in the article, we are using the organization’s estimated average daily cash balance to calculate the historical and future opportunity costs. The average daily cash balance is the amount that an organization literally makes interest on; I can’t think of a better number to use [emphasis added] to calculate the interest an organization previously made and could be making in the future.

Even this approach seems incorrect or misleading:

• The page for their Maximum Impact Fund shows the grants they make (which would come out of the cash balance shown in the 990), and it seems to show a couple of months lag between donations and grants.

• The same page shows that funds received in the final three months of 2019 were substantially greater than those received in the first nine months of 2019, which is another reason that the end-of-2019 cash balance paints a misleading picture.

• Second paragraph: The year you’ve chosen for your analysis happens to be the year with the highest cash rates in the past 12 years. This isn’t acknowledged, and you don’t backtest your strategy in other years (e.g. all the years where rates were ~0%).

• Third & fourth paragraphs: Same analytical error as first paragraph.

### Potential returns of other investment options

You say:

The 2.16% U.S. federal funds rate in 2019 is one of the most conservative interest rates possible. Higher interest rates would result in significantly higher annual opportunity costs. Our top recommendation for an ultra-short-term bond ETF (JPST) in our recommendation article published in early 2019 earned 3.34% in 2019, according to Portfolio Visualizer. There were no drops in account value on a monthly basis (0% maximum drawdown). Multiplying GiveWell’s \$30,033,677 reasonable savings balance by 3.34% generates an opportunity cost of over one million dollars per year, at \$1,003,124.

Citing the one year, backward looking return of JPST is misleading, and citing the one year drawdown performance is naive:

• There’s nothing about that asset that makes its one year backward-looking return able to be extrapolated. Indeed this is basically the equivalent of picking a different asset that you know now performed well over a one year period and using that to estimate opportunity cost. Why not Bitcoin in 2017 or Tesla stock in 2020?

• And re citing a year of drawdown data—the fund had a drawdown in March 2020! If you look at a similar index with a longer timeseries you’ll see there have been drawdowns, even against the backdrop of a period of secularly falling interest rates that juiced bond returns.

• What’s more, that particular fund tracks a bond benchmark but is actively managed, meaning its managers can take more risk than the benchmark implies (and charge a management fee for doing so).

Using an even higher interest rate of 5.51%, which is what a very conservative portfolio of 20% stocks and 80% bonds earned on average (CAGR) from January 2000 to December 2019 calculated using Portfolio Visualizer, the one-year opportunity cost on GiveWell’s 2019 reasonable savings balance would be \$1.65 million dollars a year. The 5.51% figure is an approximation of GiveWell’s expected opportunity cost rather than what it would have experienced, which would have been a small loss of −1.15% in 2018 followed by a 13.01% gain in 2019. For those curious about how this portfolio would have fared during COVID-19, the maximum negative month-over-month change in account value was −3.20% since the beginning of 2020; the portfolio is now up 9.11% for the YTD period ending November 30, 2020.

• This analysis suffers from the same problem. At least there’s an attempt to look at a window longer than one year, but again it’s a cherry-picked period of 20 years, and it’s backward looking returns data implied to be forward-looking. You also mention the drawdown in 2018. What was the maximum drawdown in 2008 to 2009?

• Given the organization is granting funds a couple of months after receiving them, it has little ability to tolerate drawdowns, making comparisons to a higher risk fund incorrect to misleading.

### Conclusion and staff time

You say:

Regardless of the estimation method, it’s very likely the counterfactual impact of changing GiveWell’s financial practices is in the millions of dollars. Not bad for a time investment that should take between 1–10 hours of staff time to set up, and 0–2 hours of staff time to administer on an annual basis!

• As shown above, getting a counterfactual impact number in the millions of dollars is not “very likely… regardless of the estimation method”. That statement is somewhere between incorrect and misleading. For instance if the cash balance is 10% of what you cite, and the interest rate is more like 0.50%, it doesn’t hold up.

• The staff time estimate is also somewhere between incorrect and misleading:

• If it’s just opening bank accounts, then the impact is GiveWell on average earning roughly the prevailing bank interest rate (close to zero for much of the relevant period)

• If instead you want them (without the benefit of hindsight that you employ) picking actively managed JP Morgan bond funds, or creating and balancing a bond + stock fund, then I’d hazard it’s more than two hours of work per year!

• So either it’s not much effort, and not much return, or there can be higher returns, but clearly it’s more than 2 hours a year of effort.

Cash balance methodology

Dividing interest earned over the year by end of year cash is not an upper bound estimate interest

In the sentence directly above this quote, I explain what I mean by upper bound: “GiveWell’s total cash and investment income in 2019 was \$26,260. GiveWell held investments in 2019, so using the full amount on Line 3 to estimate GiveWell’s interest on cash in 2019 likely overestimates GiveWell’s actual interest earned on cash in 2019.”

I am saying that the number overstates the interest earned on cash alone because of the confounding effect of investment income.

It’s actually more like a lower bound estimate, because it implicitly assumes that the cash level at end of year was the same as the average cash level throughout the year.

The reason why this methodology is being used is stated in this first line of the article “This methodology is easy for anyone to replicate and is based on publicly available nonprofit financial statements (IRS Form 990) that all U.S. charities are required to file yearly.”

The Form 990 does not include a better estimate of an organization’s average daily cash balance. This is why I am calling this an “estimation” methodology. In the “Estimation Methodology Caveats” section I cover this and other issues in depth (and I’ve already mentioned this to other commenters, who also brought up this valid point). A short quote: “The average of a nonprofit’s start of year and end of year cash could noticeably underestimate or overestimate a nonprofit’s actual average cash balance during the year because it only uses two days as data points...”

Since GiveWell is a grant-making organization that’s a very dangerous assumption to make (and their 990 form shows them making ~\$30m in grants over the course of the year).

GiveWell is used as an example—this analysis is meant to highlight the methodology, including its shortcomings, not be a deep dive into GiveWell itself. I don’t have the data needed to estimate this better. Neither you nor I know how much money GiveWell actually holds during other times of the year. They receive funds year-round and need reserves on hand to pay operational expenses.

You acknowledge this in the next section, but then propose an equally questionable workaround

That is not correct, the workaround is, as I state in the article, meant to account for the confounding effects of money made from investments rather than cash holdings by only looking at the cash balance we know isn’t earning any interest. This “workaround” has nothing to do with the cash balance issue you mentioned.

The same page shows that funds received in the final three months of 2019 were substantially greater than those received in the first nine months of 2019, which is another reason that the end-of-2019 cash balance paints a misleading picture

Thanks for pointing this out. I didn’t look more closely into GiveWell’s specifics because GiveWell is used as an example of a methodology that generalizes to other charities as I stated in the article. Data like this is very difficult to correlate with an organization’s cash balance. We don’t know what percentage of GiveWell’s cash balance the Maximum Impact Fund is, what the Maximum Impact Fund balance is after a grant is made, whether the “amount” column reflects all donor inflows during a specified period or also granting from reserves or other revenue sources, etc.

The year you’ve chosen for your analysis happens to be the year with the highest cash rates in the past 12 years.

The most recent data is most relevant to GiveWell’s current and future opportunity costs.

This isn’t acknowledged, and you don’t backtest your strategy in other years (e.g. all the years where rates were ~0%).

That is a good point! Time did not permit, but as I stated in my reply to Rob, I do think that “to get savings accounts yields themselves, we might need to use a third-party service or design a benchmark ourselves. Given the current situation with the federal funds rate, it would make sense for us to find a better proxy for good savings accounts yields in 2020 so that we can better estimate the opportunity cost for 2020 once the 2020 Form 990s come out as well as use the rate as a point of reference for current and future expected earnings.”

So it is a good idea to use an alternative metric to the federal funds rate, particularly for lower interest rate years, so they more accurately reflect the opportunity cost. Off the top of my head I think that rates were around 1% (see Ally Bank’s 2015 and 2016 historical yields when the fed funds rate was 0.13% and 0.39% respectively), which is the exact same rate I used for the forward-looking estimate for GiveWell. Using savings yields from one bank or a blend of banks instead still wouldn’t change the article’s conclusions, given that those rates would be both higher than the federal funds rate and also significant enough to warrant consideration even in a zero interest rate environment.

Potential returns of other investment options

There’s nothing about that asset that makes its one year backward-looking return able to be extrapolated. Indeed this is basically the equivalent of picking a different asset that you know now performed well over a one year period and using that to estimate opportunity cost. Why not Bitcoin in 2017 or Tesla stock in 2020?

Exactly, why wouldn’t I pick Bitcoin or Tesla? Because both ultra-short-term bonds and a standard conservative risk portfolio are extremely common investment portfolios for more risk averse investors such as many nonprofits who require a steady rate of return with minimal drawdowns. It doesn’t matter what examples you use, low-risk options would have performed well during this time period and other time periods, just worse than riskier options.

Citing the one year, backward looking return of JPST is misleading

And another reason I used JPST as an example is because I recommend JPST in my early-2019 article before I could see its historical performance. So I’m not sure if “backward looking” is the right verbiage.

And re citing a year of drawdown data—the fund had a drawdown in March 2020! If you look at a similar index with a longer timeseries you’ll see there have been drawdowns, even against the backdrop of a period of secularly falling interest rates that juiced bond returns.

I already state in the article that I am quoting month-to-month drawdown data, not day-to-day data. This means the fund did not decline in value month by month. The highest historical month-to-month drawdown was −1.72% and it recovered in two months. Hardly a volatile investment.

This analysis suffers from the same problem. At least there’s an attempt to look at a window longer than one year, but again it’s a cherry-picked period of 20 years, and it’s backward looking returns data implied to be forward-looking. You also mention the drawdown in 2018. What was the maximum drawdown in 2008 to 2009?

Given the organization is granting funds a couple of months after receiving them, it has little ability to tolerate drawdowns, making comparisons to a higher risk fund incorrect to misleading.

I did not cherry pick the time period, stocks and bonds have generally performed well across time. Portfolio Visualizer has data starting from 1987, which indicates an annual return even higher than what I quoted, at 7.07%. Maximum drawdown is −8.49% from the 2008 crash. Exactly why this is considered a standard conservative portfolio by financial professionals.

Given the organization is granting funds a couple of months after receiving them, it has little ability to tolerate drawdowns, making comparisons to a higher risk fund incorrect to misleading.

As we can see, the drawdown is very low, and the expected value is positive and significant. For granted funds that GiveWell cannot afford to have a drawdown on (this does not represent all of their funds), then a “zero-risk” option like a savings account would make more sense. It all depends on the charity. Investment examples are for illustrative purposes, I have no idea what approaches GiveWell can or cannot take in actuality, beyond a savings account, which it should be able to do.

Conclusion and staff time

As shown above, getting a counterfactual impact number in the millions of dollars is not “very likely… regardless of the estimation method”. That statement is somewhere between incorrect and misleading. For instance if the cash balance is 10% of what you cite, and the interest rate is more like 0.50%, it doesn’t hold up.

That depends on what numbers we’re using. I think 10% is very low. As I’ve said, the Maximum Impact Fund cannot be used as a reliable estimate or predictor of GiveWell’s average daily cash balance. But if we assume just the fund itself represents all of GiveWell’s money, we can see that there’s a 3-month grant collection period (averages to 1.5 months of cash at 100%) plus a 2–3 month delay in granting (averages to 2.5 months of cash at 100%) which would suggest a minimum percentage of at least 33% (4/​12 months) just based on the money collected during the last three months of the year.

The staff time estimate is also somewhere between incorrect and misleading:

• If it’s just opening bank accounts, then the impact is GiveWell on average earning roughly the prevailing bank interest rate (close to zero for much of the relevant period)

• If instead you want them (without the benefit of hindsight that you employ) picking actively managed JP Morgan bond funds, or creating and balancing a bond + stock fund, then I’d hazard it’s more than two hours of work per year!

• So either it’s not much effort, and not much return, or there can be higher returns, but clearly it’s more than 2 hours a year of effort.

This is probably evident from the rest of my replies, but a 1% account savings rate or even 0.5% still produces significant financial returns. These numbers should be used instead of 0%.

And it would take a very small amount of time to keep funds in one fund for multiple years, whether that’s JPST or a multi-asset fund that has, say, a 20% stock allocation and 80% bond allocation. Vanguard, a well-known and trusted provider, has a great ultra-short-term bond fund as a JPST alternative. A nonprofit might not maximize returns by shifting between funds on the reg (which is, as you say, time consuming, although Antigravity Investments is more than happy to help with that), but evidently they will do much better than nothing. Many nonprofits self-manage their investment approach.