Thanks for taking the time to list those out Steve. I downvoted your comments because while I’m very happy to engage in this type of discussion with commenters, and have with others that have commented on this article, I feel like you are jumping to conclusions that things are “incorrect” or “misleading.” In fact, many of your points are already mentioned in the article itself. Additionally, you are attacking my writing by making comments like something is “naive” which I feel is not conducive to a positive and intellectual discussion about this. You’ll find that I’ve thought out the reasoning for all of the points you mention, either in the article itself, or because it’s fairly easy to understand why.
Cash balance methodology
Dividing interest earned over the year by end of year cash is not an upper bound estimate interest
In the sentence directly above this quote, I explain what I mean by upper bound: “GiveWell’s total cash and investment income in 2019 was $26,260. GiveWell held investments in 2019, so using the full amount on Line 3 to estimate GiveWell’s interest on cash in 2019 likely overestimates GiveWell’s actual interest earned on cash in 2019.”
I am saying that the number overstates the interest earned on cash alone because of the confounding effect of investment income.
It’s actually more like a lower bound estimate, because it implicitly assumes that the cash level at end of year was the same as the average cash level throughout the year.
The reason why this methodology is being used is stated in this first line of the article “This methodology is easy for anyone to replicate and is based on publicly available nonprofit financial statements (IRS Form 990) that all U.S. charities are required to file yearly.”
The Form 990 does not include a better estimate of an organization’s average daily cash balance. This is why I am calling this an “estimation” methodology. In the “Estimation Methodology Caveats” section I cover this and other issues in depth (and I’ve already mentioned this to other commenters, who also brought up this valid point). A short quote: “The average of a nonprofit’s start of year and end of year cash could noticeably underestimate or overestimate a nonprofit’s actual average cash balance during the year because it only uses two days as data points...”
Since GiveWell is a grant-making organization that’s a very dangerous assumption to make (and their 990 form shows them making ~$30m in grants over the course of the year).
GiveWell is used as an example—this analysis is meant to highlight the methodology, including its shortcomings, not be a deep dive into GiveWell itself. I don’t have the data needed to estimate this better. Neither you nor I know how much money GiveWell actually holds during other times of the year. They receive funds year-round and need reserves on hand to pay operational expenses.
You acknowledge this in the next section, but then propose an equally questionable workaround
That is not correct, the workaround is, as I state in the article, meant to account for the confounding effects of money made from investments rather than cash holdings by only looking at the cash balance we know isn’t earning any interest. This “workaround” has nothing to do with the cash balance issue you mentioned.
The same page shows that funds received in the final three months of 2019 were substantially greater than those received in the first nine months of 2019, which is another reason that the end-of-2019 cash balance paints a misleading picture
Thanks for pointing this out. I didn’t look more closely into GiveWell’s specifics because GiveWell is used as an example of a methodology that generalizes to other charities as I stated in the article. Data like this is very difficult to correlate with an organization’s cash balance. We don’t know what percentage of GiveWell’s cash balance the Maximum Impact Fund is, what the Maximum Impact Fund balance is after a grant is made, whether the “amount” column reflects all donor inflows during a specified period or also granting from reserves or other revenue sources, etc.
The year you’ve chosen for your analysis happens to be the year with the highest cash rates in the past 12 years.
The most recent data is most relevant to GiveWell’s current and future opportunity costs.
This isn’t acknowledged, and you don’t backtest your strategy in other years (e.g. all the years where rates were ~0%).
That is a good point! Time did not permit, but as I stated in my reply to Rob, I do think that “to get savings accounts yields themselves, we might need to use a third-party service or design a benchmark ourselves. Given the current situation with the federal funds rate, it would make sense for us to find a better proxy for good savings accounts yields in 2020 so that we can better estimate the opportunity cost for 2020 once the 2020 Form 990s come out as well as use the rate as a point of reference for current and future expected earnings.”
So it is a good idea to use an alternative metric to the federal funds rate, particularly for lower interest rate years, so they more accurately reflect the opportunity cost. Off the top of my head I think that rates were around 1% (see Ally Bank’s 2015 and 2016 historical yields when the fed funds rate was 0.13% and 0.39% respectively), which is the exact same rate I used for the forward-looking estimate for GiveWell. Using savings yields from one bank or a blend of banks instead still wouldn’t change the article’s conclusions, given that those rates would be both higher than the federal funds rate and also significant enough to warrant consideration even in a zero interest rate environment.
Potential returns of other investment options
There’s nothing about that asset that makes its one year backward-looking return able to be extrapolated. Indeed this is basically the equivalent of picking a different asset that you know now performed well over a one year period and using that to estimate opportunity cost. Why not Bitcoin in 2017 or Tesla stock in 2020?
Exactly, why wouldn’t I pick Bitcoin or Tesla? Because both ultra-short-term bonds and a standard conservative risk portfolio are extremely common investment portfolios for more risk averse investors such as many nonprofits who require a steady rate of return with minimal drawdowns. It doesn’t matter what examples you use, low-risk options would have performed well during this time period and other time periods, just worse than riskier options.
Citing the one year, backward looking return of JPST is misleading
And another reason I used JPST as an example is because I recommend JPST in my early-2019 article before I could see its historical performance. So I’m not sure if “backward looking” is the right verbiage.
And re citing a year of drawdown data—the fund had a drawdown in March 2020! If you look at a similar index with a longer timeseries you’ll see there have been drawdowns, even against the backdrop of a period of secularly falling interest rates that juiced bond returns.
I already state in the article that I am quoting month-to-month drawdown data, not day-to-day data. This means the fund did not decline in value month by month. The highest historical month-to-month drawdown was −1.72% and it recovered in two months. Hardly a volatile investment.
This analysis suffers from the same problem. At least there’s an attempt to look at a window longer than one year, but again it’s a cherry-picked period of 20 years, and it’s backward looking returns data implied to be forward-looking. You also mention the drawdown in 2018. What was the maximum drawdown in 2008 to 2009?Given the organization is granting funds a couple of months after receiving them, it has little ability to tolerate drawdowns, making comparisons to a higher risk fund incorrect to misleading.
This analysis suffers from the same problem. At least there’s an attempt to look at a window longer than one year, but again it’s a cherry-picked period of 20 years, and it’s backward looking returns data implied to be forward-looking. You also mention the drawdown in 2018. What was the maximum drawdown in 2008 to 2009?
Given the organization is granting funds a couple of months after receiving them, it has little ability to tolerate drawdowns, making comparisons to a higher risk fund incorrect to misleading.
I did not cherry pick the time period, stocks and bonds have generally performed well across time. Portfolio Visualizer has data starting from 1987, which indicates an annual return even higher than what I quoted, at 7.07%. Maximum drawdown is −8.49% from the 2008 crash. Exactly why this is considered a standard conservative portfolio by financial professionals.
As we can see, the drawdown is very low, and the expected value is positive and significant. For granted funds that GiveWell cannot afford to have a drawdown on (this does not represent all of their funds), then a “zero-risk” option like a savings account would make more sense. It all depends on the charity. Investment examples are for illustrative purposes, I have no idea what approaches GiveWell can or cannot take in actuality, beyond a savings account, which it should be able to do.
Conclusion and staff time
As shown above, getting a counterfactual impact number in the millions of dollars is not “very likely… regardless of the estimation method”. That statement is somewhere between incorrect and misleading. For instance if the cash balance is 10% of what you cite, and the interest rate is more like 0.50%, it doesn’t hold up.
That depends on what numbers we’re using. I think 10% is very low. As I’ve said, the Maximum Impact Fund cannot be used as a reliable estimate or predictor of GiveWell’s average daily cash balance. But if we assume just the fund itself represents all of GiveWell’s money, we can see that there’s a 3-month grant collection period (averages to 1.5 months of cash at 100%) plus a 2–3 month delay in granting (averages to 2.5 months of cash at 100%) which would suggest a minimum percentage of at least 33% (4/12 months) just based on the money collected during the last three months of the year.
The staff time estimate is also somewhere between incorrect and misleading:If it’s just opening bank accounts, then the impact is GiveWell on average earning roughly the prevailing bank interest rate (close to zero for much of the relevant period)If instead you want them (without the benefit of hindsight that you employ) picking actively managed JP Morgan bond funds, or creating and balancing a bond + stock fund, then I’d hazard it’s more than two hours of work per year!So either it’s not much effort, and not much return, or there can be higher returns, but clearly it’s more than 2 hours a year of effort.
The staff time estimate is also somewhere between incorrect and misleading:
If it’s just opening bank accounts, then the impact is GiveWell on average earning roughly the prevailing bank interest rate (close to zero for much of the relevant period)
If instead you want them (without the benefit of hindsight that you employ) picking actively managed JP Morgan bond funds, or creating and balancing a bond + stock fund, then I’d hazard it’s more than two hours of work per year!
So either it’s not much effort, and not much return, or there can be higher returns, but clearly it’s more than 2 hours a year of effort.
This is probably evident from the rest of my replies, but a 1% account savings rate or even 0.5% still produces significant financial returns. These numbers should be used instead of 0%.
And it would take a very small amount of time to keep funds in one fund for multiple years, whether that’s JPST or a multi-asset fund that has, say, a 20% stock allocation and 80% bond allocation. Vanguard, a well-known and trusted provider, has a great ultra-short-term bond fund as a JPST alternative. A nonprofit might not maximize returns by shifting between funds on the reg (which is, as you say, time consuming, although Antigravity Investments is more than happy to help with that), but evidently they will do much better than nothing. Many nonprofits self-manage their investment approach.