I guess this is a valid point of view. Just in case, I emailed GiveWell about this issue.
guzey
Karma: 475
CN: I don’t agree with you
PlayPumps: I don’t agree with your assessment of points 1, 2, 4.
At this point, I really don’t think you can justifiably continue to hold your either of your positions: that DGB is significantly inaccurate, or that MacAskill is dishonest. I really do believe that you’re in this in good faith, and that your main error (save the ad hominem attack, likely a judgement error) was in not getting to the bottom of these questions. But now the questions feel very well resolved. Unless the four issues listed above constitute systemic inaccuracy, I really don’t see an argument for it.
Sincerely, thank you for engaging, and if you find these arguments correct, I hope you’ll uphold our value of honesty and apologize to MacAskill for the ad hominem attacks, as well as give him a kinder, more accurate explanation of his inaccuracies. I hope I’ve helped.
I have already apologized to MacAskill for the first, even harsher, version of the post. I will certainly apologize to him, if I conclude that the arguments he made were not made in bad faith, but at this point I find that my central point stands.
As I wrote in another comment, thank you for your time and I will let you know later about my conclusions. I will likely rewrite the post after this.
Thank you a ton for the time and effort you put into this. I find myself disagreeing with you, but this may reflect my investment in my arguments. I will write to you later, once I reflect on this further.
I never posted the draft that had this quote on EA Forum. Further, I clearly asked everyone I sent the drafts not to share them with anybody.
Also, I wonder what you think about the second half of this comment of mine in this thread.
There, I point out that MacAskill responds not to any of the published versions of the essay but to a confidential draft (since he says that I’m quoting him on something that I only quoted him about in a draft).
What do you think about it? Is my interpretation here plausible? What are the other plausible explanations for this? Maybe I fail to see charitable interpretations of how that happened.
Thank you for a thoughtful response.
1. Deworming. Seems fair.
2. GiveWell. This seems like a good argument. I will think about it.
3. CN. If you read my post and not William’s response to it, I never accuse him of conflating CEO pay and overhead. He deflects my argument by writing about this. This is indeed a minor point.
I specifically accuse him of misquoting CN. As I wrote in other comments here, yes this might indeed be CN’s position and in the end, they would judge the doughnuts charity highly. I do not contend this point and never did. I only wrote that MacAskill (1) quotes CN, (2) makes conclusions based on this quote about CN, (3) the very page that MacAskill takes the quote from says that their position does not lead to these conclusions. And maybe CN is being completely hypocritical! This is not a point. It is still dishonest to misquote them.
4. PlayPumps: I feel like you’re kind of missing the point and I’m wondering if it might be some sort of a fundamental disagreement about unstated assumptions? I think that making dishonest argument that lead to the right conclusions is still dishonest. It seems that you (and many other EAs) feel that if the conclusion is correct, then the fact that the argument was dishonest is not so important (same as with CN). Here’s what you say:
But this is popular writing, and he really doesn’t have space to fully review all the ins and outs of PlayPumps.
And here’s what I wrote in that comment specifically about this argument:
All of what you say seems reasonable. If Doing Good Better was just a popular book—I would not care about all of this stuff. But this book serves as an introduction to Effective Altruism and the whole premise of the book is that it’s objective and uses evidence to arrive to conclusions, etc, and advocates and evidence-based approach to philanthropy. And, although I don’t consider myself EA, a lot of my friends do, and I care about the movement. …
So we cannot judge the book as we would any other popular book where the author has a narrative and peppers it with random studies they found. I’m not so bothered by the misrepresentations per se but by the hypocrisy. …
just in the Introduction, William first trashes PlayPumps (not saying a single good word about them and very liberally exaggerating his sources) and then praises deworming almost as a salvation. And again, this is entirely natural for a popular book—but not for a book that introduces Effective Altruism and evidence-based approach to philanthropy. …
3. MacAskill:
According to the UNICEF report, children sometimes fell off and broke limbs, and some vomited from the spinning. [emphasis mine]
UNICEF report:
Some users reported that children had fallen off and been injured with bruises and cuts, and in one case a child fractured their arm. [emphasis mine]
This is a very good example of a point I’m making—of course a popular book will exaggerate things like that. But again—not a book that advocates an even-handed, evidence-based approach to philanthropy.
And in your conclusion you write:
Except perhaps the deworming example, none of these inaccuracies, if corrected, would change anything important about the conclusions of the book.
Yes! I mostly agree with this! But (1) these are not just inaccuracies. I point out misrepresentations. (2) I believe that making dishonest arguments that advance the right conclusions is dishonest.
Do I understand you correctly that you disagree with me on point (2)?
This seems like a good argument. Thank you. I will think about it.
Hi smithee,
I do wonder if I should’ve written this post in a less personal tone. I will consider writing a follow up to it.
About me deciding that MacAskill is deliberately misleading, please see my comment in /r/slatestarcodex in response to /u/scottalexander about it. Would love to know what you think.
About cost-effectiveness estimates: I don’t think your interpretation is plausible. The GiveWell page that gives the $3400 estimate, specifically asks not to interpret it literally.
About me deciding that MacAskill is deliberately misleading. Please see my comment in /r/slatestarcodex in response to /u/scottalexander about it. Would love to know what you think.
I wonder why my reply has so many downvotes (-8 score) and no replies. This could of course indicate that my arguments are so bad that they’re not worth engaging with, but the fact that many of the members of the community find my criticism accurate and valuable, this seems unlikely.
Hi Jan,
Thanks for the feedback.
You write:
In some of the examples, it seems adding more caveats and reporting in more detail would have been better for readers interested in precision.
I should point out that in the post I show not just a lack of caveats and details. William misrepresents the evidence. Among other things, he:
cherry picks the variables from a deworming paper he cites
interprets GW’s AMF estimate in a way they specifically asked not to interpret them (“five hundred times” more effective thing — Holden wrote specifically about such arguments that they seem to require taking cost-effectiveness estimates literally)
quotes two sentences from Charity Navigator’s site when the very next sentence shows that the interpretation of the previous sentences is wrong
In a long response William posted here, he did not address any of these points:
he doesn’t mention cherry picking (and neither does his errata page)
he doesn’t mention the fact that GiveWell asked not to interpret their AMF estimate literally
and he writes “I represent CN fairly, and make a fair criticism of its approach to assessing charities.”, which may be true about some general CN’s position, but which has nothing to do with misquoting Charity Navigator.
If the issue was just a lack of detail, of course I would not have written the post in such a tone. Initially, I considered simply emailing him a list of mistakes that I found, but as I mentioned in the post, the volume and egregiousness of misrepresentations lead me to conclude that he argued in bad faith.
edit: I will email GiveWell to clarify what they think about William making claims about 500 times more benefit on the basis of their AMF estimate.
Hi Gregory,
I should point out that
the essay posted to the Effective Altruism Forum never contained the bit about disavowing Will. I did write this in the version that I posted on my site, and I removed it, after much feedback elsewhere and wrote:
I updated this post significantly, based on feedback from the community. Several of my points were wrong and my tone and conclusions were sometimes inappropriate. I believe that my central point stands, but I apologize to William MacAskill for the first versions of the essay. For previous versions please see Web Archive.
As I wrote in a comment above responding to Will, prior to the publication of my essay I reached out to one of the employees of the CEA and asked them to review my draft. They first agreed, but after I sent the draft, they declined to review it.
see edit above
see edit above
Regarding to your point about Cost-effectiveness estimates. Your other objections to my article follow a similar pattern and do not address the substantive points that I raise (I invite the reader to check for themselves).
2. Cost-effectiveness estimates
Given the previous debate that had occurred between us on how to think and talk about cost-effectiveness estimates, and the mistakes I had made in this regard, I wanted to be sure that I was presenting these estimates in a way that those at GiveWell would be happy with. So I asked an employee of GiveWell to look over the relevant parts of the manuscript of DGB before it was published; in the end five employees did so, and they were happy with how I presented GiveWell’s views and research.
How can that fact be reconciled with the quotes you give in your blog post? It’s because, in your discussion, you conflate two quite different issues: (i) how to represent that cost-effectiveness estimates provided by DCP2, or by single studies; (ii) how to represent the (in my view much more rigorous) cost-effectiveness estimates provided by GiveWell. Almost all the quotes from Holden that you give are about (i). But the quotes you criticise me for are about (ii). So, for example, when I say ‘these estimates’ are order of magnitude estimates that’s referring to (i), not to (ii).
...
You do not address my concern here. Here’s the first Web Archive version of the post
My reasoning regarding cost-effectiveness estimates on that page is as follows (I invite the reader to check it):
1. Quote from DGB that shows that you refer to GiveWell’s AMF cost-effectiveness estimates as to “most rigorous” (that does not show much by itself, aside from the fact that it is very strange to write “most rigorous” when GiveWell’s page specifically refers to the “significant uncertainty”)
2. Quote from GW that says:
As a general note on the limitations to this kind of cost-effectiveness analysis, we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them.
3. Three quotes from DGB, which demonstrate that you interpret the GW AMF cost-effectiveness estimate literally. In the first two you write about “five hundred times” the benefit on the basis of these estimates. In the third quote you simply cite the one hundred dollars per QALY number, which does not show much by itself, and which I should not have included. Nonetheless, in the first two quotes I show that you interpret GW AMF cost-effectiveness estimates literally.
4. On the basis of these quote I conclude that you misquote GiveWell. Then I ask a question: can I be sure that GW and I mean the same thing by “the literal interpretation” of a cost-effectiveness estimate?
5. I provide quotes from Holden that demonstrate that we mean the same thing by it. In one of the quotes, Holden writes that your 100 times argument (based there on DCP2 deworming estimate) seems to mean that you interpret cost-effectiveness estimates literally.
These 5 steps constitute my argument for your misinterpretation of GW AMF cost-effectiveness estimates.
You do not address this argument in your comment.
technical edit: conflation of deworming and AMF estimates
You write:
How can that fact be reconciled with the quotes you give in your blog post? It’s because, in your discussion, you conflate two quite different issues: (i) how to represent that cost-effectiveness estimates provided by DCP2, or by single studies; (ii) how to represent the (in my view much more rigorous) cost-effectiveness estimates provided by GiveWell. Almost all the quotes from Holden that you give are about (i). But the quotes you criticise me for are about (ii). So, for example, when I say ‘these estimates’ are order of magnitude estimates that’s referring to (i), not to (ii).
If the reader takes their time and looks at the Web Archive link I provided they will see that I do not conflate these estimates. However, it is true that I did conflate them previously: in a confidential draft of the post that I sent to one of the CEA’s employees asking to look at the post prior to its publication and which I requested to not be shared with anyone besides that specific employee I did conflate them (in the end that employee declined to review my draft). I jumped from deworming estimates to AMF estimates in that draft. This fact was pointed out to me by one of my friends and I fixed it prior to the publication.
Edit: besides that CEA employee, I also shared the draft with several of my friends (also asking not to share it with anybody), so I cannot be sure to which exactly version of the post you are replying.
In your comment you write:
But the quotes you criticise me for are about (ii). So, for example, when I say ‘these estimates’ are order of magnitude estimates that’s referring to (i), not to (ii).
As if I quoted you saying something about order of magnitude estimates. I did—in that confidential draft. Again, I invite the reader to check the first public version of my essay archived by Internet Archive and to check whether I provided any quotes where William talks about order of magnitude estimates.
You write:
(Also, Alexey’s post keeps changing, so if it looks like I’m responding to something that’s no longer there, that’s why.)
I did update the essay after the first publication. However, the points you’re responding to here were removed before my publication of the essay. I am not sure why you are responding to the confidential draft.
Edit2: Here is the draft I’m referring to. Please note its status as a draft and that I did not intend it to be seen by public. It contains strong language and a variety of mistakes.
If you CTRL+F “orders of magnitude” in this draft, you will find the quote William refers to.
A technical comment: neither Web Archive, nor archive.fo archive the comments to this post, so I archived this page manually. PDF from my site captured at 2018-11-17 16-48 GMT
edit: a reddit user suggested this archive of this page: http://archive.fo/jUkMB
I heavily criticize one of the founders of CEA and heavily use the words of the founder of Open Phil in my post, which lead me to believe that I need to disclose that I applied to both organizations.
In particular, the version of the essay that I initially posted here did not discuss the strength of the relationship between income and happiness in rich and poor countries—I agree that this was a weak argument.
Hi William,
Thank you for your response. I apologize for the stronger language that I used in the first public version of this post. I believe that here you do not address most of the points I made either in the first public version or in the version that was up here at the moment of your comment.
I will not change the post here without explicitly noting it, now that you have replied.
I’m in the process of preparing a longer reply to you.
Well, happens. Although if you forwarded it to Will, then he probably read the part of an email where I ask not to share it with anybody, but proceeded to read that draft and respond to a confidential draft anyway.