Regarding to your point about Cost-effectiveness estimates. Your other objections to my article follow a similar pattern and do not address the substantive points that I raise (I invite the reader to check for themselves).
2. Cost-effectiveness estimates
Given the previous debate that had occurred between us on how to think and talk about cost-effectiveness estimates, and the mistakes I had made in this regard, I wanted to be sure that I was presenting these estimates in a way that those at GiveWell would be happy with. So I asked an employee of GiveWell to look over the relevant parts of the manuscript of DGB before it was published; in the end five employees did so, and they were happy with how I presented GiveWell’s views and research.
How can that fact be reconciled with the quotes you give in your blog post? It’s because, in your discussion, you conflate two quite different issues: (i) how to represent that cost-effectiveness estimates provided by DCP2, or by single studies; (ii) how to represent the (in my view much more rigorous) cost-effectiveness estimates provided by GiveWell. Almost all the quotes from Holden that you give are about (i). But the quotes you criticise me for are about (ii). So, for example, when I say ‘these estimates’ are order of magnitude estimates that’s referring to (i), not to (ii).
My reasoning regarding cost-effectiveness estimates on that page is as follows (I invite the reader to check it):
1. Quote from DGB that shows that you refer to GiveWell’s AMF cost-effectiveness estimates as to “most rigorous” (that does not show much by itself, aside from the fact that it is very strange to write “most rigorous” when GiveWell’s page specifically refers to the “significant uncertainty”)
2. Quote from GW that says:
As a general note on the limitations to this kind of cost-effectiveness analysis, we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them.
3. Three quotes from DGB, which demonstrate that you interpret the GW AMF cost-effectiveness estimate literally. In the first two you write about “five hundred times” the benefit on the basis of these estimates. In the third quote you simply cite the one hundred dollars per QALY number, which does not show much by itself, and which I should not have included. Nonetheless, in the first two quotes I show that you interpret GW AMF cost-effectiveness estimates literally.
4. On the basis of these quote I conclude that you misquote GiveWell. Then I ask a question: can I be sure that GW and I mean the same thing by “the literal interpretation” of a cost-effectiveness estimate?
5. I provide quotes from Holden that demonstrate that we mean the same thing by it. In one of the quotes, Holden writes that your 100 times argument (based there on DCP2 deworming estimate) seems to mean that you interpret cost-effectiveness estimates literally.
These 5 steps constitute my argument for your misinterpretation of GW AMF cost-effectiveness estimates.
You do not address this argument in your comment.
technical edit: conflation of deworming and AMF estimates
You write:
How can that fact be reconciled with the quotes you give in your blog post? It’s because, in your discussion, you conflate two quite different issues: (i) how to represent that cost-effectiveness estimates provided by DCP2, or by single studies; (ii) how to represent the (in my view much more rigorous) cost-effectiveness estimates provided by GiveWell. Almost all the quotes from Holden that you give are about (i). But the quotes you criticise me for are about (ii). So, for example, when I say ‘these estimates’ are order of magnitude estimates that’s referring to (i), not to (ii).
If the reader takes their time and looks at the Web Archive link I provided they will see that I do not conflate these estimates. However, it is true that I did conflate them previously: in a confidential draft of the post that I sent to one of the CEA’s employees asking to look at the post prior to its publication and which I requested to not be shared with anyone besides that specific employee I did conflate them (in the end that employee declined to review my draft). I jumped from deworming estimates to AMF estimates in that draft. This fact was pointed out to me by one of my friends and I fixed it prior to the publication.
Edit: besides that CEA employee, I also shared the draft with several of my friends (also asking not to share it with anybody), so I cannot be sure to which exactly version of the post you are replying.
In your comment you write:
But the quotes you criticise me for are about (ii). So, for example, when I say ‘these estimates’ are order of magnitude estimates that’s referring to (i), not to (ii).
As if I quoted you saying something about order of magnitude estimates. I did—in that confidential draft. Again, I invite the reader to check the first public version of my essay archived by Internet Archive and to check whether I provided any quotes where William talks about order of magnitude estimates.
You write:
(Also, Alexey’s post keeps changing, so if it looks like I’m responding to something that’s no longer there, that’s why.)
I did update the essay after the first publication. However, the points you’re responding to here were removed before my publication of the essay. I am not sure why you are responding to the confidential draft.
Edit2: Here is the draft I’m referring to. Please note its status as a draft and that I did not intend it to be seen by public. It contains strong language and a variety of mistakes.
If you CTRL+F “orders of magnitude” in this draft, you will find the quote William refers to.
I wonder why my reply has so many downvotes (-8 score) and no replies. This could of course indicate that my arguments are so bad that they’re not worth engaging with, but the fact that many of the members of the community find my criticism accurate and valuable, this seems unlikely.
As a datapoint, I thought that your reply was so bad that it was not worth engaging in, although I think you did find a couple of inaccuracies in DGB and appreciate the effort you went to. I’ll briefly explain my position.
I thought MacAskill’s explanations were convincing and your counter-argument missed his points completely, to the extent that you seem to have an axe to grind with him. E.g. if GiveWell is happy with how their research was presented in DGB (as MacAskill mentioned), then I really don’t see how you, as an outsider and non-GW representative, can complain that their research is misquoted without having extremely strong evidence. You do not have extremely strong evidence. Even if you did, there’s still the matter that GW’s interpretation of their numbers is not necessarily the only reasonable one (as Jan_Kulveit points out below).
You completely ignored MacAskill’s convincing counter-arguments while simultaneously accusing him of ignoring the substance your argument, so it seemed to me that there was little point in debating it further with you.
Regarding to your point about Cost-effectiveness estimates. Your other objections to my article follow a similar pattern and do not address the substantive points that I raise (I invite the reader to check for themselves).
You do not address my concern here. Here’s the first Web Archive version of the post
My reasoning regarding cost-effectiveness estimates on that page is as follows (I invite the reader to check it):
1. Quote from DGB that shows that you refer to GiveWell’s AMF cost-effectiveness estimates as to “most rigorous” (that does not show much by itself, aside from the fact that it is very strange to write “most rigorous” when GiveWell’s page specifically refers to the “significant uncertainty”)
2. Quote from GW that says:
3. Three quotes from DGB, which demonstrate that you interpret the GW AMF cost-effectiveness estimate literally. In the first two you write about “five hundred times” the benefit on the basis of these estimates. In the third quote you simply cite the one hundred dollars per QALY number, which does not show much by itself, and which I should not have included. Nonetheless, in the first two quotes I show that you interpret GW AMF cost-effectiveness estimates literally.
4. On the basis of these quote I conclude that you misquote GiveWell. Then I ask a question: can I be sure that GW and I mean the same thing by “the literal interpretation” of a cost-effectiveness estimate?
5. I provide quotes from Holden that demonstrate that we mean the same thing by it. In one of the quotes, Holden writes that your 100 times argument (based there on DCP2 deworming estimate) seems to mean that you interpret cost-effectiveness estimates literally.
These 5 steps constitute my argument for your misinterpretation of GW AMF cost-effectiveness estimates.
You do not address this argument in your comment.
technical edit: conflation of deworming and AMF estimates
You write:
If the reader takes their time and looks at the Web Archive link I provided they will see that I do not conflate these estimates. However, it is true that I did conflate them previously: in a confidential draft of the post that I sent to one of the CEA’s employees asking to look at the post prior to its publication and which I requested to not be shared with anyone besides that specific employee I did conflate them (in the end that employee declined to review my draft). I jumped from deworming estimates to AMF estimates in that draft. This fact was pointed out to me by one of my friends and I fixed it prior to the publication.
Edit: besides that CEA employee, I also shared the draft with several of my friends (also asking not to share it with anybody), so I cannot be sure to which exactly version of the post you are replying.
In your comment you write:
As if I quoted you saying something about order of magnitude estimates. I did—in that confidential draft. Again, I invite the reader to check the first public version of my essay archived by Internet Archive and to check whether I provided any quotes where William talks about order of magnitude estimates.
You write:
I did update the essay after the first publication. However, the points you’re responding to here were removed before my publication of the essay. I am not sure why you are responding to the confidential draft.
Edit2: Here is the draft I’m referring to. Please note its status as a draft and that I did not intend it to be seen by public. It contains strong language and a variety of mistakes.
If you CTRL+F “orders of magnitude” in this draft, you will find the quote William refers to.
I wonder why my reply has so many downvotes (-8 score) and no replies. This could of course indicate that my arguments are so bad that they’re not worth engaging with, but the fact that many of the members of the community find my criticism accurate and valuable, this seems unlikely.
As a datapoint, I thought that your reply was so bad that it was not worth engaging in, although I think you did find a couple of inaccuracies in DGB and appreciate the effort you went to. I’ll briefly explain my position.
I thought MacAskill’s explanations were convincing and your counter-argument missed his points completely, to the extent that you seem to have an axe to grind with him. E.g. if GiveWell is happy with how their research was presented in DGB (as MacAskill mentioned), then I really don’t see how you, as an outsider and non-GW representative, can complain that their research is misquoted without having extremely strong evidence. You do not have extremely strong evidence. Even if you did, there’s still the matter that GW’s interpretation of their numbers is not necessarily the only reasonable one (as Jan_Kulveit points out below).
You completely ignored MacAskill’s convincing counter-arguments while simultaneously accusing him of ignoring the substance your argument, so it seemed to me that there was little point in debating it further with you.
I guess this is a valid point of view. Just in case, I emailed GiveWell about this issue.
see edit above
see edit above