Thank you very much for taking the time to write this detailed reply, Michael! I haven’t read the To WELLBY or not to WELLBY? post, but definitely want to check that out to understand this all better.
I also want to apologize for my language sounding overly critical/harsh in my previous comment. E.g. Making my first sentence “This post didn’t address my concerns related to using WELLBYs...” when I knew full well that wasn’t what the post was intending to address was very unfair of me.
I know you’ve put a lot of work into researching the WELLBY approach and are no doubt advancing our frontier of knowledge of how to do good effectively in the process, so I want to acknowledge that I appreciate what you do regardless of any level of skepticism I may still have related to heavily relying on WELLBY measurements as the best way to evaluate impact.
As a final note, I want to clarify that while my previous comment may have made it sound like I was confident that the WELLBY approach was no good, in fact my tone was more reflective of my (low-information) intuitive independent impression, not my all-things-considered view. I think there’s a significant chance that when I read into your research on neutrality, linearity, and comparability, etc, that I’ll update toward thinking that the WELLBY approach makes considerably more sense than I initially assumed.
Thanks for saying that. Yeah, I couldn’t really understand where you were coming from (and honestly ended up spending 2+ hours drafting a reply).
On reflection, we should probably have done more WELLBY-related referencing in the post, but we were trying to keep the academic side light. In fact, we probably need to recombine our various scratching on the WELLBY and put them onto a single page on our website—it’s been a lower priority than doing the object-level charity analysis work.
If you’re doing the independent impression thing again, then, as a recipient, it would have been really helpful to know that. Then I would have read it more as a friendly “I’m new to this and sceptical and X and Y—what’s going on with those?” and less as a “I’m sceptical, you clearly have no idea what you’re talking about” (which was more-or-less how I initially interpreted it… :) )
Then I would have read it more as a friendly “I’m new to this and sceptical and X and Y—what’s going on with those?” and less as a “I’m sceptical, you clearly have no idea what you’re talking about”
Ah, I’m really sorry I didn’t clarify this!
For the record, you’re clearly an expert on WELLBYs and I’m quite new to thinking about them.
My initial exposure to HLI’s WELLBY approach to evaluating interventions was the post Measuring Good Better and this post is only my second time reading about WELLBYs. I also know very little about subjective wellbeing surveys. I’ve been asked to report my subjective wellbeing on surveys before, but I’ve basically never read about them before besides that chapter of WWOTF.
The rest of this comment is me offering an explanation on what I think happened here:
Scott Alexander has a post called Socratic Grilling that I think offers useful insight into our exchange. In particular, while I absolutely could and should have written my initial comment to be a lot friendlier, I think my comment was essentially an all-at-once example of Socratic grilling (me being the student and you being the teacher). As Scott points out, there’s a known issue with this:
Second, to a hostile observer, it would sound like the student was challenging the teacher. Every time the teacher tried to explain germ theory, the student “pounced” on a supposed inconsistency. When the teacher tried to explain the inconsistency, the student challenged her explanations. At times he almost seems to be mocking the teacher. Without contextual clues – and without an appreciation for how confused young kids can be sometimes – it could sound like this kid is an arrogant know-it-all who thinks he’s checkmated biologists and proven that germ theory can’t possibly be true. Or that he thinks that he, a mere schoolchild, can come up with a novel way to end all sickness forever that nobody else ever thought of.
Later:
Tolerating this is harder than it sounds. Most people can stay helpful for one or two iterations. But most people are bad at explaining things, so one or two iterations isn’t always enough I’ve had times when I need five or ten question-answer rounds with a teacher in order to understand what they’re telling me. The process sounds a lot like “The thing you just said is obviously wrong”…”no, that explanation you gave doesn’t make sense, you’re still obviously wrong”…”you keep saying the same thing over and over again, and it keeps being obviously wrong”…”no, that’s irrelevant to the point that’s bothering me”…”no, that’s also irrelevant, you keep saying an obviously wrong thing”…”Oh! That word means something totally different from what I thought it meant, now your statement makes total sense.”
But it’s harder even than that. Sometimes there is a vast inferential distance between you and the place where your teacher’s model makes sense, and you need to go through a process as laborious as converting a religious person to a materialist worldview (or vice versa) before the gap gets closed.
When I first read about HLI’s approach in the Measuring Good Better article my reaction was “Huh, this seems like a poor way to evaluate impact given [all the aspects of subjective wellbeing surveys that intuitively seemed problematic to me].”
If I was talking with you in person about it I probably would have done a back-and-forth Socratic grilling with you about it. But I didn’t comment. I then got to this post some weeks later and was hoping it would provide some answer to my concerns, was disappointed that that was not the post’s purpose, and proceeded to write a long post explaining all my concerns with the WELLBY approach so that you or someone could address them. In short, I dumped a lot of work on you and completely failed to think about how (Scott’s words:) ” it would sound like the student was challenging the teacher,” and how I could come across as an “arrogant know-it-all who thinks he’s checkmated” you, and how “Tolerating this is harder than it sounds”.
So I’m really sorry about that and will make it a point to make sure I actually think about how my comments will be received next time I’m tempted to “Socratically grill” someone, that way I can make sure my comment comes across as friendly.
Thank you very much for taking the time to write this detailed reply, Michael! I haven’t read the To WELLBY or not to WELLBY? post, but definitely want to check that out to understand this all better.
I also want to apologize for my language sounding overly critical/harsh in my previous comment. E.g. Making my first sentence “This post didn’t address my concerns related to using WELLBYs...” when I knew full well that wasn’t what the post was intending to address was very unfair of me.
I know you’ve put a lot of work into researching the WELLBY approach and are no doubt advancing our frontier of knowledge of how to do good effectively in the process, so I want to acknowledge that I appreciate what you do regardless of any level of skepticism I may still have related to heavily relying on WELLBY measurements as the best way to evaluate impact.
As a final note, I want to clarify that while my previous comment may have made it sound like I was confident that the WELLBY approach was no good, in fact my tone was more reflective of my (low-information) intuitive independent impression, not my all-things-considered view. I think there’s a significant chance that when I read into your research on neutrality, linearity, and comparability, etc, that I’ll update toward thinking that the WELLBY approach makes considerably more sense than I initially assumed.
Hello William,
Thanks for saying that. Yeah, I couldn’t really understand where you were coming from (and honestly ended up spending 2+ hours drafting a reply).
On reflection, we should probably have done more WELLBY-related referencing in the post, but we were trying to keep the academic side light. In fact, we probably need to recombine our various scratching on the WELLBY and put them onto a single page on our website—it’s been a lower priority than doing the object-level charity analysis work.
If you’re doing the independent impression thing again, then, as a recipient, it would have been really helpful to know that. Then I would have read it more as a friendly “I’m new to this and sceptical and X and Y—what’s going on with those?” and less as a “I’m sceptical, you clearly have no idea what you’re talking about” (which was more-or-less how I initially interpreted it… :) )
Ah, I’m really sorry I didn’t clarify this!
For the record, you’re clearly an expert on WELLBYs and I’m quite new to thinking about them.
My initial exposure to HLI’s WELLBY approach to evaluating interventions was the post Measuring Good Better and this post is only my second time reading about WELLBYs. I also know very little about subjective wellbeing surveys. I’ve been asked to report my subjective wellbeing on surveys before, but I’ve basically never read about them before besides that chapter of WWOTF.
The rest of this comment is me offering an explanation on what I think happened here:
Scott Alexander has a post called Socratic Grilling that I think offers useful insight into our exchange. In particular, while I absolutely could and should have written my initial comment to be a lot friendlier, I think my comment was essentially an all-at-once example of Socratic grilling (me being the student and you being the teacher). As Scott points out, there’s a known issue with this:
Later:
When I first read about HLI’s approach in the Measuring Good Better article my reaction was “Huh, this seems like a poor way to evaluate impact given [all the aspects of subjective wellbeing surveys that intuitively seemed problematic to me].”
If I was talking with you in person about it I probably would have done a back-and-forth Socratic grilling with you about it. But I didn’t comment. I then got to this post some weeks later and was hoping it would provide some answer to my concerns, was disappointed that that was not the post’s purpose, and proceeded to write a long post explaining all my concerns with the WELLBY approach so that you or someone could address them. In short, I dumped a lot of work on you and completely failed to think about how (Scott’s words:) ” it would sound like the student was challenging the teacher,” and how I could come across as an “arrogant know-it-all who thinks he’s checkmated” you, and how “Tolerating this is harder than it sounds”.
So I’m really sorry about that and will make it a point to make sure I actually think about how my comments will be received next time I’m tempted to “Socratically grill” someone, that way I can make sure my comment comes across as friendly.