MichaelPlant

• MichaelPlant 10 Jan 2022 19:16 UTC

  12 points

  in reply to: lukeprog’s comment on: Democratis­ing Risk—or how EA deals with critics

  Hello Luke,

  Thanks for this too. I appreciate you’ve since moved on to other things, so this isn’t really your topic to engage on anymore. However, I’ll make two comments.

  First, you said you read various things in the area, including by me, since 2015. It would have been really helpful (to me) if, given you had different views, you had engaged at the time and set out where you disagreed and what sort of evidence would have changed your mind.

  Second, and similarly, I would really appreciate it if the current team at Open Philanthropy could more precisely set out their perspective on all this. I did have a few interactions with various Open Phil staff in 2021, but I wouldn’t say I’ve got anything like canonical answers on what their reservations are about 1. measuring outcomes in terms of SWB - Alex Berger’s recent technical update didn’t comment on this—and 2. doing more research or grantmaking into the things that, from the SWB perspective, seem overlooked.

• MichaelPlant 6 Jan 2022 23:14 UTC

  45 points

  in reply to: lukeprog’s comment on: Democratis­ing Risk—or how EA deals with critics

  Hello Luke, thanks for this, which was illuminating. I’ll make an initial clarifying comment and then go on to the substantive issues of disagreement.

  At least for me, I don’t think this is a case of an EA funder repeatedly ignoring work by e.g. Michael Plant — I think it’s a case of me following the debate over the years and disagreeing on the substance after having familiarized myself with the literature.

  I’m not sure what you mean here. Are you saying GiveWell didn’t repeatedly ignore the work? That Open Phil didn’t? Something else? As I set out in another comment, my experience with GiveWell staff was of being ignored by people who weren’t at that familiar with the relevant literature—FWIW, I don’t recall the concerns you raise in your notes being raised with me. I’ve not had interactions with Open Phil staff prior to 2021 - for those reading, Luke and I have never spoken—so I’m not able to comment regarding that.

  Onto the substantive issues. Would you be prepared to more precisely state what your concerns are, and what sort of evidence would chance your mind? Reading your comments and your notes, I’m not sure exactly what your objections are and, in so far as I do, they don’t seem like strong objections.

  You mention “weakly validated measures” as an issue but in the text you say “for some scales, reliability and validity have been firmly established”, which implies to me you think (some) scales are validated. So which scales are you worried about, to what extent, and why? Are they so non-validated we should think they contain no information? If some scales are validated, why not just use those ones? By analogy, we wouldn’t give up on measuring temperature if we thought only some of our thermometers were broken. I’m not sure if we’re even on the same page about what it is to ‘validate’ a measure of something (I can elaborate, if helpful).

  On “unconvincing intervention studies”, I take it you’re referring to your conversation notes with Sonja Lyubormirsky. The ‘happiness interventions’ you talk about are really just those from the field of ‘positive psychology’ where, basically, you take mentally healthy people and try to get them to change their thoughts and behaviours to be happier, such as by writing down what they’re grateful for. This implies a very narrow interpretation of ‘happiness interventions’. Reducing poverty or curing diseases are ‘happiness interventions’ in my book because they increase happiness, but they are certainly not positive psychology interventions. One can coherently think that subjective wellbeing measures, eg self-reported happiness, are valid and capture something morally important but deny gratitude journalling etc. are particularly promising ways, in practice, of increasing it. Also, there’s a big difference between the lab-style experiments psychologists run and the work economists tend to do looking at large panel and cohort data sets.

  Regarding “one entire literature using the wrong statistical test for decades”, again, I’m not sure exactly what you mean. Is the point about ‘item response theory’? I confess that’s not something that gets discussed in the academic world of subjective wellbeing measurement—I don’t think I’ve ever heard it mentioned. After a quick look, it seems to be a method to relate scores of psychometric tests to real-world performance. That seems to be a separate methodological ballgame from concerns about the relationship between how people feel and how they report those feelings on a numerical scale, e.g. when we ask “how happy are you, 0-10?”. Subjective wellbeing researchers do talk about the issue of ‘scale cardinality’, ie, roughly, does your “7/​10” feel the same to you as my “7/​10″ does to me? This issue has been starting to get quite a bit of attention in just the last couple of years but has, I concede, been rather neglected by the field. I’ve got a working paper on this under review which is (I think) the first comprehensive review of the problem.

  To me, it looks like in your initial investigation you had the bad luck to run into a couple of dead ends and, quite understandably given those, didn’t go further. But I hope you’ll let me try to explain further to you why I think happiness research (like happiness itself) is worth taking seriously!

• MichaelPlant 6 Jan 2022 18:23 UTC

  63 points

  in reply to: Aaron Gertler’s comment on: Democratis­ing Risk—or how EA deals with critics

  I’ve only just seen this and thought I should chime in. Before I describe my experience, I should note that I will respond to Luke’s specific concerns about subjective wellbeing separately in a reply to his comment.

  TL;DR Although GiveWell (and Open Phil) have started to take an interest in subjective wellbeing and mental health in the last 12 months, I have felt considerable disappointment and frustration with their level of engagement over the previous six years.

  I raised the “SWB and mental health might really matter” concerns in meetings with GiveWell staff about once a year since 2015. Before 2021, my experience was that they more or less dismissed my concerns, even though they didn’t seem familiar with the relevant literature. When I asked what their specific doubts were, these were vague and seemed to change each time (“we’re not sure you can measure feelings”, “we’re worried about experimenter demand effect”, etc.). I’d typically point out their concerns had already been addressed in the literature, but that still didn’t seem to make them more interested. (I don’t recall anyone ever mentioning ‘item response theory’, which Luke raises as his objection.) In the end, I got the impression that GiveWell staff thought I was a crank and were hoping I would just go away.

  GiveWell’s public engagement has been almost non-existent. When HLI published, in August 2020, a document explaining how GiveWell could (re)estimate their own ‘moral weights’ using SWB, GiveWell didn’t comment on this (a Founders Pledge researcher did, however, provide detailed comments). The first and only time GiveWell has responded publicly about this was in December 2020, where they set out their concerns in relation to our cash transfer vs therapy meta-analyses; I’ve replied to those comments (many of which expressed quite non-specific doubts) but not yet received a follow-up.

  The response I was hoping for—indeed, am still hoping for—was the one Will et al. gave above, namely, “We’re really interested in serious critiques. What do you think we’re getting wrong, why, and what difference would it make if you were right? Would you like us to fund you to work on this?” Obviously, you wouldn’t expect an organisation to engage with critiques that are practically unimportant and from non-credible sources. In this case, however, I was raising fundamental concerns that, if true, could substantially alter the priorities, both for GiveWell and EA more broadly. And, for context, at the time I initially highlighted these points I was doing a philosophy PhD supervised by Hilary Greaves and Peter Singer and the measurement of wellbeing was a big part of my thesis.

  There has been quite good engagement from other EAs and EAs orgs, as Aaron Gertler notes above. I can add to those that, for instance, Founders Pledge have taken SWB on board in their internal decision-making and have since made recommendations in mental health. However, GiveWell’s lack of engagement has really made things difficult because EAs defer so much to GiveWell: a common question I get is “ah, but what does GiveWell think?” People assume that, because GiveWell didn’t take something seriously, that was strong evidence they shouldn’t either. This frustration was compounded by the fact that because there isn’t a clear, public statement of what GiveWell’s concerns were, I could neither try to address their concerns nor placate the worries of others by saying something like “GiveWell’s objection is X. We don’t share that because of Y”.

  This is pure speculation on my part, but I wonder if GiveWell (and perhaps Open Phil too) developed an ‘ugh field’ around subjective wellbeing and mental health. They didn’t look into it initially because they were just too damn busy. But then, after a while, it became awkward to start engaging with because that would require admitting they should have done so years ago, so they just ignored it. I also suspect there’s been something of an information cascade where someone originally looked at all this (see my reply to Luke above), decided it wasn’t interesting, and then other staff members just took that on trust and didn’t revisit it—everyone knew an idea could be safely ignored even if they weren’t sure why.

  Since 2021, however, things have been much better. In late 2020, as mentioned, HLI published a blog post showing how SWB could be used to (re)estimate GiveWell’s ‘moral weights’. I understand that some of GiveWell’s donors asked them for an opinion on this and that pushed them to engage with it. HLI had a productive conversation with GiveWell in February 2021 (see GiveWell’s notes) where, curiously, no specific objections to SWB were raised. GiveWell are currently working on a blog post responding to our moral weights piece and they kindly shared a draft with us in July asking for our feedback. They’ve told us they plan to publish reports on SWB and psychotherapy in the next 3-6 months.

  Regarding Open Phil, it seemed pointless to engage unless GiveWell came on board, because Open Phil also defer strongly to GiveWell’s judgements, as Alex Berger has recently stated. However, we recently had some positive engagement from Alex on Twitter, and a member of his team contacted HLI for advice after reading our report and recommendations on global mental health. Hence, we are now starting to see some serious engagement, but it’s rather overdue and still less fulsome than I’d want.

  What links here?

  • MichaelPlant's comment on Democratis­ing Risk—or how EA deals with critics by CarlaZoeC (6 Jan 2022 23:14 UTC; 45 points)

• MichaelPlant 10 Dec 2021 12:32 UTC

  2 points

  in reply to: MichaelPlant’s comment on: Donat­ing money, buy­ing hap­piness: new meta-analy­ses com­par­ing the cost-effec­tive­ness of cash trans­fers and psy­chother­apy in terms of sub­jec­tive well-being

  We would guess that if we, for example, gave households the choice between 3-4 months of therapy and a $1,000 transfer, many more would choose the transfer.

  To clarify, the intervention is not to provide therapy to anyone, it’s just to provide it to those who are depressed. I expect that even some depressed people would choose cash over therapy. But it’s reasonable to assume people don’t always know what’s best for them and under/​overconsume on certain goods due to lack of information, etc. That’s why we need studies to see what truly improves people’s subjective well-being.

  If one was serious about always giving people what they choose, then you would just give people cash and let them decide. Given that GiveWell claims that bednets and deworming are better than cash, it seems they already accept cash is not necessarily best. Hence, it’s unclear how they could raise this as a problem for therapy without being inconsistent.

• MichaelPlant 10 Dec 2021 12:29 UTC

  2 points

  in reply to: MichaelPlant’s comment on: Donat­ing money, buy­ing hap­piness: new meta-analy­ses com­par­ing the cost-effec­tive­ness of cash trans­fers and psy­chother­apy in terms of sub­jec­tive well-being

  Third, it seems intuitively surprising that a $1,000 cash transfer for a household in a low-income country would have a substantially smaller effect than 3-4 months of therapy (0.5 SD for cash transfers vs. 1.6 SD for therapy in HLI’s study).

  What I think might have been overlooked here is that therapy is only being given to people diagnosed with mental illnesses, but the cash transfers go to poor people in general (only some of whom will be depressed). Hence, it’s perhaps not so surprising that directly treating the depression of depressed people is more impactful than giving out money (even if those people are poor). If you were in pain but also poor, no one would assume that giving you money would do more for your happiness than morphine would.

• MichaelPlant 10 Dec 2021 12:28 UTC

  2 points

  in reply to: MichaelPlant’s comment on: Donat­ing money, buy­ing hap­piness: new meta-analy­ses com­par­ing the cost-effec­tive­ness of cash trans­fers and psy­chother­apy in terms of sub­jec­tive well-being

  Second, we’ve taken a very shallow look at a recent trial that compares cash transfers with therapy head-to-head. This trial finds a much smaller effect of therapy vs. cash.

  We account for this trial in our meta-analysis—if we hadn’t incorporated it, therapy would look even a bit more cost-effective. Of course, the point of meta-analyses is to look at the whole evidence base, rather than just selecting one or two pieces of evidence; one could discount any meta-analysis this way by pointing to the trial with the lowest effect.

  We don’t think one study should overshadow the results of a meta-analysis, which aggregates a much wider set of data (“beware the man of only one study” etc). If there was one study finding no impact of bednets, I doubt GiveWell would conclude it would be reasonable to discount all the previous data on bednets.

• MichaelPlant 10 Dec 2021 12:24 UTC

  2 points

  in reply to: MichaelPlant’s comment on: Donat­ing money, buy­ing hap­piness: new meta-analy­ses com­par­ing the cost-effec­tive­ness of cash trans­fers and psy­chother­apy in terms of sub­jec­tive well-being

  First, we’ve looked at the effect of therapy under our current moral weights, which we use to trade off outcomes like increasing consumption, averting deaths, and averting morbidity. Under this approach and using the effect of depression on DALYs, we find similar cost-effectiveness between therapy and cash transfers.

  What is the conversion rate here between DALYs and income increases, and on what is it based? I’m not sure what method could be being used here except by inputting one’s intuitions. In which case, it would be good to make that clear, as people may think the conversation rate is an authoritative fact, rather than (just) an opinion. It would be interesting to state how much readers’ opinions would need to differ from GiveWell’s to reach alternative conclusions!

  To bang a familiar drum, the reason to use subjective wellbeing measures is that we can observe how much health and income changes improve wellbeing, rather than having to guess.

• MichaelPlant 10 Dec 2021 12:23 UTC

  2 points

  in reply to: MichaelPlant’s comment on: Donat­ing money, buy­ing hap­piness: new meta-analy­ses com­par­ing the cost-effec­tive­ness of cash trans­fers and psy­chother­apy in terms of sub­jec­tive well-being

  However, we think there are some limitations to these measures, and give weight to other factors. I expect that GiveWell would find interventions that have benefits beyond subjective well-being more cost-effective than HLI would.

  It’s not easy to respond to this—it’s not stated what the limitations and other factors are.

  More generally, there’s no reason to think in the abstract that, if you’re pluralist rather than monist about value, this changes the relative cost-effectiveness ranking of different actions. You’d need to provide a specific argument about what the different values are, how each intervention relatively does on this, and how the units of value are commensurate.

  For example, imagine a scenario where intervention X provides 15 units of happiness/​$ but does nothing for autonomy and intervention Y provides 10 units of happiness/​$ and 10 units of autonomy/​$. If we take one unit of happiness as being as valuable as one unit of autonomy, then Y is better than X. However, someone who only valued happiness would think X is better.

• MichaelPlant 10 Dec 2021 12:18 UTC

  2 points

  in reply to: MichaelPlant’s comment on: Donat­ing money, buy­ing hap­piness: new meta-analy­ses com­par­ing the cost-effec­tive­ness of cash trans­fers and psy­chother­apy in terms of sub­jec­tive well-being

  It seems possible there are significant spillovers from therapy, but our current best guess is these would be much smaller than for cash.

  It would be helpful if GiveWell could share what their current best guess is. Even if spillovers are 30% for therapy and 100% for cash, assuming the original 12x multiple and 3 other household members, then the multiple would still be 5.7.

  Cash = 1 + (1*3) = 4

  Psychotherapy = 12 + ((0.3*12)*3) = 12 + 10.8 = 22.8

  Hence, therapy still looks quite a bit better even if the spillover effects are small.

• MichaelPlant 10 Dec 2021 12:17 UTC

  2 points

  in reply to: MichaelPlant’s comment on: Donat­ing money, buy­ing hap­piness: new meta-analy­ses com­par­ing the cost-effec­tive­ness of cash trans­fers and psy­chother­apy in terms of sub­jec­tive well-being

  HLI assumes any spillover effects to other household members are proportional across interventions—i.e., if a cash transfer benefits other household members’ subjective well-being x% as much as it benefits the recipient, the same is true for therapy.

  If there are 4 to 5 members per household (roughly what we estimate for participants in GiveDirectly’s program) and there were no household multipliers from therapy, psychotherapy would be 2x-3x as cost-effective as cash transfers, taking HLI’s other assumptions as given.

  This is definitely not what we think, particularly the assumption it will be proportional across ‘any’ intervention! I’m sure why someone would believe that.

  Our position, as outlined in this twitter thread, is quite a bit more nuanced. There wasn’t much evidence we could find on household spillovers—five studies for cash, one for mental health—and in each case it indicated very large spillover effects, i.e. in the range that other household members got 70-100% of the benefitted the recipient did. We didn’t include that in the final estimate because there was so little evidence and, if we’d taken it at face value, it would only have modestly changed the results (making therapy 8-10x better). Even in the extreme, and implausible, case where therapy has no household spillovers, it wouldn’t have yielded the result that psychotherapy is more cost-effective than cash transfers. We discussed this in the individual cost-effectiveness analysis reports and flagged it as something to come back to for further research.

  We agree that the effects of household spillovers from cash are large. Where our priors may diverge is that HLI (and others) think that the spillovers from therapy are also large, whereas GiveWell is very sceptical about this. We are now conducting a thorough search for more evidence on household spillovers, so we are not just swapping priors.

• MichaelPlant 10 Dec 2021 12:15 UTC

  2 points

  in reply to: GiveWell’s comment on: Donat­ing money, buy­ing hap­piness: new meta-analy­ses com­par­ing the cost-effec­tive­ness of cash trans­fers and psy­chother­apy in terms of sub­jec­tive well-being

  Before I respond to the details, I’d like to thank GiveWell for engaging with these questions. I’m delighted our research has led to them producing their own reports into group psychotherapy and using subjective wellbeing to determine one’s moral weights.

  GiveWell kindly shared a draft of their reply with us in advance and we made several comments clarifying our position. However, they decided to publish their original draft unchanged (without offering a further explanation) so we’re restating our comments here so that readers can build a better understanding of where our positions differ. I’ll split these up so it’s easier to follow, first quoting the response from GiveWell, then providing our reply.

• MichaelPlant 26 Nov 2021 12:44 UTC

  9 points

  on: A Red-Team Against the Im­pact of Small Donations

  A lot’s been said. Is this a fair summary: small donors can do a lot of good (and earning and giving can be much higher impact than other altruistic activities, like local community volunteering) but as the amount of ‘EA dedicated’ money goes up, small donors are less impactful and more people should consider careers which are directly impactful?

• MichaelPlant 24 Nov 2021 12:44 UTC

  35 points

  on: Min­i­mal-trust investigations

  I have to say, I rather like putting a name to this concept. I know this wasn’t the upshot of the article, but it immediately struck me, on reading this, that it would be a good idea for the effective altruist community to engage in some minimal trust investigations of each other’s analyses and frame them as such.

  I’m worried about there being too much deference and actually not very much criticism of the received wisdom. Part of the issue is that to criticise the views of smart, thoughtful, well-intentioned people in leadership positions might imply either that you don’t trust them (which is rude) or that you’re not smart and well-informed enough to ‘get it’; there are also the normal fears associated with criticising those with greater power.

  These issues are somewhat addressed by saying “look, I have a lot of respect for X and assume there are right about lots of things, but I wanted to get to the bottom of this issue myself and not take anything they said for granted. So I did a ‘minimal-trust investigation’. Here’s what I found...”

• MichaelPlant 24 Nov 2021 10:51 UTC

  8 points

  in reply to: Kerry_Vaughan’s comment on: De­spite billions of ex­tra fund­ing, small donors can still have a sig­nifi­cant impact

  Yeah, does not seem like a good outcome if people are donating, say, 10% of their salary, then they come to EA events and they get the feeling that people look down their noses at them as if to say “that’s it? You don’t have an ‘EA’ job?”

• MichaelPlant 23 Nov 2021 22:22 UTC

  46 points

  on: De­spite billions of ex­tra fund­ing, small donors can still have a sig­nifi­cant impact

  [restating and elaborating on what I said on twitter]

  Thanks very for this update, Ben. The “EA has loads of money” meme has unfortunately led people to (incorrectly) assume that everything ‘within EA’ was fully funded. This made it harder to fundraise, particularly for small orgs, like mine, who do need new donors, because prospective donors assumed they weren’t necessary.

  Of course, the meme had no impact on organisations that are already fully-funded—which is more or less only those orgs being funded by Open Philanthropy.

  What links here?

  • What are the bad EA memes? How could we re­frame them? by Nathan Young (16 Nov 2021 11:18 UTC; 15 points)

• MichaelPlant 19 Nov 2021 20:25 UTC

  1 point

  in reply to: RobBensinger’s comment on: Ngo and Yud­kowsky on AI ca­pa­bil­ity gains

  Ok, that helps—a litte! - but it’s still not quite at TL;DR. :)

• MichaelPlant 19 Nov 2021 11:36 UTC

  6 points

  in reply to: Ben Williamson’s comment on: Sleep: effec­tive ways to im­prove it

  Hello Ben!

  Good to know this was based on existing literature. In most cases, it helps to show the reader you know that literature, to outline what it is, and then go on say what your new contribution is. Like I say, you missed a few of the obvious things, which is unfortunate. A piece of “what works for X” should, I say, include the things that work for X, then perhaps go on to flag which of these are likely to be a surprise, rather than assuming on the reader’s behalf what they will already know. If you are going to have a piece on “what works for X that might but that might surprise you” you should at least clearly flag that, and then point to something such as “standard guidance on X”.

  Re strength of interventions being “40%” that still seems a confused way of presenting the information. 40% of what? Of a maximum score? A maximum score of what? Of cost-effectiveness? Well, why not just present the effectiveness numbers and divide them by the costs then?

  I agree that this sort of thing can have a lower level of rigour but I stand by my concern that the method you use is so puzzling it’s questionably useful at all. You gathered quite a bit of relevant info, but I think you presented it in a less-than-ideal way. Here, simpler would have been better: I’d have preferred a post that just said “here’s a list of evidence-based ways to improve sleep” and then listed them and provided a brief discussion on each. That seems the way to go unless you have the data and time to do a quantitative (cost-)effectiveness analysis.

  Glad you think we (at HLI) do good work. Like I see, feel free to reach out if you want to chat about research methods etc.! You can get me at michael@happierlivesinstitute.org

• MichaelPlant 19 Nov 2021 11:16 UTC

  5 points

  on: Ngo and Yud­kowsky on AI ca­pa­bil­ity gains

  I might be interested in this, but I’d be really helped by a TL;DR or similar providing some context on what’s being discussed.

• MichaelPlant 18 Nov 2021 10:48 UTC

  34 points

  on: Sleep: effec­tive ways to im­prove it

  While I am as much a fan of wellbeing research as the next fellow—indeed, probably a much bigger fan—I have to say I found the methodology and conclusions of this research rather confusing.

  If I were approaching this topic, I would have (1) done a review of the existing literature to find out what people thought was effective and what the possible interventions were, then (2) tried to assess the options in terms of (a) a comparable metric of effectiveness and (b) cost, so readers could think about what would do the most for them at the least effort.

  As it is, this research seems to have missed out many of the standard pieces of advice like avoiding alcohol, napping after 3pm, having a large meal before bed, or having a sleep routine. The author doesn’t mention having looked at the existing literature, but does note that other EAs have mentioned sleep. I don’t mean to single out the OP, but I do want to deride the myopic and self-referential tendency among effective altruists in general to overlook work done outside effective altruism. Lots of good has happened ‘out there’ and we ignore it at our peril.

  What I found least satisfying about this research was how this (partial list of) interventions were assessed. As far as I can see, the ‘weighted-factor model’ involved assigning unexplained subjective numbers to various seemingly-arbitrarily chosen properties, then assigning a seemingly-arbitrary weight to each factor to aggregate them. * I am reminded of “Garbage-In-Garbage-Out” concept in computer science where nonsense inputs products nonsense outputs. As a reader, I have no idea how to interpret the rankings or numbers—what does it mean that melatonin gets “5.95/​10″ or that CBT-I gets “5.78”? - or how much to update off them. The results are basically uninterpretable.

  I would strongly recommend that the OP heavily revise their methods and the presentation of their research for any further work. The main thing would be to present the results of the interventions in a standardised metric, e.g. total sleep time, or standard deviations of something, so readers can make a comparison themselves, then make comments on cost and, if necessary research quality. I am happy to provide advice if that’s helpeful.

  *I recognise the weighted-factor model is something Charity Entrepreneurship use. I have raised it with them several times that, for the reasons given, I find this approach hard to follow or justify and thus questionably useful.

• MichaelPlant 27 Oct 2021 16:32 UTC

  11 points

  on: Re­move An Om­nivore’s Statue? De­bate En­sues Over The Le­gacy Of Fac­tory Farming

  It seems that people object most to statues of those who became rich, famous, or powerful from doing something objectionable. For instance, you get rich from slavery, become a philanthropist, then get a statue for being a philanthropist, cf Edward Colston in the UK. People mind less if you so happen to do something objectionable, but that’s not your reason from prominence.

  Therefore, the story would have much more oompfh if it was about a factory farming magnate who, say, set up a university with the proceeds, instead of just a guy who ate meat. You could have had quite a lot of fun teasing out those parallels.