I’m a doctor working towards the dream that every human will have access to high quality healthcare. I’m a medic and director of OneDay Health, which has launched 35 simple but comprehensive nurse-led health centers in remote rural Ugandan Villages. A huge thanks to the EA Cambridge student community in 2018 for helping me realise that I could do more good by focusing on providing healthcare in remote places.
NickLaing
Karma: 5,437
Brilliant Harfe and Laura. Here in Uganda many primary students are beaten badly, while in high schools there’s a common system, which is borderline torture, where many students only get 4-6 hours sleep a night. My wife works in advocacy and has considered trying to tackle this—I’m not sure where this might fit on the effectiveness scale, but potentially reducing suffering in millions of kids could be a pretty good intervention.
Like you both I struggle to see a scenario where this would be more effective in Richer countries than places like here in Uganda.
Thanks Cienna! I’m sure you are far from clueless :)
Like with any large scale policy issue, tractability is a big issue—but in this kind of case I have a couple of vaguely similar examples which show that it could well be tractable. I’m an enormous fan of advocacy and actvism as one of the most cost effective ways to solve specific, clear problems, perhaps even a bigger fan than most effective altruists as I’ve seen my wife succeed spectacularly a couple of times at least.
My wife worked on something 5 years ago which has some similarity in that it was a policy that was changed, having a big positve effect. Her and a community group from scratch managed to get a local district level law passed to effectively ban alcohol “sachets”, tiny 50ml plastic bags of heavy spirits of varying qualities. 2 years later later this was followed up with a nation al ban. This was based on both strong community will to ban the sachets as they could see the enormous harm caused by them, and a large amount of research that shows if you increase the quantity of the minimum size unit of alcohol (e.g. from 50 to 150ml like what happened here) , you hugely lower the damage done by alcohol.
Also the banning of lead paint thing in Malawi has some similarities in that it’s a harmful policy being overturned through government advocacy, and that seems to be going pretty well.
https://forum.effectivealtruism.org/posts/ErKzbKWnQMwvzRX4m/seven-things-that-surprised-us-in-our-first-year-working-in
What it would take (for example the high school sleep issue) specifically is advocacy to the ministry of education to pass a local (district wide) or national ordinance decreeing minimum sleep in a high school, then ensuring enforcement (harder than getting the law passed). The physical abuse of primary school children would be much harder as it is culturally ingrained—it’s such a well known and horrible issue there are in fact a number of BINGOS (big international NGOs like world vision etc.) doing a terrible job already trying to fix that problem mainly through education (posters, trainings, etc.). High level advocacy would be far better.
The thing is you could throw $30,000 or something at a specific problem such as the borderline torture of sleep deprivation in high school kids and probably have something like a 1 in 10 chance of success(with astronomical uncertainly and probably better odds if it’s someone like my wife working on it!) . I’m not going to do the math but it might well be a relatively cost-effective campaign.
Wow I just wrote an essay oh dear...
Sorry I forget there’s a bunch of context I forget to lay out
Maybe 80% of high schools here are boarding schools, that’s what I’m talking about.
So most of these these boarding schools force the students to get up at 4:00am-5:00am to prepare ,and work until 10pm in the evening. It sounds crazy but the schools honestly think this torrid regime will get better results for their students. Of course students end up trying to steal extra sleep wherever they can, just to get by.
With 2 students we were helping with high school where I think this contributed to serious mental health problems at the school, and one ended up leaving school.
Thanks so much Joel and I’m stoked by your response. I don’t think I’ve been in a forum where discussion and analysis is this good.
I’m not sure that having close to no evidence on a major concern about potential validity of before and after surveys should be very reassuring.
That tiny piece of evidence you cited only looks at the “experimenter demand effect”, which is a concern yes, but not my biggest concern. My largest concern is let’s say the “future hope” effect which I gave the example from in my first reply – where participants rate the effect of interventions more positively than their actual experience, because they CORRECTLY assess that a higher rating may bring them better help in future. That’s what I think is most likely to wreck these before and after surveys.
I don’t know this field well at all like you so yes it seems likely this is a poorly researched issue . We have experience and anecdotes like those from Dambisa Moyo, me and many others working in the NGO world, that these incentives and vested interests can greatly affect before and after surveys. You recently wrote an article which included (to your credit) 4 disadvantages of the WELLBY. My inclination is (with enormous uncertainty) that these problems I’ve outlined with before and after subjective surveys in low income countries are at least as big a disadvantage to the WELLBY approach as any of the 4 issues you outlined. I agree that SWB is a well validated tool for one time surveys and there are no serious vested interests involved. It’s the before and after surveys in low income countries that are problematic.
These are the 3 major un-accounted for problems I see with before and after self reporting (first 2 already discussed).
My biggest concern is the “future hope” effect—people positively responding because they correctly believe a positive response is likely to get them more and even better help in future.
The “ experimenter demand effect” (as you discussed) is if interviewees are primed to give people the answer they think the experimenters want.
A third potential problem (which I haven’t mentioned already but have experienced) is that interviewers manipulate results positively in the direction of the intervention either through the way they ask questions, or even through increasing scores fraudulently. This is again rational as their jobs often depend on the NGO raising more money to support “successful interventions” to keep them employed. I have seen this here a number of times during surveys within my own organisation AND in other organisations, It’s very hard to stop this effect even after talking to researchers about it – I have tried but there was still manipulation present. This is probably a smaller problem than the other two, and easier to control and stamp out.
On the “Also, we’re comparing self-reports to other self-reports” front—I agree this could be be fine if comparing between two interventions (e.g. cash and psychotherapy) and the effect might be similar between interventions. I think though most of the studies that you have listed compare intervention to no intervention, so your point may not stand in these cases.
I’ll change my mind somewhat in your direction and give you some benefit of the doubt on your point about objective measures not working well for wellbeing assessment, given that I haven’t researched it very well and I’m not an expert. Let’s leave objective measures out of the discussion for the moment!
I love the idea of your RCT cash transfers vs. psychotherapy but I’m confused about a number of parts of the design and have a few questions if you will humour me .
- The study design seems to say you are giving cash only the intervention groups and not the control group? I suspect this is a mistake in the reporting of the study design but please clarify. To compare cash vs psychotherapy would you not give the cash to the whole control group and either not to the intervention group at all, or only a small subsection of the intervention group? I might well have missed something here...
- Why are you giving money at the end of the intervention rather than at the start? Does that not give less time for the benefits of the cash to take effect.
- Why are you only giving $50 cash transfer and not around $130 (the cost of the therapy). Would it not would be a fairer comparison to compare like for like in terms of money spent on the intervention?
It seems logical that most RCTs in the effective altruism space now should be intervention vs cash transfer, or at least having 3 arms Intervention vs cash transfer vs nothing. Hopefully I’ve read it wrong and that is the case in this study!
To finish I like positivity and I’ll get alongside your point about fixing the boat rather than dismissing it. I feel like the boat is more than a little leak at this stage, but I hope I’m wrong I love the idea of using before and after subjective wellness measures to assess effectiveness of interventions, I’m just not yet convinced yet it can give very meaningful results based on my issues above.
Thanks so much if you got this far and sorry it’s so long!
Matt these are fantastic questions that I definitely don’t have great answers to, but here are a few thoughts.
First I’m not saying at all that the Strong minds intervention is likely useless—I think it is likely very useful. Just that the positive effects may well be grossly overstated for the reasons outlined above.My take on the result of that original 2002 RCT and Strong Minds. Yes like you say in both cases it could well be that the treatment group are giving positive answers both to appease the interviewer (Incredibly the before and after interviews were done by the same researcher in that study which is deeply problematic!) and because they may have been hoping positive responses might provide them with further future help.
Also in most of these studies, participants are given something physical for being part of the intervention groups. Perhaps small allowances for completing interviews, or tea and biscuits during the sessions. These tiny physical incentives can be more appreciated than the actual intervention. Knowing World Vision this would almost certainly be the case
I have an immense mistrust of World vision for a whole range of reasons, who were heavily involved in that famous 2002 RCT. This is due to their misleading advertising and a number of shocking experiences of their work here in Northern Uganda which I won’t expand on. I even wrote a blog about this a few years ago, encouraging people not to give them money. I know this may be a poor reason to mistrust a study but my previous experience heavily biases me all the same.
Great point about the NBER paper which featured a pure control group. First it was a different intervention—individual CBT not group therapy.
Second it feels like the Kenyan study was more dispassionate than some of the other big ones. I might be wrong but a bunch of the other RCTs are partly led and operated by organisations with something to prove. I did like that the Kenyan RCT felt less likely to be biased as there didn’t seem to be as much of an agenda as with some other studies.
Third, the Kenyan study didn’t pre-select people with depression, the intervention was performed on people randomly selected from the population. Obviously this means you are comparing different situations when comparing this to the studies with group psychotherapy for people with depression.
Finally allow me to speculate with enormous uncertainty. I suspect having the huge 1000 dollar cash transfers involved really changed the game here. ALL participants would have known for sure that some people people were getting the cash and this would have changed dynamics a lot. One outcome could have been that other people getting a wad of cash might have devalued the psychotherapy in participants eyes. Smart participants may even have decided the were more likely to get cash in future if they played down the effect of the therapy. Or even more extreme the confounding could go in the opposite direction of other studies, if participants assigned to psychotherapy undervalued a potentially positive intervention, out of disappointment at not getting the cash in hand. Again really just summising, but never underrate the connectivity and intelligence of people in villages in ths region!
Apologies for assuming that the RCT involved HLI—the strong minds involvement lead me to that wrong assumption! They will then be addressing different questions than we are here—unfortunately for us that trial as stated in the protocol doesn’t test cash transfers vs. psychotherapy.
I don’t quite understand the distinction in your question sorry, will need rephrasing! I’m referencing the problems with any SWB measurement which involves measuring SWB at baseline and then after an intervention. Whether there is a control arm or not.
Looking forward to hearing more nice one!
I love this. Living more simply does so much good for so many reasons. I would extend this as a challenge to the EA community, as well as something to be praised in the ultra rich. Living more simply creates value on so many fronts no matter how rich you are, including...
Minimise waste
Minimise carbon emissions
Minimise spending to maximise giving
Builds integrity in your EA position (More than just a bunch of rich tech bros who want to feel good about themselves) , and forge a small degree of solidarity with the poor we claim to be supporting.
(Perhaps most importantly) Creates curiosity from others as to why you live simply, allowing EA evangelistic opportunities ;).
I would say however that living simply is so much more than doing nothing. The norm of modern society is to spend as much (or more than) you earn, so it takes great thought, discipline and even sacrifice to live more simply. It’s far far harder than doing nothing
I have wondered why effective altruism doesn’t make a bigger deal of simplicity within our own community, especially from an Evangelistic point of view where I think it can work wonders.
Hi Joel—Nice one again
”Right, our concern is that if this bias exists, it is stronger for one intervention than another. E.g., say psychotherapy is more susceptible than cash transfers. If the bias is balanced across both interventions, then again, not as much of an issue.”
I would have thought the major concern would have been if the bias existed at all, rather than whether it balanced between interventions. Both StrongMinds evidence assessing their own program and most of the studies used in your WELLBY analysis are NOT vs. another intervention, but rather Psychotherapy vs. No intervention. This is often labelled “Normal care”, which is usually nothing in low income countries. So if it exists at all in any magnitude, it will be affecting your results.
Onto your question though, which is still important but I believe of secondary importanceYour idea of a kind of “fake” placebo arm would work—providing the fake placebo was hyped up as much and taken as seriously as the treatment arm, AND that the study participants really didn’t know it was a placebo. Unfortunately you can’t do this as It’s not ethical to have an RCT with an intervention that you think has no or little effect. So not possible I don’t think
I like your idea of interviewers in a trial stating for a subset of people that their answers won’t change whether they get more or not n future. I doubt this would mitgate the effect much or at all, but it’s a good idea to try!
My very strong instinct is that cash transfers would illicit a FAR stronger effect than psychotherapy. It’s hard to imagine anything that would illicit “future hope” more than the possibility of getting cash in future. This seems almost self-evident. Which extremely poor person in their right mind (whether their mental health really is better or not) is going to say that cash didn’t help their mental health if they think it might increase their chance of getting more cash in future?
Again I think just direct RCTs vs. Cash Transfers is the best way to test your intervention and control for this bias. It’s hard to imagine anything having a larger “future hope” effect than cash. If psychotherapy really beat the cash given that cash will almost certainly have a bigger “future hope” bias than the psychotherapy, then I’d say you’d have a slam dunk case.
I am a bit bewildered that StrongMinds has never done a psychotherapy vs. Cash transfer trial. This is the big question, and you claim that Strongminds therapy produces more WELLBYs than Cash Transfers yet there is no RCT? It looks like that trial you sent me is another painful missed opportunity to do that research as well. Why there is no straight cash transfer arm in that trial doesn’t make any sense.As far as I can see the Kenyan trial is the only RCT with Subjective wellness assessment which pits Cash Transfers vs. Psychotherapy (although it was a disproportionately large cash transfer and not the Strongminds style group therapy). If group Psychotherapy beat an equivalent cash Cash transfer in say 2 RCTs I might give up my job and start running group psychotherapy sessions—that would be powerful!
Should all RCTs now be Intervention X Vs. Cash Transfer?
Thanks so much Rory and for the links to your earlier post and the USAID stuff!
I think your criticism is a good criticism of RCTs in general, but it seems to me more a criticism comment about RCT design then being a clear argument against comparing with cash transfers. RCTs on development NEED longer term outcome measurement, and surely need at a minimum 2 data points at 2 different times after the study. And of course the most important data point is after many months or even many years as you talked about in your article.
I’m not at all sure about the ethical side either . Medical RCTs compare a new trial treatment against the most up-to-date treatment—not so much because we worry about “tricking” a patient like you say (there are still plenty of RCTs with sugar placebo pills which is deemed ethically OK), we are still OK with a kind of ‘deception’. What we AREN’T OK with is doing a trial where we give the control arm nothing at all, when we know there is a better option than nothing for the medical condition. And I’d argue that cash is usually a better option than nothing for many development conditions.
That’s a great and sobering point about the counterfactual potentially being inaction if cash transfers won the day. Why should the counterfactual be inaction though? I would hope as development people we are good enough that if Cash was equivalent or better than intervention X, this wouldn’t lead us not to inaction but instead to give more cash instead. Maybe I’m naive and idealistic though, and maybe you’re right that there is actually a practical advantage in seeing a positive impact of intervention X, even if it is worse than a cash transfer. I don’t think that should be the case though.
That’s the whole question really—should we spend our millions on RCTs asking “What is the impact of X”, or “Is X better than cash”. What we really want to know, the practical question which underlies the research question is is “Should we be implementing this intervention at scale”. I’d argue that to answer that, the question vs. Cash is the one that matters more.Thanks so much for your reply, I can see you’ve thought about this far more than me and I loved your original post—weird that searches on the forum didn’t bring it up, maybe they should employ google search on the site haha.
Wow thanks so much for this effort - as someone who runs a small charity, it’s so encouraging to see smallish EA aligned organisations getting a look in for some funding and going through this great process. I have a couple of comments :).
1. As someone working in a global health charity, I often find it strange how little weighting delivery is given in Effective Altruism in general. There are a million good ideas that could have great impact, what matters more is whether the intervention will happen or not. It almost feels like delivery could almost be a multiplier for other scores rather than a smaller score on it’s own, or at least it could have a higher weighting maybe?. Does the fidelity of all the other scores not depend in a sense on the project actually playing out as planned?
2. I also have questions about how good a measure importance, tractability and neglectedness translate as a measure for rating an intervention, when I think they emerged in effective altruism for rating a problem. Were the judges using these criteria to rate the problem being addressed or the solution itself? For example on neglectedness some of the solutions (Nuclear winter one, Existential risk one) might be the only people doing that exact thing to contribute to the issue (say a score 10⁄10), while the issues themselves might be neglected but less so (e.g. 7⁄10).
3. (Selfish question!) Do you know of other EA organisations or grantees doing anything vaguely similar—smaller grants to smaller organisations? Is there any online database or list on the forum of EA aligned donor orgs?Thanks so, so much I found your whole process and system very interesting and informative—must be the most transparent grantee of all time ;). Was very encouraging
Thanks so much for this (I’m in Uganda). Love hearing from Africans on the EA forum nice one!
I love the idea of phages, and I hope they can become a useful antimicrobial agent in time. Unfortunately though I’m not sure it’s worth focusing on things like phage banks, until phages are proven to be useful as an antimicrobial agent in the general population.Also tragic as it is, I sadly don’t have confidence that investing in phage research within Africa will yield fruit. I could be wrong about this, but north of South Africa there have been very few meaningful biomedical innovations come out of any country. I can’t think of any vaccine, medication or test which has been innovated This may be due to a range of factors like poor education systems, poor infrastructure and most importantly I think horrendous beauracracies which stifle talent like yours and have no agility.
In saying that, if something was going to work it would more likely be a smaller “cottage” type project like yours. I think that’s why Tech innovators in Africa have so many amazing achievements, because they can innovate alone and in small groups, without needing the infrastructure and beuracratic support needed by biomedical researches.
Anyway I love the idea, nice one!
Hey Mathias, I’ve work in development here in Northern Uganda for 10 years, and I think this is a fantastic idea for an org. I love your examples and I have no doubt that this issue is tractable, although I’ve failed in my own feeble local efforts here in Northern Uganda ;). Here’s a couple of comments off the top of my head :). I could go on and on but it’s too much for a comment .
My experience has been that most government aid projects are terrible, either doing harm or very little good at all. The worst are government-to-government aid, where unfortunately most of the aid money goes as spelled out well in Dambisa Moyo’s “Dead Aid” book. Here in Uganda there are some great high impact exceptions such as USAID’s Pepfar, providing free malaria commodities, and some results based healthcare projects (although these can go both ways) but they are exceptions not the rule.
We currently have GIZ (German development agency with an unfortunate name) spending millions here in Northern Uganda doing lot’s of close to zero impact projects while causing potential harm while they are at it. Their projects are so painfully bad it sometimes makes me feel physically sick, and I’ve failed completely in my few efforts to talk to GIZ members about what they do. I don’t think local lobbying is necessarily very useful because no-one working on the ground wants to even consider the possibility that what they are doing here in Uganda might be useless or harmful, and they don’t make many decisions about what they do. This kind of advocacy then needs to happen at a higher level. I understand EA is fairly active in Berlin, and this is the kind of thing that could well be lobbied on successfully.
My experience here in Northern Uganda is that most climate mitigation funding is ineffectual, with a surprising amount of it just funnelled into corrupt official’s pockets and a lot of the rest spent on meaningless trainings and meetings. Like Stephen said much of it would be far better spent just on development initiatives—that will be better climate mitigation than the current useless projects. Redirecting he 100 billion of climate funding could be a GREAT area for advocacy, perhaps the most important at the moment. I might write a post on this even…
I think a key factor to consider will be which governments might be willing to be nudged in the right direction. My instinct is that USAID might be a waste of time to lobby with their rigid policies more concerned with politics and accountability than impact (although they would claim otherwise), while you might be able to get more traction in European countries.
Can’t believe your Hans Neiman joke as well. About the most niche joke I’ve ever seen lol what percent of people here will get it haha love it!
Keen to discuss this more!
Thanks Sanjay. I agree in the UK context. I think this is a great example of how every country is very different. Yes England have absolutely routed their overseas aid and might be another good example of a country not to focus on with this kind of advocacy. Which is sad because until recently UKAID had some pretty good programs.
Love it listen to this person ;).
For a start, read that list and see if you can find even a handful of initiatives that seem likely to have a reasonable impact. Most of these projects are not clear, potentially high impact interventions but usually a bunch of trainings, meetings, meaningless “capacity building” of government staff.
This one here really made me angry https://www.giz.de/en/worldwide/42196.html“It advises policy-makers in East African countries on the opportunities offered by carbon markets and carbon pricing instruments. In addition, the project provides expert and technical advice to government agencies on updating NDCs and long-term climate strategies.… In addition to this, workshops and networking meetings provide information on Article 6 and market-based approaches in the region.”
Uganda is run by a dictator who’s Army and police continue to oversee the pillaging of their few remaining forests for charcoal, and here GIZ is pouring more money into those same thieve’s pockets (as reported below).
https://www.independent.co.ug/black-gold-report-pins-security-for-protecting-charcoal-cartels/
https://www.monitor.co.ug/uganda/news/national/security-forces-aiding-charcoal-trade-report-3367774
Obviously I only have encountered GIZ first hand here in Northern Uganda. Just to throw one concrete example in here , there’s one particularly bad ongoing project here on rubbish collection (much more mundane than charcoal cartels!). I’d hardly call rubbish collection the biggest problem we have here in the first place. GIZ’s mad approach was to try to get poor people here in the city to pay a fee and bring their rubbish to collection points—basically trying to conjure up huge behaviour change overnight which of course was never going to work. We even had one of their staff come to our door and announce the new program—which of course never happened. Who here is going to pay to bring rubbish to a collection point when they don’t even see it as a problem?
And as part of the project they might have spent half a million dollars on a handful of German imported rubbish collection trucks for local government, only for them to get seized by a debt collector apparently because of some phony debt probably trumped up from inside the local government itself.
They also do a bunch of water accessibility stuff here in Gulu town which is highly dubious from an inequality perspective, as it favours only rich people in town who already are doing OK, and neglects the 80% of poorer Ugandans who live outside of town.
You’d think these kind of cliche aid mistakes would be a thing of the past, but unfortunately not.Often GIZ projects don’t even make it make sense on paper, forget even about the disasters while implementing them. Read this garbage, especially the “Approach” section. What really even is their approach it’s certainly not clear t me? https://www.giz.de/en/worldwide/59817.html
It’s really sad, imagine the good they could do if they instead just gave all that money they are spending driving around in fancy cars and feeding rich corrupt government officials to poor people.
Hey Elliot nice one!
Government aid projects almost never consider impact the way effective altruists do in terms of DALYs, and so comparing interventions is very very difficult.
One of the biggest problems is that most aid is “Government to government” aid, and so either gets eaten by corruption or is pumped into extremely inefficient government programs. An example of better aid might be money given to the Global vaccine alliance (GAVI) to fund vaccination programs.A book I really like (although a bit old) is “Dead Aid” by Dambisa Moyo a Zambian born economist who worked for the world bank. She argues that government to government aid does more harm than good, which I agree with. I don’t agree with everything in the book but it gives a good overview of aid and the problems with it.
Nice one Zoe love these a lot
Great points Finn. Interested to know your background as well ;)!
In my opinion smaller country development offices might be a lot easier to shift than the bigger ones. You could well be right that the OP and enthusiastic people like myself are overestimating the tractability.
I agree there are a lot of academics and researchers in this space, but that work doesn’t usually lead to action. I would argue that the Paris agreement and Accra agenda were more about how countries should work together to deliver aid, and less about what aid money should actually be spent n.Could you give an example of an organisation which has a primary role of lobbying for better evidence based use of aid?
If I look at CGD’s impact report for this year, the had no numerical assessment of their progres, and their stated achievements are
- Increasing the number of women peacekeepers
- Increasing ID cards for development
- Maximising the benefits of migration
- Changing the conversation on China’s development debt
- Advance market commitments for pharmeceuticals
Among those I would only consider the advance market comitments for pharmecuticals and perhaps ID cards are likely to be high impact intervention sfor the world’s poorest. I found it interesting that the Center for Effective Altruism is one of their funders.
As a reply to your comments above“Dambisa Moyo (who worked for the World Bank for 2 years—before going to Goldman Sachs) is one view on how aid effectiveness… but I think it’s fair to say the consensus is that govt ownership is generally seen as a GOOD thing in aid effectiveness literature. “
Dambisa Moyo isn’t against all aid (a common misconception), just against direct government to government aid. She seems to not mind NGOs so much. Yes you are correct that government ownership is generally seen as a good thing in aid effectiveness literature. I disagree with this consensus, at least I disagree that aid money should pass through government hands. Of course this would be the best approach in theory, and is a self serving opinion if you want to keep your job as a diplomat or aid practitioner. Unfortunately in practise it fails over and over again. The best Aid projects I have seen, work in conjunction with government systems while not actually giving government money or full control. These kind of projects include PEPFAR, Mosquito net distribution, cash transfers to refugees, building schools (underrated intervention).
”In my experience development agencies are slow to update their practices (but quicker to organize interminably long workshops about updating practices). E.g. we’ve known for some time that cash transfers are an effective way of doing humanitarian aid in many/most contexts—and we still see that they are not the common way of doing things.”100% agree with that, well put.
Keep in mind I’m very biased living and working here in Uganda while trying to keep a broader perspective.
DISCLAIMER: (perhaps a double edge sword) I’ve lived in Uganda here for 10 years working in Healthcare.
Thanks Michael for all your efforts. I love StrongMinds and am considering donating myself. I run health centers here in Northern Uganda and have thought about getting in touch with you see if we can use something like the Strong minds in the health centers we manage. While working as a doctor here my estimate from experience that for perhaps between 1 in 5 and 1 and 10 of our patients, depression or anxiety is the biggest medical problem in their lives. I feel bad every year that we do nothing at all to help these people.
Point 1
First I read a reply below that seriously doubted that improving depression could have more positive psychological effect than preventing the grief of the death of a child. On this front I think it’s very hard to make a call in either direction, but it seems plausible to me that lifting someone out of depression could have a greater effect in many cases.
Point 2
I however strongly disagree with your statement here about self reporting. Sadly I think it is not a good measure especially as a primary outcome measure.
“Also, what’s wrong with the self-reports? People are self-reporting how they feel. How else should we determine how people feel? Should we just ignore them and assume that we know best? Also, we’re comparing self-reports to other self-reports, so it’s unclear what bias we need to worry about.”
Self reporting doesn’t work because poor people here in Northern Uganda at least are primed to give low marks when reporting how they feel before an intervention, and then high marks afterwards—whether the intervention did anything or not. I have seen it personally here a number of times with fairly useless aid projects. I even asked people one time after a terrible farming training, whether they really thought the training helped as much as they had reported on the piece of paper. A couple of people laughed and said something like “No of course it didn’t help, but if we give high grades we might get more and better help in future”. this is an intelligent and rational response by recipients of aid, as of course good reports of an intervention increase their chances of getting more stuff in future, useful or not.
Dambisa Moyo says it even better in her book “Dead Aid”, but couldn’t find the quote. There might also be good research papers and other effective altruism posts that describe this failing of self reporting better than me so apologies if this is the case.
You also said “Also, we’re comparing self-reports to other self-reports”, which doesn’t help the matter, because those who don’t get help are likely to keep scoring the survey lowly because they feel like they didn’t get help
Because of this I struggle to get behind any assessment that relies on self-reporting, especially in low income countries like Uganda where people are often reliant on aid, and desperate for more. Ironically perhaps I have exactly the same criticism of GiveDirectly. I think that researchers of GiveDirectly should use exclusively (or almost exclusively) objective measures of improved life (hemoglobin levels, kids school grades, weight for height charts, assets at home) rather the before and after surveys they do. To their credit, recent GiveDirectly research seem to be using more objective measures in their effectiveness research.
https://www.givedirectly.org/research-at-give-directly/
We can’t ignore how people feel, but we need to try and find objective ways of assessing it, especially in contexts like here in Uganda where NGOs have wrecked any chance of self reporting being very accurate. I feel like measuring improvement in physical aspects of depression could be a way forward. Just off the top of my head you could measure before and after mental agility scores, which should improve as depression improves, or quality of sleep before and after using a smart watch or phone. Perhaps even you could use continuous body monitoring for a small number of people, as they did here
https://www.vs.inf.ethz.ch/edu/HS2011/CPS/papers/sung05_measures-depression.pdf
Alternatively I’d be VERY interested in a head to head Cash transfer vs Strongminds RCT—should be pretty straightforward , even potentially using your same subjective before and after scores. Surely this would answer some important questions.
A similar comparative RCT was done in Kenya in 2020 of cash transfer vs. Psychotherapy, and the cash transfers clearly came through on top https://www.nber.org/papers/w28106.
Anyway I think Strong minds is a great idea and probably works well to the point I really want to use it myself in our health centers, but I don’t like the way you measure it’s effectiveness and therefore doubt whether it is as effective as stated here.
Thanks for all the good work!