Thanks, super-interesting!
On “We still need a RCT powered for Mortality”, someone from GW could confirm, but I heard through the grapevine that they are going to fund Kremer to do a large multi-country RCT powered for mortality to get at these very questions.
Hi Nick, thanks for the thoughtful response. I think you make a lot of good points and I agree that there are numerous incentives can can lead an M+E provider to bias results positively. That’s why there is a ton of bad M+E out there.
One main reaction: for an employee who works in an M+E org, there is arguably no worse situation than being pressured to skew your results positively, or even worse, taking on projects where you know a certain results is expected by your clients. It makes you feel you work is meaningless, and really sucks. And when you are put in this situations, you sure as hell don’t want to work for the same client again.
Yes, i hear you that for bean-counters in an organization (or those who get dividends in a for-profit org), there are strong incentives to make clients happy and get more contracts. But I think that the job-satisfaction incentive for rank-and-file employees skews the other way. And in the course of my experience, I think it is this latter incentive toward truth-telling that has dominated in most cases.
[Disclaimer: I’m the Chief Economist of IDinsight, an M+E provider who has worked with GiveWell and many others. I have a LOT of experience with evaluators being pressured to sugarcoat results, or lack thereof. ]
Strong disagree on this conclusion that M+E providers are inherently biased.
Yes, there are situations where M+E have incentives that can lead to bias. For instance, if an NGO hires an M+E provider to do an external evaluation of themselves, the NGO is therefore the ‘client’ of the researchers. This can be problematic, since the NGO will need to approve deliverables before payments are made. I’ve been involved in these situations and it is tricky.
But in general, arrangements can be made to align incentives with the truth. For instance if a funder (like GiveWell) hires an M+E provider to do an evaluation of one of its grantees, the incentives of the M+E provider are aligned with the funder, who hopefully would like to know the unvarnished truth. We’ve done numerous evaluations for GiveWell (most notably the New Incentives RCT) and have never felt any incentive to skew results one way or another.
From an organizational perspective, a well-run evaluation organization has much stronger long-term incentives to have a reputation for being honest, transparent, and truth-seeking, rather than getting repeat business from any particular client.
Thanks, super-helpful.
Is it still true though that FTX Foundation Inc has not filed for bankruptcy? If that’s true, returning any funds from FTX Foundation through this mechanism seems premature. But well, I’m no lawyer.
By “very careful”, I mean they shouldn’t make the case that their org is higher-impact than the current org unless they are damn sure. And this is an extremely difficult judgement call to make, when comparing two organizations whose mission is social impact. Given that impact is integral to an EA’s worldview, it would be a pretty incendiary accusation for a headhunter to make the case that org X is higher-impact than org Y, so someone should switch jobs. It’s one thing to make this case if hiring someone away from Exxon, but another to make the case within a community of arguably impactful organizations. I think these kinds of tactics have potential to cause major rifts within the community so should be avoided.
OP here. Thanks for all of the engagement with this post and for the varying opinions. People have brought up some important points on the benefits of headhunting (increased information, better outcomes for employees, overall better job matches, etc), and I agree with a lot of what is said. After taking these into account and mulling what has been said, here’s where I stand (subject to change):
It’s clearly ok for EA orgs to hire employees who work at other friendly orgs. Refusing to do so is illegal and I’d say also unethical.
I think it’s ok if you work at an EA org and you know someone who might be a great fit for a role at your org, totally find to reach out to them and let them know.
It’s not ok to use misinformation. I’ll be clear that in the headhunting instances I witnessed from EA orgs, I did not see any misinformation, and all communication was respectful.
I think recruiters and headhunters within the community should aim to inform rather than persuade recruits, which I do not think is the norm in the headhunting world. This can be a very fine line. The headhunting I witnessed was mostly in the “information” column, yet there was some stuff on the margin. For instance, I think a headhunter should be very careful about insinuate that they job they are recruiting for is higher impact than someone’s current job. In one of the emails I saw, the recruiter did not explicitly say this, but it was implied (or at least that was my interpretation). I’m not sure if this was purposeful.
I think it would be a positive norm for friendly orgs to alert one another if they plan to headhunt, especially if reaching out to multiple people for a role. This allows the original org an opportunity to do what they can to ensure that the employee is having their needs met in the current job. This did happen in one of the instances of headhunting that IDinsight encountered, and it was appreciated.
I think foundations/donor orgs have an extra layer of responsibility, as they wield a lot of power in the ecosystem, and in most cases pay more money than their grantees. When considering headhunting the staff of non-profits, they should be extra sure to follow best practices and ensure that they don’t destabilize the orgs they are poaching from. I’m not saying they shouldn’t recruit from NGOs, but I think they should be very careful about avoiding persuasion and alerting the recruitee orgs.
I think the bar for paid headhunters is even higher. For instance, if the EA Infrastructure Fund is going to fund headhunters for the EA ecosystem, I think that these headhunters should have a strong bias toward headhunting from lower-impact jobs. (Agreed this is not so easy to figure out what is high and low impact.) If they are instead just contributing to churn among high-impact orgs, I think they have an ambiguous amount of total social impact. Could be high if it increases information, improved job matches, improves employee satisfaction. But could be low if those positive elements are outweighed by switching costs of jobs and the cost of the headhunter themselves.
Good points- I take back my earlier “Clearly...” statement, and agree it needs to also include utility gains for the worker in the calculation.
Just to clarify, I wouldn’t be advocating that orgs don’t hire from peer orgs. Of course, post jobs, make them widely known, take and consider applications from all place. But I think it’s different to spend money on dedicated staff to directly target and aggressively recruit staff from friendly orgs within your ecosystem.
Oh that’s very interesting! I had no idea, seems relevant. Also not a lawyer, but I think that this would just apply to agreements not to hire others’ employees, as opposed to an agreement not to aggressively recruit.
Thanks for the comment- I see where you are coming from. As noted in a previous reply, I think a lot has to do with how much the headhunter informs vs convinces. There are a lot of parallels with advertising. Do we think that advertising performs a positive social function? Well, it could if it simply provides information about a new product and allows consumers to make more informed choices. But also the advertiser has incentives to increase sales, so why would we trust them to be truthful and have everyone’s best interests at heart? Headhunters/recruiters have incentives to fill roles, so I don’t think we should assume that they are playing a neutral, information-providing role.
Thanks for the comment- I understand where you are coming from, and see how this could go either ways. But I think I’d tend to disagree. I’m always happy for people to be aware of other opportunities and consider them, but I think there’s a difference when there are paid professionals targeting specific people to switch jobs. These professions tend to not just inform, but also convince. So in the situation of a job switch, you end up with a situation where the recruiting organization gains, the recruited organization loses, and actual job-seeker perhaps gains but this isn’t totally clear, depends on the amount that their decision was motivated by information vs convincing. And there’s a deadweight loss from the salary of the headhunter. Therefore, I think that the net effect of a headhunter could be positive or negative. Certainly it seems like they would have a higher impact if they recruited people from low-impact orgs to move to high-impact orgs.
Hi Joel, thanks for this write-up and for the work you’re doing on this. For some context, I’m the Chief Economist at IDinsight and worked on the GW-funded study you mentioned.
A few comments:
Thanks for this work and these thoughts! I haven’t dug into the math in detail, but I’m intrigued about systematically measuring and correcting for social desirability bias. But one question: are you arguing that SDB is more likely to be an issue in the community perspective than the individual perspective? I don’t really see why this would be the case- i think SDB is likely to be present in all frames.
I’m a bit confused why you think the community perspective is more reliable than the individual perspective? It seems to me that your assumption that both methods are downward biased is pretty strong, and the conclusion that the larger of the two is “right” is not apparent.
IDinsight is quite interested in doing more work in this area. We are currently pursuing a couple of opportunities, and are interested in working with funders who want to do more work on measuring preferences and integrating them into their programs. We house this work under our Dignity Initiative, since we believe that understanding and respecting preferences are a key part of upholding the dignity of the populations we serve. If CEARCH is working on funding more work in this area, we should talk.
Open Phil is considering funding a large replication/expansion of our preference work, where we would use a number of new tools to minimize the biases pointed our in your article. If this goes forward, we’ll have a lot more to say about this problem in the coming years! Assuming it does, we’re always looking for more ideas and would be interested in brainstorming with CEARCH to add to our pile of ideas.
Hi Richard,
I think you’ve identified a problem in the funding space, and I’ve had numerous conversations with others about this. A couple of comments:
As mentioned in another comment, I think that Open Phil’s Global Health and Development team is evolving to fill some of this gap. But they have certain issue areas of concentration, and also have a limited team evaluating grants, so I think they aren’t well-suiting to identifying high-impact opportunities in all areas (especially small grants).
I think the right venue for this would be EA Funds’ ‘Global Health and Development Fund’. Currently this fund is managed by Eli Hassenfeld and GW staff, which I think is a missed opportunity to provide a venue for more high-risk opportunities. While this fund has dispersed to some more ‘speculative’ orgs (like GCD, IPA), the most decision was to give 4.2 million to the Against Malaria Fund in Jan 2022. It seem like they don’t give many small grants either. I personally think it would be great if this fund had different managers, who explicitly looked for funding opportunities that are high impact in expectation but don’t fit into the processes and priorities of Open Phil or GiveWell.
Nice work Ryan! This is really interesting. I would agree that exposing IDEV professors to EA would be a net positive. I’m wondering what are the best ways to do this. My cynical take (from my time in the academy) is that it’s hard to get econ profs to take seriously knowledge generated in non-Econ circles. So maybe the way forward is to try to get an article on EA (or that would at least ex-post expose profs to EA) in QJE, or at least JDE. I’m not sure the right angle. But maybe you should come up with an idea and pitch it to EA Funds again!
I’m wondering where you think a paper on this topic will land. Probably not JDE, right?
Hi Stephen, just to add a bit to what my colleague Kim has said, we at Giving Green are working on something very similar to this. We aren’t explicitly starting with the Drawdown list of solutions, but have attempted to create a comprehensive list of areas for philanthropic engagement within climate space, and the drawdown list was one source of ideas. We’ve gone through a first cut to try to determine which are potentially most cost-effective, and are writing reports on some of the ones we think ex-ante have promise (such as the examples Kim wrote below). If you are considering doing a project such as you discuss, we’d welcome you to coordinate with us. We don’t have the resources to dig deep into every solution, so more hands are always welcome. But it would be a shame if we are overlapping work! Please feel free to be in touch at givinggreen@idinsight.org.
Hi James, thanks for this and I feel like this research is super-helpful. As you know, we at Giving Green have also explored the question of protest (as a form of what we call “outsider legislative advocacy”), and are also generally bullish on these techniques. But also, (as you mention), I think only a small minority of protest movements are really successful. We’ve had a lot of trouble identifying organizations we want to recommend in the context of climate policy in the US.
We’re looking forward to applying your findings as our search continues!
Sorry, looks like our write-up of CCL didn’t make it into that document. Sorry for the wrong info. I’m going to stick with my opinion there there is no viable path towards carbon pricing in the near term in the US, and also that carbon pricing in practice has not been a very effective lever for reducing GHGs. I’d be happy to change my mind on either of these points in the future. I like this article as a nice argument to be skeptical on carbon prices: https://bostonreview.net/articles/leah-c-stokes-matto-mildenberger-tk/
Hi Max, thanks for your comment. We quickly looked into CCL in 2020, and wrote up a “shallow dive” here. While we support the carbon tax push of CCL in theory, our assessment is that there is currently no clear pathway to passage of such a provision as it’s not supported by either party. We therefore decided not to pursue further research into CCL at that point, but are open to revisiting them in the future.
I don’t agree that a carbon pricing is the only way to reach our climate goals- my view is that reductions can alternatively be reached through regulations, and this is a more realistic path forward given political headwinds in the US. At least this seems to be the case for the foreseeable future.
Thanks so much for the engagement. We at Giving Green share your concern around some of CATF’s activities around carbon capture, though I wouldn’t go as far as to say that CATF’s work on 45Q is “net harmful”. Instead we acknowledge there are tradeoffs from CATF’s strategy in this sector that have uncertain overall impacts. We have noted this element as a “Key Uncertainty” in our report. The relevant text is copied at the bottom of this post.
Our recommendation of CATF was primarily based on our assessment of their work in Shipping/Aviation and Enhanced Geothermal, which were two of our focus areas this year. However, we are not specifically recommending restricted donations for two reasons:
We think that CATF is an overall strong organization with many important work streams, all focused on climate change mitigation. In these cases, we have a strong bias toward unrestricted funding, as it allows the greatest flexibility for our recommended organizations.
For an organization like CATF that receives a lot of unrestricted funding, recommending restricted funding can be mostly meaningless, since the organization can always funge a restricted donation with unrestricted money.
From the ‘Key Uncertainties’ Section of our Deep Dive on CATF
”Advocacy for incentives for power sector CCUS and captured CO2 storage via EOR: CATF’s advocacy for enhancements to the US Section 45Q tax credit included continued eligibility for power sector applications of carbon capture utilization and storage (CCUS). There is concern that the tax incentives may extend the life of US coal and natural gas-fired power plants. One analysis suggests that 45Q could increase the operating years of an otherwise end-of-life coal plant into the 2040s, resulting in at least 6 million metric tons of additional CO2e emissions. CATF claims that it foresees little deployment of CCS in the US power sector but that the plants that use it will help bring down its cost through learning by doing, resulting in accelerated uptake in emerging economies. While Giving Green thinks CCS could be a valuable technology in contexts such as heavy industry or power plants in emerging economies, we share concerns about incentivizing its use for US fossil fuel power plants. In addition, CATF has continued to advocate for the inclusion of enhanced oil recovery (EOR) for storage of captured emissions or atmospheric removals in subsidies such as 45Q. CATF argues that EOR is climate beneficial, that it serves as the primary niche market for scaling capture technologies, and that it can help transition to large-scale saline storage. Others question the need to subsidize it, especially given concerns over environmental justice, the potential to prolong oil extraction, and the involvement of the fossil fuel industry in the trajectory of emerging climate technologies like DAC.