Sjir Hoeijmakers🔸

• Sjir Hoeijmakers🔸Jan 3, 2023, 3:32 PM

  10 points

  1 ∶ 0

  in reply to: Nathan Young’s comment on: StrongMinds should not be a top-rated char­ity (yet)

  Hi Nathan, I don’t think the results of the RCT will be negative, just that they could cause us to update (in either direction) which adds uncertainty, though I’d admit that at a <50% forecast this could plausibly increase my forecast rather than lower it (though this isn’t immediately clear; depends on the interactions with the other reasons).

  And I hope the more elaborate reply I just wrote to Simon answers your remaining question.

• Sjir Hoeijmakers🔸Jan 3, 2023, 3:20 PM

  18 points

  4 ∶ 0

  in reply to: Jeff Kaufman 🔸’s comment on: StrongMinds should not be a top-rated char­ity (yet)

  Thanks Jeff, I think your summary is helpful and broadly correct, except for two (somewhat relevant) details:

  • GWWC didn’t recommend SM based on FP’s recommendation in 2019 but based on FP’s decision to still recommend SM as of this giving season (which is based on FP’s internal re-evaluation of SM).

  • I don’t expect there to be any new, decision-relevant information in FP’s recent internal re-evaluation that isn’t captured by the 2019 report + the recent HLI analysis (but I’m not sure about this—Matt can correct me if I’m wrong, though also see his comment here). Obviously the internal re-evaluation has extra “information” in the sense that FP has reviewed the HLI analysis, converted metrics to their new system, and run some extra tests, as Matt has explained, so maybe this is just semantics, but I think it’s relevant to the extent that a crux would be “FP is still recommending SM because of something only they know”.

  I understand the reasons for your suggestion w.r.t. GWWC’s inclusion criteria—we’ve seriously considered doing this before—but I explain at length why I still think we shouldn’t under (4) here. Would welcome any further comments if you disagree!

• Sjir Hoeijmakers🔸Jan 3, 2023, 2:57 PM

  88 points

  15 ∶ 4

  in reply to: Simon_M’s comment on: StrongMinds should not be a top-rated char­ity (yet)

  Hi Simon,

  I’m back to work and able to reply with a bit more detail now (though also time-constrained as we have a lot of other important work to do this new year :)).

  I still do not think any (immediate) action on our part is required. Let me lay out the reasons why:

  (1) Our full process and criteria are explained here. As you seem to agree with from your comment above we need clear and simple rules for what is and what isn’t included (incl. because we have a very small team and need to prioritize). Currently a very brief summary of these rules/​the process would be: first determine which evaluators to rely on (also note our plans for this year) and then rely on their recommendations. We do not generally have the capacity to review individual charity evaluations, and would only do so and potentially diverge from a trusted evaluator’s recommendation under exceptional circumstances. (I don’t believe we have had such a circumstance this giving season, but may misremember)

  (2) There were no strong reasons to diverge with respect to FP’s recommendation of StrongMinds at the time they recommended them—or to do an in-depth review of FP’s evaluation ourselves—and I think there still aren’t. As I said before, you make a few useful points in your post but I think Matt’s reaction and the subsequent discussion satisfactorily explain why Founders Pledge chose to recommend StrongMinds and why your comments don’t (immediately) change their view on this: StrongMinds doesn’t need to meet GiveWell-tier levels of confidence and easily clears FP’s bar in expectation—even with the issues you mention having been taken into account—and nearly all the decision-relevant reasoning is already available publicly in the 2019 report and HLI’s recent review. I would of course be very interested and we could reconsider our view if any ongoing discussion brings to light new arguments or if FP is unable to back up any claims they made, but so far I haven’t seen any red or even orange flags.

  (3) The above should be enough for GWWC to not prioritize taking any action related to StrongMinds at the moment, but I happen to have a bit more context here than usual as I was a co-author on the 2019 FP report on StrongMinds, and none of the five issues you raise are a surprise/​new to me or change my view of StrongMinds very much. Very briefly on each (note: I don’t have much time /​ will mostly leave this to Matt /​ some of my knowledge may be outdated or my memory may be off):

  • I agree the overall quality of evidence is far short from e.g. GiveWell’s standards (cf. Matt’s comments—and would have agreed on this back in 2019. At this point, I certainly wouldn’t take FP’s 2019 cost-effectiveness analysis literally: I would deflate the results by quite a bit to account for quality of evidence, and I know FP have done so internally for the past ~2 years at least. However, AFAIK such accounting—done reasonably—isn’t enough to change the overall conclusion of StrongMinds meeting the cost-effectiveness bar in wellbeing terms. I should also note that HLI’s cost-effectiveness analysis seems to take into account more pieces of evidence, though I haven’t reviewed it; just skimmed it.

  • As you say yourself, The 2019 FP report already accounted for social desirability bias to some extent, and it further highlights this bias as one of its key uncertainties (section 3.8, p.31).

  • I disagree with depression being overweighted here for various reasons, including that DALYs plausibly underweight mental health (see section 1, p.8-9 of the FP mental health report. Also note that HLI’s recent analysis—AFAIK—doesn’t rely on DALY’s in any way.

  • I don’t think the reasons StrongMinds mention for not collecting more evidence (than they already are) are as unreasonable as you seem to think. I’d need to delve more into the specifics to form a view here, but just want to reiterate StrongMinds’s first reason that running high-quality studies is generally very expensive, and may often not be the best decision for a charity from a cost-effectiveness standpoint. Even though I think the sector as a whole could probably still do with more (of the right type of) evidence generation, from my experience I would guess it’s also relatively common charities collect more evidence (of the wrong kind) than would be optimal.

  • I don’t like what I see in at least some of the examples of communication you give—and if I were evaluating StrongMinds currently I would certainly want to give them this feedback (in fact I believe I did back in 2018, which I think prompted them to make some changes). However, though I’d agree that these provide some update on how thoroughly one should check claims StrongMinds makes more generally, I don’t think they should meaningfully change one’s view on the cost-effectiveness of StrongMinds’s core work.

  (4) Jeff suggested (and some others seem to like) the idea of GWWC changing its inclusion criteria and only recommending/​top-rating organisations for which an up-to-date public evaluation is available. This is something we discussed internally in the lead-up to this giving season, but we decided against it and I still feel that was and is the right decision (though I am open to further discussion/​arguments):

  • There are only very few charities for which full public and up-to-date evaluations are available, and coverage for some worldviews/​promising cause areas is structurally missing. In particular, there are currently hardly any full public and up-to-date evaluations in the mental health/​subjective well-being, longtermist and “meta” spaces. And note that - by this standard—we wouldn’t be able to recommend any funds except for those just regranting to already-established recommendations.

  • If the main reason for this was that we don’t know of any cost-effective places to donate in these areas/​according to these worldviews, I would have agreed that we should just go with what we know or at least highlight that standards are much lower in these areas.

  • However, I don’t think this is the case: we do have various evaluators/​grantmakers looking into these areas (though too few yet IMO!) and arguably identifying very cost-effective donation opportunities (in expectation), but they often don’t prioritise sharing these findings publicly or updating public evaluations regularly. Having worked at one of those myself (FP), my impression is this is generally for very good reasons, mainly related to resource constraints/​prioritisation as Jeff notes himself.

  • In an ideal world—where these resource constraints wouldn’t exist—GWWC would only recommend charities for which public, up-to-date evaluations are available. However, we do not live in that ideal world, and as our goal is primarily to provide guidance on what are the best places to give to according to a variety of worldviews, rather than what are the best explainable/​publicly documented places to give, I think the current policy is the way to go.

  • Obviously it is very important that we are transparent about this, which we aim to do by clearly documenting our inclusion criteria, explaining why we rely on our trusted evaluators, and highlighting the evidence that is publicly available for each individual charity. Providing this transparency has been a major focus for us this giving season, and though I think we’ve made major steps in the right direction there’s probably still room for improvement: any feedback is very welcome!

  • Note that one reason why more public evaluations would seem to be good/​necessary is accountability: donors can check and give feedback on the quality of evaluations, providing the right incentives and useful information to evaluators. This sounds great in theory, but in my experience public evaluation reports are almost never read by donors (this post is an exception, which is why I’m so happy with it, even though I don’t agree with the author’s conclusions), and they are a very high resource cost to create and maintain—in my experience writing a public report can take up about half of the total time spent on an evaluation (!). This leaves us with an accountability and transparency problem that I think is real, and which is one of the main reasons for our planned research direction this year at GWWC.

  Lastly, FWIW I agree that we actively recommend StrongMinds (and this is our intention), even though we generally recommend donors to give to funds over individual charities.

  I believe this covers (nearly) all of the GWWC-related comments I’ve seen here, but please let me know if I’ve missed anything!

  What links here?

  • GWWC Should Re­quire Public Char­ity Evaluations by Jeff Kaufman 🔸 (Jan 9, 2023, 8:11 PM; 93 points)
  • Sjir Hoeijmakers🔸's comment on StrongMinds should not be a top-rated char­ity (yet) by Simon_M (Dec 29, 2022, 10:36 AM; 52 points)
  • Sjir Hoeijmakers🔸's comment on GWWC Should Re­quire Public Char­ity Evaluations by Jeff Kaufman 🔸 (Jan 10, 2023, 12:04 AM; 46 points)
  • GWWC Should Re­quire Public Char­ity Evaluations by jefftk (LessWrong; Jan 9, 2023, 8:10 PM; 28 points)
  • Sjir Hoeijmakers🔸's comment on StrongMinds should not be a top-rated char­ity (yet) by Simon_M (Jan 3, 2023, 3:20 PM; 18 points)
  • Sjir Hoeijmakers🔸's comment on StrongMinds should not be a top-rated char­ity (yet) by Simon_M (Jan 3, 2023, 3:32 PM; 10 points)
  • Sjir Hoeijmakers🔸's comment on StrongMinds should not be a top-rated char­ity (yet) by Simon_M (Jan 3, 2023, 3:50 PM; 3 points)

• Sjir Hoeijmakers🔸Dec 29, 2022, 1:41 PM

  29 points

  7 ∶ 1

  in reply to: Nathan Young’s comment on: StrongMinds should not be a top-rated char­ity (yet)

  As I tried to clarify above, this is not a case of secret info having much—if any—bearing on a recommendation. As far as I’m aware, nearly all decision-relevant information is and has been available publicly, and where it isn’t Matt has already begun clarifying things and has offered to provide more context next week (see discussion between him and Simon above). I certainly can’t think of any secret info that is influencing GWWC’s decision here.

  FWIW my personal forecast wouldn’t be very far from the current market forecast (probably closer to 30%), not because I think the current recommendation decision is wrong but for a variety of reasons, incl. StrongMinds’ funding gaps being filled to a certain extent by 2025; new data from the abovementioned RCT; the research community finding even better funding opportunities etc.

  I’m fine with the wording: it’s technically “top-rated charity” currently but both naming and system may change over the coming years, as we’ll hopefully be ramping up research efforts.

• Sjir Hoeijmakers🔸Dec 29, 2022, 12:25 PM

  30 points

  14 ∶ 1

  in reply to: Nathan Young’s comment on: StrongMinds should not be a top-rated char­ity (yet)

  Tbh I think this is a bit unfair: his criticism isn’t being disregarded at all. He received a substantial reply from FP’s research director Matt Lerner—even while he’s on holiday—within a day, and Matt seems very happy to discuss this further when he’s back to work.

  I should also add that almost all of the relevant work is in fact public, incl. the 2019 report and HLI’s analysis this year. I don’t think what FP has internally is crucial to interpreting Matt’s responses.

  I do like the forecasting idea though :).

• Sjir Hoeijmakers🔸Dec 29, 2022, 10:36 AM

  52 points

  8 ∶ 6

  on: StrongMinds should not be a top-rated char­ity (yet)

  Edit 03-01-23: I have now replied more elaborately here
  
  Hi Simon, thanks for this post! I’m research director at GWWC, and we really appreciate people engaging with our work like this and scrutinising it.
  
  I’m on holiday currently and won’t be able to reply much more in the coming few days, but will check this page again next Tuesday at the latest to see if there’s anything more I/​the GWWC team need to get back on.
  
  For now, I’ll just very quickly address your two key claims that GWWC shouldn’t have recommended StrongMinds as a top-rated charity and that we should remove it now, both of which I disagree with.
  
  Our process and criteria for making charity recommendations are outlined here. Crucially, note that we generally don’t do (and don’t have capacity to do) individual charity research: we almost entirely rely on our trusted evaluators—including Founders Pledge—for our recommendations. As a research team, we plan to specialize in providing guidance on which evaluators to rely rather than in doing individual charity evaluation research.
  
  In the case of StrongMinds, they are a top-rated charity primarily because Founders Pledge recommended them to us, as you highlight. There were no reasons for us to doubt the quality of FP’s research supporting this recommendation at the time, and though you make a few good points on StrongMinds that showcase some of the uncertainties/​caveats/​counterarguments involved in this recommendation, I think Matt Lerner adequately addresses these in his reply above, so I don’t see reason for us to consider diverging from FP’s recommendation now either.
  
  I do have a takeaway from your post on our communication. On our website we currently state
  
  “StrongMinds meets our criteria to be a top-rated charity because one of our trusted evaluators, Founders Pledge, has conducted an extensive evaluation highlighting its cost-effectiveness as part of its report on mental health.”
  
  We then also highlight HLI’s recent report as evidence supporting this recommendation. However, I think we should have been/​be clearer on the fact that it’s not only FP’s 2019 report that supports this recommendation, but their ongoing research, some of which isn’t public yet (even though the report still represents their overall view, as Matt Lerner mentions above). I’ll make sure to edit this.
  
  Thanks again for your post, and I look forward to engaging more next week at the latest if there are any other comments/​questions you or anyone else have.

  What links here?

  • Jeff Kaufman 🔸's comment on StrongMinds should not be a top-rated char­ity (yet) by Simon_M (Dec 31, 2022, 2:24 PM; 46 points)
  • Simon_M's comment on StrongMinds should not be a top-rated char­ity (yet) by Simon_M (Jan 2, 2023, 10:52 AM; 9 points)

What are the best char­i­ties to donate to in 2022?

• Sjir Hoeijmakers🔸Dec 6, 2022, 9:44 AM

  4 points

  2 ∶ 0

  in reply to: Hendrik Broering’s comment on: “Eval­u­at­ing the eval­u­a­tors”: GWWC’s re­search direction

  Thanks for your comment Hendrik!

  To address this, I think it’s important to look at the value each additional layer of evaluation provides. It seems (with the multitude of evaluators and fundraisers) we are now at a point where at least some work in the second layer is necessary/​useful, but I don’t think a third layer would currently be justified (with 0-1 organisations active in the second layer).

  Another way to see this is: the “turtles all the way down” concern already works for the first layer of evaluators (why do we need one if charities are already evaluating themselves and reporting on their impact? who is evaluating these evaluators?): the relevant question is whether the layer adds enough value, which this first layer clearly does (given how many charities and donors there are and the lack of public and independent information available on how they compare), and I argue above the second does as well.

  FWIW I don’t think this second layer should be fully or forever centralised in GWWC, and I see some value in more fundraising organisations having at least some research capacity to determine their recommendations, but we need to start somewhere and there are diminishing returns to adding more. Relatedly, I should say that I don’t expect fundraising organisations to just “listen to whatever GWWC says”: we provide recommendations and guidance, and these organisations may use that to inform their choices (which is a significant improvement to having no guidance at all to choose among evaluators).

Why Giv­ing What We Can recom­mends us­ing ex­pert-led char­i­ta­ble funds

• Sjir Hoeijmakers🔸Nov 25, 2022, 11:34 AM

  4 points

  1 ∶ 0

  in reply to: Jesus de Sivar’s comment on: “Eval­u­at­ing the eval­u­a­tors”: GWWC’s re­search direction

  Thank you both for offering to help! I’m not yet clear on whether it’ll make sense to work with volunteers on this, but it is certainly something we’ll consider. Could you please indicate your interest by filling out this form? (select “skilled volunteering”-->”impact analysis and evaluation”)

  Conditional on fundraising for GWWC’s 2023 budget, we’ll very likely hire an extra researcher to work on this early next year. If this is something you’d be interested in as well, please do feel free to reach out at sjir@givingwhatwecan.org and I’ll let you know once the position opens up for applications.

“Eval­u­at­ing the eval­u­a­tors”: GWWC’s re­search direction

An­nounc­ing GWWC’s new giv­ing recommendations

• Sjir Hoeijmakers🔸Sep 2, 2022, 9:19 AM

  1 point

  0 ∶ 0

  in reply to: Brendon_Wong’s comment on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Thank you!

• Sjir Hoeijmakers🔸Mar 9, 2022, 9:09 PM

  6 points

  0 ∶ 0

  in reply to: Sjir Hoeijmakers🔸’s comment on: In cur­rent EA, scal­a­bil­ity matters

  I also think it’s worth stressing that the best alternative to finding a great (above-bar) option to spend money on now is not to spend on options below the bar, but to wait /​ keep looking and spend it at an above-bar opportunity later (and ideally invest to give while you’re at it).

  In your example, this cashes out (roughly) in us using Research multiple times to find as many Alpha-like projects as possible and fund those, and to only start looking for and funding Beta-like projects when there are no more Alpha-like projects to find. Even if there is only one extra Alpha and one extra Beta to find, it’s better (with the parameters as provided in your example) to find and fund that Alpha and find and fund Beta, than to find at fund only one of the two.

  Cases somewhat akin to “you can only use Research for either Alpha or Beta” can occur, but only under very specific conditions, e.g. when opportunities are time-sensitive and/​or when there is a very tight bottleneck on research resources (=strongly increasing marginal costs to doing research), which might in fact be the case currently.

  (As a side point: given the option of investing to give, it’s important to “set” the bar taking into account our expectations of how cost-effective future opportunities will be, investment benefits one can achieve in the meantime, value drift and expropriation risks etc.)

• Sjir Hoeijmakers🔸Mar 9, 2022, 9:08 PM

  7 points

  0 ∶ 0

  on: In cur­rent EA, scal­a­bil­ity matters

  I would like to push back a bit, as I don’t think it’s true that scalability per se matters more now than it did in the past.

  Instead, I think the availability of more funding has pushed down the cost-effectiveness bar for funding opportunities, thereby “unlocking” some new worthy funding opportunities, including some very scalable ones.

  To see this, consider that the added value of discovering/​creating any new funding opportunity for the community is roughly given by (not accounting for diminishing returns when spending at bar level):

  “value created by adding a new funding opportunity” = (“average cost-effectiveness of the opportunity”—“current cost-effectiveness bar”) * “room for funding of the opportunity”

  I.e. what you’re effectively doing by adding a new opportunity is improving the cost-effectiveness of money that would have otherwise been spent at the bar level.

  This implies that any opportunity that is above the current bar in terms of its cost-effectiveness can be worth discovering if it’s scalable enough. But that is nothing new: it was true as much in 2010-2014 as it is now. It’s just that the bar was higher, so some very scalable but below-bar opportunities weren’t worth discovering back then but are now.

• Sjir Hoeijmakers🔸Mar 4, 2022, 10:42 AM

  11 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  A social media platform with better incentives

  Epistemic Institutions, Values and Reflective Processes

  Social media has arguably become a major way in which people consume information and develop their values, and the most popular platforms are far from optimally set up to bring people closer to truthfulness or altruistic ends. We’d love to see experiments with social media platforms that provide more pro-social incentives and yet have the potential to reach a large audience.

• Sjir Hoeijmakers🔸Mar 4, 2022, 10:42 AM

  22 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Institutions as coordination mechanisms

  Artificial Intelligence, Biorisk and Recovery from Catastrophe, Great Power Relations, Space Governance, Values and Reflective Processes

  A lot of major problems—such as biorisk, AI governance risk and the risks of great power war—can be modeled as coordination problems, and may be at least partially solved via better coordination among the relevant actors. We’d love to see experiments with institutions that use mechanism design to allow actors to coordinate better. One current example of such an institution is NATO: Article 5 is a coordination mechanism that aligns the interests of NATO member states. But we could create similar institutions for e.g. biorisk, where countries commit to a matching mechanism—where “everyone acts in a certain way if everyone else does”—with costs imposed to defectors to solve a tragedy of the commons dynamic.

• Sjir Hoeijmakers🔸Mar 4, 2022, 10:41 AM

  22 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Experiments with and within video games

  Values and Reflective Processes, Empowering Exceptional People

  Video games are a powerful tool to reach hundreds of millions of people, an engine of creativity and innovation, and a fertile ground for experimentation. We’d love to see experiments with and within video games that help create new tools to address major issues. For instance, we’d love experiments with new governance and incentive systems and institutions, new ways to educate people about pressing problems, games that simulate actual problems and allow players to brainstorm solutions, and games that help identify and recruit exceptional people.

• Sjir Hoeijmakers🔸Mar 4, 2022, 10:41 AM

  10 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Representation of future generations within major institutions

  Values and Reflective Processes, Epistemic Institutions

  We think at least part of the issues facing us today would be better handled if there was less political short-termism, and if there were more incentives for major political and non-political institutions to take into account the interests of future generations. One way to address this is to establish explicit representation of future generations in these institutions through strategic advocacy, which can be done in many ways and has been piloted in the past few decades.

• Sjir Hoeijmakers🔸Mar 4, 2022, 10:40 AM

  13 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Scaling successful policies

  Biorisk and Recovery from Catastrophe, Economic Growth

  Information flow across institutions (including national governments) is far from optimal, and there could be large gains in simply scaling what already works in some places. We’d love to see an organization that takes a prioritized approach to researching which policies are currently in place to address major global issues, identifying which of these are most promising to bring to other institutions and geographies, and then bringing these to the institutions and geographies where they are most needed.