Sjir Hoeijmakers🔸

Karma: 2,777

Sjir Hoeijmakers🔸Jan 11, 2023, 10:50 PM
22 points
4 ∶ 0
on: GWWC’s Handling of Conflicting Funding Bars
Hi Jeff, thanks for another helpful suggestion! (previous one) In this case, I agree there is room for improvement, and we’ll aim to update our inclusion criteria throughout 2023 and to provide more details where we can. The reasons this particular thing isn’t in there yet are (1) we simply haven’t prioritised writing this out yet, as it is quite detailed/applies to just one of the four “cause areas” we cover and to just one evaluator (FP) and it would require quite a bit of extra context to explain to the broad audience we try to reach (e.g. laying out what we mean by these ratings, what is measured, limitations etc.) which was beyond the scope we were able to set for this giving season (during which we already had a lot of things to update with a small team) (2) this relies on FP’s internal ratings and I’m not sure whether FP would want these to be public, e.g. given how rough they are/how much context they need/to avoid over-updating, but I’ll leave it to them to respond on that point.

On SCI specifically, my recollection is that GiveWell’s November 2021 analysis is no longer current/that SCI has made significant changes to its programme since that evaluation was done, though I’m not 100% sure. In any case, for the deworming charities more generally we decided to stick closely with our inclusion criteria, which meant not recommending them as top-rated (because they don’t clear GW’s nor FP’s current bar at this moment/they weren’t recommended to us by either) and listing until we receive FP’s updated estimates. This seemed the better option in particular because we know FP will have updated estimates relatively soon, and IIRC they don’t expect all deworming charities to necessarily clear the 3x bar (though again referring to them here for a response, if they are willing to comment before finalizing the evaluation). Hope that clarifies!

Sjir Hoeijmakers🔸Jan 10, 2023, 12:04 AM
46 points
8 ∶ 1
on: GWWC Should Require Public Charity Evaluations
Thanks again for this suggestion Jeff! However, for reasons mostly outlined in my comment here (under (4)) GWWC’s position remains that we should not restrict charity recommendations only to those who have a recent public evaluation available. I’d be interested in any more arguments coming out of this discussion that would update our view though, and these could feed into a revision of our inclusion criteria later this year.

There’s one thing I’d like to add—based on the emphasis of your new post: as you mention, there are multiple reasons why people choose to donate to charities over funds, even while we generally think that donating to funds will be the higher-impact option. I think I have lower credence than you seem to have in “not trusting funds” being the most prominent one, but even if it is, I don’t think the current situation is problematic for donors for whom this is the main reason: those donors can easily see whether a particular top-rated charity has a recent public evaluation available (e.g. this will be highlighted on its charity page on the GWWC website), and adjust their decisions accordingly. By keeping the current policy, the “top-rated” label remains representative of where we expect money will actually do the most good, rather than it being adjusted for a subgroup of donors who have a lower trust in funds.

(As an aside, I don’t see why the other reasons you mention for giving to charities (e.g. tax deductibility) would be more characteristic of “sophisticated and committed” donors than having a view on whether or not to trust particular evaluators/funds)

Sjir Hoeijmakers🔸Jan 6, 2023, 6:43 PM
2 points
0 ∶ 0
in reply to: Jeff Kaufman 🔸’s comment on: StrongMinds should not be a top-rated charity (yet)
Yeah agreed. And another one could be as a way of getting involved more closely with a particularly charity when one wants to provide other types of support (advice, connections) in addition to funding. E.g. even though I don’t think this should help a lot, I’ve anecdotally found it helpful to fund individual charities that I advise, because putting my personal donation money on the line motivates me to think even more critically about how the charity could best use its limited resources.

Thanks again for engaging in this discussion so thoughtfully Jeff! These types of comments and suggestions are generally very helpful for us (even if I don’t agree with these particular ones).

Sjir Hoeijmakers🔸Jan 4, 2023, 11:39 AM
4 points
1 ∶ 0
in reply to: Jeff Kaufman 🔸’s comment on: StrongMinds should not be a top-rated charity (yet)
I think trust is one of the reasons why a donor may or may not decide to give to a fund over a charity, but there are others as well, e.g. a preference for supporting more specific causes or projects. I expect donors with these other reasons (who trust evaluators/fund managers but would still prefer to give to individual charities (as well)) will value charity recommendations in areas for which there are no public and up-to-date evaluations available.

I think what I’d like to see is funds saying something like, if you want to support our work the best thing is to give to the fund, but the second best is to support orgs X, Y, Z. This recommendation wouldn’t be based on a public evaluation, but just on your trust in them as a funder.

Note that this is basically equivalent to the current situation: we recommend funds over charities but highlight supporting charities as the second-best thing, based on recommendations of evaluators (who are often also fund managers in their area).

Sjir Hoeijmakers🔸Jan 3, 2023, 4:08 PM
8 points
2 ∶ 0
in reply to: Jeff Kaufman 🔸’s comment on: StrongMinds should not be a top-rated charity (yet)

FWIW I’m not asking for immediate action, but a reconsideration of the criteria for endorsing a recommendation from a trusted evaluator.

I wasn’t suggesting you were, but Simon certainly was. Sorry if that wasn’t clear.

In cases where a field has only non-public or stale evaluations then fund managers are still in a position to consider non-public information and the general state of the field, check in with evaluators about what kind of stale the current evaluations are at, etc. And in these cases I think the best you can do is say that this is a field where GWWC currently doesn’t have any recommendations for specific charities, and only recommends giving via funds.

As GWWC gets its recommendations and information directly from evaluators (and aims to update its recommendations regularly), I don’t see a meaningful difference here between funds vs charities in fields where there are public up-to-date evaluations and where there aren’t: in both cases, GWWC would recommend giving to funds over charities, and in both cases we can also highlight the charities that seem to be the most cost-effective donation opportunities based on the latest views of evaluators. GWWC provides a value-add to donors here, given some of these recommendations wouldn’t be available to them otherwise (and many donors probably still prefer to donate to charities over donating to funds / might not donate otherwise).

Sjir Hoeijmakers🔸Jan 3, 2023, 3:50 PM
3 points
0 ∶ 0
in reply to: Jason’s comment on: StrongMinds should not be a top-rated charity (yet)
Thanks for the suggestion Jason, though I hope the longer comment I just posted will clarify why I think this wouldn’t be worth doing.

Sjir Hoeijmakers🔸Jan 3, 2023, 3:32 PM
10 points
1 ∶ 0
in reply to: Nathan Young’s comment on: StrongMinds should not be a top-rated charity (yet)
Hi Nathan, I don’t think the results of the RCT will be negative, just that they could cause us to update (in either direction) which adds uncertainty, though I’d admit that at a <50% forecast this could plausibly increase my forecast rather than lower it (though this isn’t immediately clear; depends on the interactions with the other reasons).

And I hope the more elaborate reply I just wrote to Simon answers your remaining question.

Sjir Hoeijmakers🔸Jan 3, 2023, 3:20 PM
18 points
4 ∶ 0
in reply to: Jeff Kaufman 🔸’s comment on: StrongMinds should not be a top-rated charity (yet)
Thanks Jeff, I think your summary is helpful and broadly correct, except for two (somewhat relevant) details:
- GWWC didn’t recommend SM based on FP’s recommendation in 2019 but based on FP’s decision to still recommend SM as of this giving season (which is based on FP’s internal re-evaluation of SM).
- I don’t expect there to be any new, decision-relevant information in FP’s recent internal re-evaluation that isn’t captured by the 2019 report + the recent HLI analysis (but I’m not sure about this—Matt can correct me if I’m wrong, though also see his comment here). Obviously the internal re-evaluation has extra “information” in the sense that FP has reviewed the HLI analysis, converted metrics to their new system, and run some extra tests, as Matt has explained, so maybe this is just semantics, but I think it’s relevant to the extent that a crux would be “FP is still recommending SM because of something only they know”.
I understand the reasons for your suggestion w.r.t. GWWC’s inclusion criteria—we’ve seriously considered doing this before—but I explain at length why I still think we shouldn’t under (4) here. Would welcome any further comments if you disagree!

Sjir Hoeijmakers🔸Jan 3, 2023, 2:57 PM
88 points
15 ∶ 4
in reply to: Simon_M’s comment on: StrongMinds should not be a top-rated charity (yet)
Hi Simon,

I’m back to work and able to reply with a bit more detail now (though also time-constrained as we have a lot of other important work to do this new year :)).

I still do not think any (immediate) action on our part is required. Let me lay out the reasons why:

(1) Our full process and criteria are explained here. As you seem to agree with from your comment above we need clear and simple rules for what is and what isn’t included (incl. because we have a very small team and need to prioritize). Currently a very brief summary of these rules/the process would be: first determine which evaluators to rely on (also note our plans for this year) and then rely on their recommendations. We do not generally have the capacity to review individual charity evaluations, and would only do so and potentially diverge from a trusted evaluator’s recommendation under exceptional circumstances. (I don’t believe we have had such a circumstance this giving season, but may misremember)

(2) There were no strong reasons to diverge with respect to FP’s recommendation of StrongMinds at the time they recommended them—or to do an in-depth review of FP’s evaluation ourselves—and I think there still aren’t. As I said before, you make a few useful points in your post but I think Matt’s reaction and the subsequent discussion satisfactorily explain why Founders Pledge chose to recommend StrongMinds and why your comments don’t (immediately) change their view on this: StrongMinds doesn’t need to meet GiveWell-tier levels of confidence and easily clears FP’s bar in expectation—even with the issues you mention having been taken into account—and nearly all the decision-relevant reasoning is already available publicly in the 2019 report and HLI’s recent review. I would of course be very interested and we could reconsider our view if any ongoing discussion brings to light new arguments or if FP is unable to back up any claims they made, but so far I haven’t seen any red or even orange flags.

(3) The above should be enough for GWWC to not prioritize taking any action related to StrongMinds at the moment, but I happen to have a bit more context here than usual as I was a co-author on the 2019 FP report on StrongMinds, and none of the five issues you raise are a surprise/new to me or change my view of StrongMinds very much. Very briefly on each (note: I don’t have much time / will mostly leave this to Matt / some of my knowledge may be outdated or my memory may be off):
- I agree the overall quality of evidence is far short from e.g. GiveWell’s standards (cf. Matt’s comments—and would have agreed on this back in 2019. At this point, I certainly wouldn’t take FP’s 2019 cost-effectiveness analysis literally: I would deflate the results by quite a bit to account for quality of evidence, and I know FP have done so internally for the past ~2 years at least. However, AFAIK such accounting—done reasonably—isn’t enough to change the overall conclusion of StrongMinds meeting the cost-effectiveness bar in wellbeing terms. I should also note that HLI’s cost-effectiveness analysis seems to take into account more pieces of evidence, though I haven’t reviewed it; just skimmed it.
- As you say yourself, The 2019 FP report already accounted for social desirability bias to some extent, and it further highlights this bias as one of its key uncertainties (section 3.8, p.31).
- I disagree with depression being overweighted here for various reasons, including that DALYs plausibly underweight mental health (see section 1, p.8-9 of the FP mental health report. Also note that HLI’s recent analysis—AFAIK—doesn’t rely on DALY’s in any way.
- I don’t think the reasons StrongMinds mention for not collecting more evidence (than they already are) are as unreasonable as you seem to think. I’d need to delve more into the specifics to form a view here, but just want to reiterate StrongMinds’s first reason that running high-quality studies is generally very expensive, and may often not be the best decision for a charity from a cost-effectiveness standpoint. Even though I think the sector as a whole could probably still do with more (of the right type of) evidence generation, from my experience I would guess it’s also relatively common charities collect more evidence (of the wrong kind) than would be optimal.
- I don’t like what I see in at least some of the examples of communication you give—and if I were evaluating StrongMinds currently I would certainly want to give them this feedback (in fact I believe I did back in 2018, which I think prompted them to make some changes). However, though I’d agree that these provide some update on how thoroughly one should check claims StrongMinds makes more generally, I don’t think they should meaningfully change one’s view on the cost-effectiveness of StrongMinds’s core work.
(4) Jeff suggested (and some others seem to like) the idea of GWWC changing its inclusion criteria and only recommending/top-rating organisations for which an up-to-date public evaluation is available. This is something we discussed internally in the lead-up to this giving season, but we decided against it and I still feel that was and is the right decision (though I am open to further discussion/arguments):
- There are only very few charities for which full public and up-to-date evaluations are available, and coverage for some worldviews/promising cause areas is structurally missing. In particular, there are currently hardly any full public and up-to-date evaluations in the mental health/subjective well-being, longtermist and “meta” spaces. And note that - by this standard—we wouldn’t be able to recommend any funds except for those just regranting to already-established recommendations.
- If the main reason for this was that we don’t know of any cost-effective places to donate in these areas/according to these worldviews, I would have agreed that we should just go with what we know or at least highlight that standards are much lower in these areas.
- However, I don’t think this is the case: we do have various evaluators/grantmakers looking into these areas (though too few yet IMO!) and arguably identifying very cost-effective donation opportunities (in expectation), but they often don’t prioritise sharing these findings publicly or updating public evaluations regularly. Having worked at one of those myself (FP), my impression is this is generally for very good reasons, mainly related to resource constraints/prioritisation as Jeff notes himself.
- In an ideal world—where these resource constraints wouldn’t exist—GWWC would only recommend charities for which public, up-to-date evaluations are available. However, we do not live in that ideal world, and as our goal is primarily to provide guidance on what are the best places to give to according to a variety of worldviews, rather than what are the best explainable/publicly documented places to give, I think the current policy is the way to go.
- Obviously it is very important that we are transparent about this, which we aim to do by clearly documenting our inclusion criteria, explaining why we rely on our trusted evaluators, and highlighting the evidence that is publicly available for each individual charity. Providing this transparency has been a major focus for us this giving season, and though I think we’ve made major steps in the right direction there’s probably still room for improvement: any feedback is very welcome!
- Note that one reason why more public evaluations would seem to be good/necessary is accountability: donors can check and give feedback on the quality of evaluations, providing the right incentives and useful information to evaluators. This sounds great in theory, but in my experience public evaluation reports are almost never read by donors (this post is an exception, which is why I’m so happy with it, even though I don’t agree with the author’s conclusions), and they are a very high resource cost to create and maintain—in my experience writing a public report can take up about half of the total time spent on an evaluation (!). This leaves us with an accountability and transparency problem that I think is real, and which is one of the main reasons for our planned research direction this year at GWWC.
Lastly, FWIW I agree that we actively recommend StrongMinds (and this is our intention), even though we generally recommend donors to give to funds over individual charities.

I believe this covers (nearly) all of the GWWC-related comments I’ve seen here, but please let me know if I’ve missed anything!
What links here?

Sjir Hoeijmakers🔸Dec 29, 2022, 1:41 PM
29 points
7 ∶ 1
in reply to: Nathan Young’s comment on: StrongMinds should not be a top-rated charity (yet)
As I tried to clarify above, this is not a case of secret info having much—if any—bearing on a recommendation. As far as I’m aware, nearly all decision-relevant information is and has been available publicly, and where it isn’t Matt has already begun clarifying things and has offered to provide more context next week (see discussion between him and Simon above). I certainly can’t think of any secret info that is influencing GWWC’s decision here.

FWIW my personal forecast wouldn’t be very far from the current market forecast (probably closer to 30%), not because I think the current recommendation decision is wrong but for a variety of reasons, incl. StrongMinds’ funding gaps being filled to a certain extent by 2025; new data from the abovementioned RCT; the research community finding even better funding opportunities etc.

I’m fine with the wording: it’s technically “top-rated charity” currently but both naming and system may change over the coming years, as we’ll hopefully be ramping up research efforts.

Sjir Hoeijmakers🔸Dec 29, 2022, 12:25 PM
30 points
14 ∶ 1
in reply to: Nathan Young’s comment on: StrongMinds should not be a top-rated charity (yet)
Tbh I think this is a bit unfair: his criticism isn’t being disregarded at all. He received a substantial reply from FP’s research director Matt Lerner—even while he’s on holiday—within a day, and Matt seems very happy to discuss this further when he’s back to work.

I should also add that almost all of the relevant work is in fact public, incl. the 2019 report and HLI’s analysis this year. I don’t think what FP has internally is crucial to interpreting Matt’s responses.

I do like the forecasting idea though :).

Sjir Hoeijmakers🔸Dec 29, 2022, 10:36 AM
52 points
8 ∶ 6
on: StrongMinds should not be a top-rated charity (yet)
Edit 03-01-23: I have now replied more elaborately here

Hi Simon, thanks for this post! I’m research director at GWWC, and we really appreciate people engaging with our work like this and scrutinising it.

I’m on holiday currently and won’t be able to reply much more in the coming few days, but will check this page again next Tuesday at the latest to see if there’s anything more I/the GWWC team need to get back on.

For now, I’ll just very quickly address your two key claims that GWWC shouldn’t have recommended StrongMinds as a top-rated charity and that we should remove it now, both of which I disagree with.

Our process and criteria for making charity recommendations are outlined here. Crucially, note that we generally don’t do (and don’t have capacity to do) individual charity research: we almost entirely rely on our trusted evaluators—including Founders Pledge—for our recommendations. As a research team, we plan to specialize in providing guidance on which evaluators to rely rather than in doing individual charity evaluation research.

In the case of StrongMinds, they are a top-rated charity primarily because Founders Pledge recommended them to us, as you highlight. There were no reasons for us to doubt the quality of FP’s research supporting this recommendation at the time, and though you make a few good points on StrongMinds that showcase some of the uncertainties/caveats/counterarguments involved in this recommendation, I think Matt Lerner adequately addresses these in his reply above, so I don’t see reason for us to consider diverging from FP’s recommendation now either.

I do have a takeaway from your post on our communication. On our website we currently state

“StrongMinds meets our criteria to be a top-rated charity because one of our trusted evaluators, Founders Pledge, has conducted an extensive evaluation highlighting its cost-effectiveness as part of its report on mental health.”

We then also highlight HLI’s recent report as evidence supporting this recommendation. However, I think we should have been/be clearer on the fact that it’s not only FP’s 2019 report that supports this recommendation, but their ongoing research, some of which isn’t public yet (even though the report still represents their overall view, as Matt Lerner mentions above). I’ll make sure to edit this.

Thanks again for your post, and I look forward to engaging more next week at the latest if there are any other comments/questions you or anyone else have.
What links here?
- Jeff Kaufman 🔸's comment on StrongMinds should not be a top-rated charity (yet) by Simon_M (Dec 31, 2022, 2:24 PM; 46 points)
- Simon_M's comment on StrongMinds should not be a top-rated charity (yet) by Simon_M (Jan 2, 2023, 10:52 AM; 9 points)

Sjir Hoeijmakers🔸Dec 6, 2022, 9:44 AM
4 points
2 ∶ 0
in reply to: Hendrik Broering’s comment on: “Evaluating the evaluators”: GWWC’s research direction
Thanks for your comment Hendrik!

To address this, I think it’s important to look at the value each additional layer of evaluation provides. It seems (with the multitude of evaluators and fundraisers) we are now at a point where at least some work in the second layer is necessary/useful, but I don’t think a third layer would currently be justified (with 0-1 organisations active in the second layer).

Another way to see this is: the “turtles all the way down” concern already works for the first layer of evaluators (why do we need one if charities are already evaluating themselves and reporting on their impact? who is evaluating these evaluators?): the relevant question is whether the layer adds enough value, which this first layer clearly does (given how many charities and donors there are and the lack of public and independent information available on how they compare), and I argue above the second does as well.

FWIW I don’t think this second layer should be fully or forever centralised in GWWC, and I see some value in more fundraising organisations having at least some research capacity to determine their recommendations, but we need to start somewhere and there are diminishing returns to adding more. Relatedly, I should say that I don’t expect fundraising organisations to just “listen to whatever GWWC says”: we provide recommendations and guidance, and these organisations may use that to inform their choices (which is a significant improvement to having no guidance at all to choose among evaluators).

Sjir Hoeijmakers🔸Nov 25, 2022, 11:34 AM
4 points
1 ∶ 0
in reply to: Jesus de Sivar’s comment on: “Evaluating the evaluators”: GWWC’s research direction
Thank you both for offering to help! I’m not yet clear on whether it’ll make sense to work with volunteers on this, but it is certainly something we’ll consider. Could you please indicate your interest by filling out this form? (select “skilled volunteering”-->”impact analysis and evaluation”)

Conditional on fundraising for GWWC’s 2023 budget, we’ll very likely hire an extra researcher to work on this early next year. If this is something you’d be interested in as well, please do feel free to reach out at sjir@givingwhatwecan.org and I’ll let you know once the position opens up for applications.

Sjir Hoeijmakers🔸Sep 2, 2022, 9:19 AM
1 point
0 ∶ 0
in reply to: Brendon_Wong’s comment on: The Future Fund’s Project Ideas Competition
Thank you!

Sjir Hoeijmakers🔸Mar 9, 2022, 9:09 PM
6 points
0 ∶ 0
in reply to: Sjir Hoeijmakers🔸’s comment on: In current EA, scalability matters
I also think it’s worth stressing that the best alternative to finding a great (above-bar) option to spend money on now is not to spend on options below the bar, but to wait / keep looking and spend it at an above-bar opportunity later (and ideally invest to give while you’re at it).

In your example, this cashes out (roughly) in us using Research multiple times to find as many Alpha-like projects as possible and fund those, and to only start looking for and funding Beta-like projects when there are no more Alpha-like projects to find. Even if there is only one extra Alpha and one extra Beta to find, it’s better (with the parameters as provided in your example) to find and fund that Alpha and find and fund Beta, than to find at fund only one of the two.

Cases somewhat akin to “you can only use Research for either Alpha or Beta” can occur, but only under very specific conditions, e.g. when opportunities are time-sensitive and/or when there is a very tight bottleneck on research resources (=strongly increasing marginal costs to doing research), which might in fact be the case currently.

(As a side point: given the option of investing to give, it’s important to “set” the bar taking into account our expectations of how cost-effective future opportunities will be, investment benefits one can achieve in the meantime, value drift and expropriation risks etc.)

Sjir Hoeijmakers🔸Mar 9, 2022, 9:08 PM
7 points
0 ∶ 0
on: In current EA, scalability matters
I would like to push back a bit, as I don’t think it’s true that scalability per se matters more now than it did in the past.

Instead, I think the availability of more funding has pushed down the cost-effectiveness bar for funding opportunities, thereby “unlocking” some new worthy funding opportunities, including some very scalable ones.

To see this, consider that the added value of discovering/creating any new funding opportunity for the community is roughly given by (not accounting for diminishing returns when spending at bar level):

“value created by adding a new funding opportunity” = (“average cost-effectiveness of the opportunity”—“current cost-effectiveness bar”) * “room for funding of the opportunity”

I.e. what you’re effectively doing by adding a new opportunity is improving the cost-effectiveness of money that would have otherwise been spent at the bar level.

This implies that any opportunity that is above the current bar in terms of its cost-effectiveness can be worth discovering if it’s scalable enough. But that is nothing new: it was true as much in 2010-2014 as it is now. It’s just that the bar was higher, so some very scalable but below-bar opportunities weren’t worth discovering back then but are now.

Sjir Hoeijmakers🔸Mar 4, 2022, 10:42 AM
11 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
A social media platform with better incentives

Epistemic Institutions, Values and Reflective Processes

Social media has arguably become a major way in which people consume information and develop their values, and the most popular platforms are far from optimally set up to bring people closer to truthfulness or altruistic ends. We’d love to see experiments with social media platforms that provide more pro-social incentives and yet have the potential to reach a large audience.

Sjir Hoeijmakers🔸Mar 4, 2022, 10:42 AM
22 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
Institutions as coordination mechanisms

Artificial Intelligence, Biorisk and Recovery from Catastrophe, Great Power Relations, Space Governance, Values and Reflective Processes

A lot of major problems—such as biorisk, AI governance risk and the risks of great power war—can be modeled as coordination problems, and may be at least partially solved via better coordination among the relevant actors. We’d love to see experiments with institutions that use mechanism design to allow actors to coordinate better. One current example of such an institution is NATO: Article 5 is a coordination mechanism that aligns the interests of NATO member states. But we could create similar institutions for e.g. biorisk, where countries commit to a matching mechanism—where “everyone acts in a certain way if everyone else does”—with costs imposed to defectors to solve a tragedy of the commons dynamic.

Sjir Hoeijmakers🔸Mar 4, 2022, 10:41 AM
22 points
0 ∶ 0
on: The Future Fund’s Project Ideas Competition
Experiments with and within video games

Values and Reflective Processes, Empowering Exceptional People

Video games are a powerful tool to reach hundreds of millions of people, an engine of creativity and innovation, and a fertile ground for experimentation. We’d love to see experiments with and within video games that help create new tools to address major issues. For instance, we’d love experiments with new governance and incentive systems and institutions, new ways to educate people about pressing problems, games that simulate actual problems and allow players to brainstorm solutions, and games that help identify and recruit exceptional people.