Students for High-Impact Charity: 2018 Update



By Bax­ter Bul­lock, Cather­ine Low, David Moss and Tee Barnett

Contents

Summary

Our Cur­rent Strategy

Sum­mary and In­ter­pre­ta­tion of Work­shop Data

Outreach Data

Quan­ti­ta­tive Sur­vey Data

Qual­i­ta­tive Results

Conclusion

Plans for Au­tumn 2018 and Beyond

The fol­low­ing post de­tails SHIC’s 2018 strate­gic shift, ev­i­dence gath­ered, and fu­ture plans. You can learn more about what SHIC does and to see our most re­cent pro­mo­tional video here.

Summary

Within the first half of 2018, Stu­dents for High-Im­pact Char­ity (SHIC) has suc­cess­fully pi­loted its work­shop model in 16 schools to 855 stu­dents in Van­cou­ver, al­low­ing for a strate­gic pivot to­wards a higher fidelity model for com­mu­ni­cat­ing key con­cepts, more ro­bust data col­lec­tion, and more op­por­tu­ni­ties for long-term stu­dent en­gage­ment. Our data thus far, con­sist­ing of both for­mal (out­reach data and the post-work­shop sur­vey) and in­for­mal (teacher, stu­dent and ad­minis­tra­tor re­ac­tions to the pro­gram) feed­back has helped af­firm the value of SHIC in sev­eral ways:

  • De­mand from both teach­ers and stu­dents for this type of pro­gram is sub­stan­tial.

  • The pro­gram is ap­peal­ing enough that the ma­jor­ity of schools have re­quested re­peat vis­its.

  • Our limited data shows broad suc­cess across a va­ri­ety of met­rics.

  • There is at least some in­ter­est from promis­ing stu­dents for fur­ther en­gage­ment.

We be­lieve this data sug­gests strong value in con­tin­u­ing to test key com­po­nents of the work­shop model.

While our goal in early 2018 was to mea­sure the broad de­mand for and ap­peal of our pro­gram, we’ve shifted our fo­cus for the up­com­ing school year (Septem­ber 2018) to­ward en­gag­ing promis­ing stu­dents in fur­ther ed­u­ca­tion and long-last­ing en­gage­ment. We be­lieve this should be the pri­mary met­ric for mea­sur­ing the suc­cess of our pro­gram.

Aside from the work­shop model, SHIC is main­tain­ing a scaled-down ver­sion of our Stu­dent Leader pro­gram due to a cou­ple of no­table suc­cesses. We will also ex­plore how the SHIC pro­gram can be im­ple­mented in a uni­ver­sity set­ting.

Our Cur­rent Strategy

In Au­tumn 2017, we re­leased a post de­tailing our rea­sons for turn­ing away from our Stu­dent Leader model of sup­port­ing in­ter­ested stu­dents in im­ple­ment­ing the SHIC pro­gram for their peers around the world. Our cur­rent work­shop-based model fo­cuses on higher-qual­ity con­tent de­liv­ery and data col­lec­tion.

Since our strate­gic pivot, SHIC car­ried out the fol­low­ing:

  • Pre­pared two ‘SHIC In­struc­tors’ to run work­shops in high schools.1

  • Re­for­mat­ted and con­soli­dated the SHIC pro­gram to fit into 3-5 hours of pro­gram­ming the amount of time we recom­mended schools provide for our work­shops.

  • Reached out to high schools through­out the greater Van­cou­ver area.

  • Pre­sented 41 work­shops in 16 schools to 855 stu­dents in the greater Van­cou­ver area, gath­er­ing data, in­clud­ing sur­veys and char­ity choices,2 in the pro­cess.

There are sev­eral rea­sons we’ve found this model to be a stronger fit with our over­ar­ch­ing goals:

1. Con­trol­ling con­tent de­liv­ery.

With our pre­vi­ous model, stu­dent lead­ers had the op­por­tu­nity to run as lit­tle or as much of the SHIC pro­gram as they wished, cherry-pick­ing el­e­ments of the pro­gram that suited their needs or start­ing the pro­gram with­out finish­ing it. While we agree that some of the pro­gram is bet­ter than none at all, we also feel like many were miss­ing out on the im­pact of SHIC as a co­he­sive nar­ra­tive; a pro­gram that ties many ideas into a set of over­ar­ch­ing con­cepts.

2. Hav­ing a pre­sen­ter equipped with the knowl­edge to an­swer ques­tions and re­spond to un­cer­tainty.

SHIC was cre­ated to spark in­ter­est and to provide foun­da­tional knowl­edge and skills. What a stu­dent chooses to build upon this foun­da­tion will vary with the in­di­vi­d­ual. SHIC In­struc­tors are knowl­edge­able enough in these con­cepts to guide stu­dents in the right di­rec­tion. Stu­dent lead­ers would be less equipped to ad­dress con­cerns on the spot, which could lead to mis­com­mu­ni­ca­tion or mi­s­un­der­stand­ing.

3. Con­trol­ling data col­lec­tion.

As we de­tailed in last year’s post, our abil­ity to col­lect post-pro­gram data was severely hin­dered by at­tri­tion. Most stu­dent lead­ers ei­ther failed to com­plete the en­tire pro­gram or were un­able to gather suffi­cient sur­vey data from par­ti­ci­pants. With our work­shop model, we have stronger as­surances that post-work­shop data will be col­lected, ei­ther through di­rect con­tact with the teacher af­ter we leave, or in many cases, by giv­ing stu­dents time at the end of the work­shop to com­plete a sur­vey.

4. Begin­ning long-last­ing di­alogues with stu­dents with high po­ten­tial.

The in­ter­ac­tive­ness of the SHIC work­shop al­lows a rap­port to form be­tween stu­dents and the In­struc­tor, in­creas­ing the chances of us con­tin­u­ing the con­ver­sa­tion af­ter the work­shop. It is far eas­ier for a SHIC In­struc­tor to iden­tify stu­dents with a high in­ter­est and/​or ap­ti­tude and en­gage them fur­ther. It is also clearer whom an in­ter­ested stu­dent should ap­proach for more in­for­ma­tion. SHIC is cur­rently ex­per­i­ment­ing with struc­tured meth­ods by which stu­dent en­gage­ment can be ex­tended. Th­ese are de­tailed in the “Plans for Fall 2018” sec­tion be­low.

Sum­mary and In­ter­pre­ta­tion of Work­shop Data

Outreach Data

SHIC staff be­gan reach­ing out to teach­ers within the greater Van­cou­ver area in De­cem­ber 2017. Our pri­mary method of on­board­ing schools was cold-emailing teach­ers of Grade 11 and 12 So­cial Stud­ies, So­cial Jus­tice, The­ory of Knowl­edge or Philos­o­phy. We also at­tended three teacher as­so­ci­a­tion meet­ings to pro­mote our work­shop. Of the 125 schools con­tacted, 25 (20%) re­sponded with in­ter­est, and 16 (13%) in­vited SHIC into the class­room, a much higher re­sponse rate than we ex­pected from cold emailing. Based on this, we are mod­er­ately con­fi­dent that the SHIC pro­gram is scal­able. By the end of the 2017-18 school year, SHIC had pre­sented 25 full work­shops (514 stu­dents) of three to five hours, and 16 short work­shops (342 stu­dents) of one to one and a half hours.

Quan­ti­ta­tive Sur­vey Data

We ad­ministered sur­veys to stu­dents be­fore and im­me­di­ately af­ter our full 3-5 hour work­shops.

Th­ese sur­veys in­ves­ti­gated changes in agree­ment or dis­agree­ment (6 point scale) with a range of state­ments about be­liefs, at­ti­tudes, plans and be­havi­ours. Stu­dents were also asked to se­lect 2 (from a list of 5) fac­tors that would be most im­por­tant for them when choos­ing a char­ity. Two of these were coded as ‘effec­tive’ and scored (1), and three as ‘not effec­tive’ and scored (0). Stu­dents were also asked to choose a char­ity (or cause) to donate to and char­i­ties were coded as ‘effec­tive’ if they were recom­mended by se­lected char­ity eval­u­a­tors, or oth­er­wise men­tioned in our work­shop.3 In the pre-work­shop sur­vey, this was a free write-in re­sponse and dona­tions were in­cen­tivised (a ran­dom sam­ple of char­i­ties cho­sen re­ceived ac­tual dona­tions), and post-work­shop the stu­dents were di­rected to our giv­ing plat­form,4 where they were asked to donate the $10 they re­ceived from SHIC to the char­ity of their choice.

Data was some­what limited due to prac­ti­cal difficul­ties in data col­lec­tion (we re­ceived ~500 pre-work­shop re­sponses and ~300 post-work­shop re­sponses). In ad­di­tion, only 168 pairs of re­sponses could be matched, due to a large num­ber of re­spon­dents failing to cor­rectly en­ter their anony­mous ‘match­ing codes’ cor­rectly. As a re­sult, the fi­nal sam­ple may be a more highly en­gaged/​at­ten­tive self-se­lected sam­ple.

We con­ducted paired sam­ple t-tests on the 168 match­ing re­sponses.5

In the char­ity se­lec­tion task, pre-work­shop 325 stu­dents stated a char­ity, 1 of which was coded as an ‘effec­tive’ char­ity (0.3%). Post-work­shop, 131 stu­dents stated a char­ity, 84 of which were ‘effec­tive’ char­i­ties (64%). It is re­as­sur­ing that post-work­shop a ma­jor­ity of re­spon­dents chose an effec­tive char­ity, though this is likely par­tially due to in­creased fa­mil­iar­ity (these char­i­ties just hav­ing been men­tioned in the work­shop).

Fund­ing to­tal­ing $4,400 CAD was pro­vided to stu­dents through the on­line dona­tion plat­form. $1,638 was di­rected to char­i­ties by the stu­dents, with $1,004 (61%) of donated money go­ing to ‘effec­tive’ char­i­ties. The re­main­ing $2,762 has yet to be donated.


We found sig­nifi­cant, if small, im­prove­ments (by our stan­dards) in knowl­edge, at­ti­tudes and val­ues for the ma­jor­ity of our ques­tions. We found null re­sults for 3 items (“I have a duty to take sig­nifi­cant ac­tion to help peo­ple that are suffer­ing.” “I plan to choose a ca­reer that will al­low me to do as much good as I can” “I have com­pared char­i­ties against each other on my own time.”) The first two of these may be partly ex­plained by the high mean re­sponses in the pre-work­shop sur­vey. The null re­sult for stu­dents com­par­ing char­i­ties “on their own time” is un­sur­pris­ing as the post-work­shop sur­vey was ad­ministered im­me­di­ately af­ter the work­shop, be­fore most stu­dents would have had time to in­de­pen­dently com­pare char­i­ties.

On the whole, we view these re­sults as fairly promis­ing and slightly in­crease our con­fi­dence that SHIC work­shops are hav­ing a pos­i­tive effect on stu­dents be­liefs and at­ti­tudes.

Two sig­nifi­cant limi­ta­tions are high rates of re­spon­dent at­tri­tion and the likely in­fluence of so­cial de­sir­a­bil­ity bias and/​or de­mand effects, as it was likely clear (post-work­shop) which were the de­sired re­sponses. In ad­di­tion, it is hard to get a sense of how sig­nifi­cant the im­pact of these self-re­ported changes are. For these rea­sons, in fu­ture iter­a­tions we plan to place less em­pha­sis on such sur­vey data, fo­cus­ing in­stead on eval­u­at­ing our im­pact on be­hav­ioral changes with a smaller num­ber of highly en­gaged stu­dents. Read more about our plans for fur­ther en­gage­ment in the “Plans for Fall 2018” sec­tion be­low.

Qual­i­ta­tive Results

Every teacher we worked with gave us pos­i­tive, some­times glow­ing feed­back, and most ex­pressed a de­sire to have us come to their school again. Some thought our work­shop should be com­pul­sory for all stu­dents, and some teach­ers later said that stu­dents con­tinued to men­tion the con­tent of the work­shop weeks af­ter we left. The re­sponse from stu­dents in the class­room was also very pos­i­tive. In all classes, most stu­dents were en­gaged in the pre­sen­ta­tions, and in most classes, there were sev­eral stu­dents who par­ti­ci­pated in dis­cus­sions en­thu­si­as­ti­cally.

Our post work­shop sur­vey in­cluded a qual­i­ta­tive ques­tion: “Has the SHIC pro­gram changed how you might act in the fu­ture? If so, how?” There were 122 re­sponses from the 176 stu­dents offered this ques­tion. 33 stu­dents de­scribed ac­tions they would take as a re­sult of the work­shop, such as donat­ing to bet­ter char­i­ties, donat­ing more money, or con­sum­ing fewer an­i­mal prod­ucts. 36 stu­dents gave re­sponses stat­ing their in­creased mo­ti­va­tion to help oth­ers or some­thing they had learned, and 13 stated that the SHIC work­shop has not changed how they will act in the fu­ture. Over­all these re­sponses in­di­cated a high level of en­dorse­ment of the ideas in the work­shop, but it is hard to know how these re­sponses will trans­late into fu­ture ac­tions or im­pact.

In or­der to get an un­der­stand­ing for in­ter­est lev­els be­yond the work­shop, at the end of the 2017-18 school year we ex­tem­po­ra­ne­ously at­tempted to re-en­gage stu­dents who had pre­vi­ously par­ti­ci­pated in a full SHIC work­shop. We first worked to sched­ule an ad­vanced work­shop with in­ter­ested stu­dents, but there was in­ad­e­quate de­mand and we were un­able to find a time that suited more than a few of the stu­dents, so the work­shop did not go ahead. Fur­ther at­tempts to en­gage via one-on-one coach­ing also yielded no re­sults. While this trial could po­ten­tially be in­dica­tive of in­suffi­cient stu­dent in­ter­est be­yond the ini­tial work­shop, we feel that more plan­ning (in­form­ing stu­dents of op­por­tu­ni­ties dur­ing the work­shop and en­gag­ing shortly af­ter) and bet­ter timing (not at the end of the school year) will re­sult in more stu­dent en­gage­ment. We re­main hope­ful that fu­ture, more struc­tured at­tempts to en­gage stu­dents will lead to a more fruit­ful up­take.

Conclusion

Over­all, though still as­cer­tain­ing whether or not SHIC work­shops are cost-effec­tive rel­a­tive to the im­pact of other move­ment build­ing ini­ti­a­tives, we feel con­fi­dent based on ev­i­dence thus far that we should con­tinue to ex­plore this strat­egy, and mod­er­ately con­fi­dent that we should at­tempt to ex­pand to other cities in early 2019. Im­pact as­sess­ments of our pro­gram will con­tinue for the fore­see­able fu­ture.

Plans for Au­tumn 2018 and Beyond

The most sig­nifi­cant up­date we’ve made since ini­ti­at­ing the work­shop ex­per­i­ment in Jan­uary 2018 is plac­ing more em­pha­sis on the need to cre­ate an in­ten­tional en­gage­ment fun­nel through which high-po­ten­tial stu­dents can con­tinue their ed­u­ca­tion.

Our cur­rent plan is to ini­ti­ate en­gage­ment in two differ­ent ways:

  • The first is to run ad­vanced SHIC work­shops (called Next Level SHIC) lo­cally for high-po­ten­tial stu­dents through­out the schools SHIC vis­its. Th­ese work­shops would be gen­er­ally more free-form and we will be pre­pared to provide con­tent and re­sources spe­cific to the par­tic­u­lar stu­dents in the work­shop. Our goal is to build a lo­cal self-sus­tain­ing com­mu­nity of young peo­ple that will al­low them to re­main en­gaged through­out high school and uni­ver­sity.

  • The sec­ond is to offer high-po­ten­tial stu­dents one-on-one coach­ing in or­der to cre­ate a last­ing di­alogue and in­crease the chances for sig­nifi­cant fu­ture im­pact. This is likely to in­volve recom­mend­ing re­sources that deepen the stu­dent’s knowl­edge, and could in­clude in­for­ma­tion about po­ten­tial ca­reer paths, choos­ing char­i­ties for school fundraisers, or the use of ra­tio­nal­ity and pro­duc­tivity tools.

Key met­rics for both coach­ing and ad­vanced work­shops will be the num­ber of stu­dents we en­gage through these meth­ods, and the ac­tions stu­dents take as a re­sult of their in­volve­ment, such as ed­u­ca­tion and ca­reer de­ci­sions, pur­su­ing vol­un­teer­ing op­por­tu­ni­ties with effec­tive char­i­ties, and at­tend­ing EA mee­tups and con­fer­ences.

In 2019, pend­ing anal­y­sis of data to be gath­ered in Au­tumn 2018, we hope to ex­pand SHIC work­shops into one or more ad­di­tional cities be­yond Van­cou­ver.

Other changes we are mak­ing for Au­tumn 2018:

  • We are now plac­ing less em­pha­sis on the fu­ture col­lec­tion of quan­ti­ta­tive sur­vey data dur­ing the work­shop, as this data doesn’t ap­pear to prop­erly re­flect our im­pact for rea­sons out­lined in the ‘Quan­ti­ta­tive Sur­vey Data’ Sec­tion above. As a re­sult we are now ad­minis­ter­ing a short­ened post-work­shop sur­vey with qual­i­ta­tive ques­tions af­ter all work­shops, and omit­ting the pre-sur­vey.

  • We are in the pro­cess of cre­at­ing a more math­e­mat­i­cally dense work­shop which we have started ad­ver­tis­ing to Grade 11 and 12 Calcu­lus and Statis­tics teach­ers, and have had a promis­ing num­ber of re­sponses com­ing from teach­ers already. This should al­low us to reach stu­dents that we may not have been able to reach so far.

  • SHIC in­tends to test out meth­ods of en­gag­ing ter­tiary stu­dents through guest lec­tures and work­shops in rele­vant stu­dent clubs. We think con­nect­ing with ter­tiary stu­dents could be very effec­tive, how­ever we cur­rently do not know how easy it will be to get an au­di­ence in this way.

  • Stu­dents around the world will con­tinue to be sup­ported in run­ning the SHIC pro­gram as part of a stu­dent or­ga­ni­za­tion. De­spite the high at­tri­tion rate of stu­dent lead­ers, the few that are suc­cess­ful are some­times profoundly af­fected by the pro­gram. Th­ese few stu­dents may make the Stu­dent Leader model worth­while. For the 2018-19 school year, most of the re­cruit­ment and one on one sup­port will be pro­vided by SHIC’s vol­un­teer team with staff over­sight.


This post was writ­ten by Bax­ter Bul­lock, with sec­tions writ­ten by Cather­ine Low and David Moss. Edit­ing and re­search by Tee Bar­nett, David Va­tousios and Marisa Jur­czyk. Data anal­y­sis by David Moss and Ta­mara Sti­matze. Data col­lec­tion by David Va­tousios and Cather­ine Low (SHIC In­struc­tors). Thanks to the CHIMP Foun­da­tion, who pro­vided us the plat­form through which we could provide work­shop par­ti­ci­pants money for dona­tions, and a spe­cial thanks to all of the schools that al­lowed SHIC into the class­room and data col­lec­tion to take place.


1. Since April, one of the two in­struc­tors is no longer Van­cou­ver-based.

2.At var­i­ous points through­out the data col­lec­tion pro­cess, SHIC pro­vides work­shop par­ti­ci­pants with $10 dona­tions through an on­line plat­form where stu­dents can give the money to their choice of char­i­ties.

3. ‘Effec­tive’ char­i­ties in­clude Cool Earth, GiveWell and its top char­i­ties, An­i­mal Char­ity Eval­u­a­tors and its recom­mended char­i­ties, and char­i­ties sup­ported by Open Philan­thropy and Founders Pledge.

4. Pro­vided by the Char­i­ta­ble Im­pact Foun­da­tion (CHIMP)

5. In­for­ma­tion about t-tests, t-val­ues and p-val­ues can be found here.