Thanks for this update, and for your valuable work.
I must admit I was frustrated by reading this post. I want this work to continue, and I don’t find the levels of engagement you report surprising or worth massively updating on (i.e. suspending outreach).
I’m also bothered by the top-level comments assuming that this didn’t work and should’ve been abandoned. What you’ve shown is that you could not provide strong evidence of the type that you hoped for the programs effectiveness, NOT that it didn’t work!
Basically, I think there should be a strong prior that this type of work is effective, and I think the question should be how to do a good job of it. So I want these results to be taken as a baseline, and for your org to continue iterating and trying to improve your outreach, rather than giving up on it. And I want funders to see your vision and stick with you as you iterate.
I’m frustrated by the focus on short-term, measurable results here. I don’t expect you to be able to measure the effects well.
Overall, I feel like the results you’ve presented here inspire a lot of ideas and questions, and I think continued work to build a better model of how outreach to high schoolers works seems very valuable. I think this should be approached with more of a scientific/tinkering/start-up mindset of “we have this idea that we believe in and we’re going to try our damndest to make it work before giving up!” I think part of “making it work” here includes figuring out how to gauge the impact. How do teachers normally tell if they’re having an impact? Probably they mostly trust their gut. So is there a way to ask them (obvious risk is they’ll tell you a white lie). Maybe you think continuing this work is not your comparative advantage, or you’re not the org to do it, which seems fine, but I’d rather you try and hire a new “CEO”/team for SHIC in that case (if possible), and not throw away existing institutional knowledge, rather than suspend the outreach.
------------------------- RE evaluating effectiveness: I’d be very curious to know more about the few students who did engage outside of class. In my mind, the evidence for effectiveness hinges to a significant extent on the quality and motivation of the students who continue engaging.
I think there are other ways you could gauge effectiveness, mostly by recruiting teachers into this process. They were more eager for your material than you expected (well, I think it makes sense, since its less work for them!) So you can ask for things in return: follow-up surveys, assignments, quiz questions, or any form of evaluation from them in terms of how well the content stuck and if they think it had any impact.
A few more specific questions: - RE footnote 3: why not use “EA” in the program? This seems mildly dishonest and liable to reduce expected impact. - RE footnote 7: why did they feel inappropriate?
Hi capybaralet, Thanks for your comments and enthusiasm for the program!
> I must admit I was frustrated by reading this post. I want this work to continue, and I don’t find the levels of engagement you report surprising or worth massively updating on (i.e. suspending outreach).
I admit when the decision was made to stop actively working on SHIC, I was pretty sad and frustrated too. However for our team, and our funders too, the main question was “do we think this is worth continuing compared to other things we could spend our time and money on?”, and the answer was “probably not”.
You might also be interested in this post which combines the experience of all the EA outreach attempts I was aware of at the time:
This probably will answer many of your questions about why we didn’t continue to test out different ideas of engaging students and teachers—we’d already tried quite a few different things and learnt from work from other EAs. The post is now nearly 2 years old and there have been other efforts in the EA community to work with high schoolers since then. But I still basically agree with my conclusion which was:
> I don’t think our outreach described in this post was a particularly effective use of resources. However, outreach could be effective if you are able to attract highly promising students to sign up for a program over a longer term. This might be possible if you have a strong brand (such as an association with elite University) allowing you to attract suitable students through schools and other networks, and the resources to run a fellowship-type program with these students.
To answer your specific questions: 1. There were only a few students who engaged significantly out of class, so it is hard to know what to conclude from a small number. Some were quite keen on EA concepts, others were eager to do good, but didn’t seem to be particularly excited about apply EA principles to their career path or volunteering, so we didn’t feel that the impact these students could have was sufficient to outweigh the small number. 2. The question of whether to use “effective altruism” was discussed a lot within our team. We ended up using the term a little in the program, referred to EA on our website, providing copies of “Doing Good Better” to the teachers to lend to interested students, and using the term with the advanced workshop students. The reason for not using the term prominently was partly because we felt some teachers/parents might be put off by the term, and also to provide brand separation between the EA community and SHIC—if we used “EA” and did a poor job that would reflect poorly on EA as a whole. Similarly, if EA got a poor reputation we might still be able to continue with SHIC. 3. Some high schools have policies around teachers not connecting with students on social media (except through official school pages), and we know some schools and parents are cautious about how minors interact over social media, especially with older folks like us! So we were wary about using social media to have one-on-one or small group conversations. Our hope was that the advanced workshop students would eventually drive their own EA group, and make their own social media presence.
Thanks for this update, and for your valuable work.
I must admit I was frustrated by reading this post. I want this work to continue, and I don’t find the levels of engagement you report surprising or worth massively updating on (i.e. suspending outreach).
I’m also bothered by the top-level comments assuming that this didn’t work and should’ve been abandoned. What you’ve shown is that you could not provide strong evidence of the type that you hoped for the programs effectiveness, NOT that it didn’t work!
Basically, I think there should be a strong prior that this type of work is effective, and I think the question should be how to do a good job of it. So I want these results to be taken as a baseline, and for your org to continue iterating and trying to improve your outreach, rather than giving up on it. And I want funders to see your vision and stick with you as you iterate.
I’m frustrated by the focus on short-term, measurable results here. I don’t expect you to be able to measure the effects well.
Overall, I feel like the results you’ve presented here inspire a lot of ideas and questions, and I think continued work to build a better model of how outreach to high schoolers works seems very valuable. I think this should be approached with more of a scientific/tinkering/start-up mindset of “we have this idea that we believe in and we’re going to try our damndest to make it work before giving up!” I think part of “making it work” here includes figuring out how to gauge the impact. How do teachers normally tell if they’re having an impact? Probably they mostly trust their gut. So is there a way to ask them (obvious risk is they’ll tell you a white lie). Maybe you think continuing this work is not your comparative advantage, or you’re not the org to do it, which seems fine, but I’d rather you try and hire a new “CEO”/team for SHIC in that case (if possible), and not throw away existing institutional knowledge, rather than suspend the outreach.
-------------------------
RE evaluating effectiveness:
I’d be very curious to know more about the few students who did engage outside of class. In my mind, the evidence for effectiveness hinges to a significant extent on the quality and motivation of the students who continue engaging.
I think there are other ways you could gauge effectiveness, mostly by recruiting teachers into this process. They were more eager for your material than you expected (well, I think it makes sense, since its less work for them!) So you can ask for things in return: follow-up surveys, assignments, quiz questions, or any form of evaluation from them in terms of how well the content stuck and if they think it had any impact.
A few more specific questions:
- RE footnote 3: why not use “EA” in the program? This seems mildly dishonest and liable to reduce expected impact.
- RE footnote 7: why did they feel inappropriate?
Hi capybaralet,
Thanks for your comments and enthusiasm for the program!
> I must admit I was frustrated by reading this post. I want this work to continue, and I don’t find the levels of engagement you report surprising or worth massively updating on (i.e. suspending outreach).
I admit when the decision was made to stop actively working on SHIC, I was pretty sad and frustrated too. However for our team, and our funders too, the main question was “do we think this is worth continuing compared to other things we could spend our time and money on?”, and the answer was “probably not”.
You might also be interested in this post which combines the experience of all the EA outreach attempts I was aware of at the time:
https://forum.effectivealtruism.org/posts/L5t3EPnWSj7D3DpGt/high-school-ea-outreach
This probably will answer many of your questions about why we didn’t continue to test out different ideas of engaging students and teachers—we’d already tried quite a few different things and learnt from work from other EAs. The post is now nearly 2 years old and there have been other efforts in the EA community to work with high schoolers since then. But I still basically agree with my conclusion which was:
> I don’t think our outreach described in this post was a particularly effective use of resources. However, outreach could be effective if you are able to attract highly promising students to sign up for a program over a longer term. This might be possible if you have a strong brand (such as an association with elite University) allowing you to attract suitable students through schools and other networks, and the resources to run a fellowship-type program with these students.
To answer your specific questions:
1. There were only a few students who engaged significantly out of class, so it is hard to know what to conclude from a small number. Some were quite keen on EA concepts, others were eager to do good, but didn’t seem to be particularly excited about apply EA principles to their career path or volunteering, so we didn’t feel that the impact these students could have was sufficient to outweigh the small number.
2. The question of whether to use “effective altruism” was discussed a lot within our team. We ended up using the term a little in the program, referred to EA on our website, providing copies of “Doing Good Better” to the teachers to lend to interested students, and using the term with the advanced workshop students. The reason for not using the term prominently was partly because we felt some teachers/parents might be put off by the term, and also to provide brand separation between the EA community and SHIC—if we used “EA” and did a poor job that would reflect poorly on EA as a whole. Similarly, if EA got a poor reputation we might still be able to continue with SHIC.
3. Some high schools have policies around teachers not connecting with students on social media (except through official school pages), and we know some schools and parents are cautious about how minors interact over social media, especially with older folks like us! So we were wary about using social media to have one-on-one or small group conversations. Our hope was that the advanced workshop students would eventually drive their own EA group, and make their own social media presence.