Lessons from two years of talent search pilots

Tl;dr:

  • Leaf supports exceptional teenagers to explore how they can do the most good.

  • We’ve run 3 residential programmes and 4 different types of online fellowship: general to having high positive impact, focused on university decision-making, cause-specific, and subject-specific.

  • I’m excited about the online programmes (especially subject-specific) as being cost-effective and highly scalable.

  • I plan to actually scale these! You might be able to help through:

    • Advising Leaf

    • Being a facilitator or guest speaker

    • Working for Leaf later this year

    • Funding Leaf

  • There are lots of other mini insights and updates, summarised below.

I wrote this post quickly, adapting from an existing internal doc, so that I could do an ‘80:20’ version of sharing insights. Please message me if you’d like access to the original doc with lots more detail (it’s 50 pages, mostly of summary tables of metrics I track), evidence, and reasoning transparency. Please briefly explain who you are and why you’re interested in access.

Background on Leaf

I’m Managing Director of Leaf; we support exceptional teenagers to explore how they can best help others, save lives, or change the course of history.

In conventional educational systems, teenagers don’t have support or mentorship to explore how they can do good. The incentives and encouragement for smart teens are mostly about getting into uni and demonstrating their intelligence, not thinking through how to use those gifts.

And yet they’re already making decisions relevant to doing good, like what subjects to study at university, what sort of internships to get or project to pursue, and just which problems to focus on finding out more about.

Meanwhile, many of the world’s most pressing problems are constrained by not having access to enough talented applicants and entrepreneurs. There’s a need to ensure that smart students explore important and neglected problems, rather than just defaulting to family- or status-driven careers, or tackling the problems made most salient to them through the media.

Leaf supports these exceptional teenagers to start exploring, make better decisions and get on a high-impact trajectory.

Programmes summary

ProgrammeDatesMotivation and goalsKey lessons

Residential pilot, 2021:

  • “Building a Better Future”

(Not me) July to October 2021 (active October 2021)I didn’t set this up so can’t really comment, but I think it was similar to the reasons here.
  • Students seemed brighter and more engaged with the content than I expected.

  • Turns out that you can get really high quality applicants to programmes if you contact lots of different places, even with cold outreach methods, no demonstrable track record, and a very MVP website with stock images. The previous team contacted ~200 schools and received ~70 applications, of which they accepted 16.

  • There was fuller attendance than I expected — all 16 applicants offered a place made it to the residential, whereas I had expected closer to ~50% drop out, as often happens with free events.

  • (Also it seemed plausibly my comparative advantage given my experience in teaching and talent search /​ community building.)

Summer residential, 2022:

February to September 2022 (active August 2022) with some subsequent follow-up and strategic planning
  • Try out slightly scaling what seemed like a successful model for talent search and community building.

  • I took on Leaf, handed over from Alex Holness-Tofts; it was partly just about replicating the success of the pilot in the summer without doing anything too ambitious.

  • Managed to identify a reasonably promising group of young people, and successfully encouraged some engagement with effective altruism and longtermism.

  • The second BBF went roughly according to expectations, with various small updates in both positive and negative directions.

  • There were a few unexpected forms of engagement in the immediate follow-up after the programme, but longer-term engagement was disappointing.

  • It seemed like participants shifted their cause prioritisation views.

  • The cold outreach methods that we used last time (emailing schools) didn’t scale as well as I had hoped; we only got 174 applicants.

  • There was anecdotal evidence that some of the concerns and risks relating to outreach to high school audiences have indeed been borne out to some extent, e.g. some evidence of overwhelmingness.

Online early 2023:

January to April 2023

(active March)

  • I had resolved to test out a cheaper programme type than in-person residentials, to see if I could achieve a large fraction of the value for a lower cost.

  • I wanted to test a uni/​degree choice programme against one focused more directly on how to tackle the world’s most pressing problems.

  • I also became excited about using them as talent search, funnelling, and filtering for the residentials (or other more resource-intensive follow-up programmes).

  • Similar application numbers (138) and quality to our previous in-person programme.

  • Degree programmes less appealing to applicants than expected.

  • Degree programmes do affect degree choices!

  • Potential for online programmes to sort by promise and engagement.

(I later concluded that some of the most promising participants in the Changemakers Fellowship had been identified by (and gained comparable value from) these programmes, and/​or the Non-Trivial fellowship.)

Summer residential, 2023:

April to September 2023 (active August)
  • I had resolved to test out ways to improve the effects per participant of the residential programme, without increasing costs by much.

  • I ran quite a few variations to try to achieve this.

  • I wanted to evaluate (cost-)effectiveness more rigorously, collaborating with Michael Noetel to run an experiment at 3-month follow-up, comparing between accepted and narrowly rejected applicants (and statistically controlling for differences between them).

  • ~5x applicant numbers (755) and improved quality compared to the 2022 residential or the 2023 online programmes

  • Some of the tweaks helped a bit, some made it worse.

  • Broadly, the results of the more rigorous experimental follow-up study were disappointing. There was some evidence of positive changes, but less than you might hope/​expect given the resource expenditure.

  • Overall, usefulness per participant seemed lower than 2022 and my guess of cost-effectiveness fell (to ~one-third of the 2022 residential) despite the benefit of experience, track record for marketing, and more time spent on the programme.

  • After the follow-up survey results I decided to indefinitely deprioritise residential programmes.

Subject-specific online programme pilot:

September to December 2023 (active November)
  • I wanted to test the subject-specific format, which I was optimistic about, e.g. on application quality and quantity, e.g. for apparent effects.

  • I had initially fundraised to run the subject-specific programmes as weekend-long in-person programmes, but pivoted to online due to the relative success of the early 2023 programmes and user feedback suggesting demand wouldn’t be sufficiently different to justify the higher costs.

  • I wanted to test something relatively quick and cheap while I waited for the results of the Changemakers Fellowship follow-up survey.

  • Disappointing application numbers (153); similar to the early 2023 online programmes, and ~0.2x the Changemakers Fellowship.

  • Seemed fairly promising in terms of effects per participant (given the costs). E.g. frequency of change in university and career plans seemed similar to the Changemakers Fellowship; self-reported change in interest in EA and support for positively influencing the long-term future was higher.

  • (I was feeling pessimistic about Leaf’s prospects at this point. I applied to several alternative jobs, but continued planning for Leaf next steps too.)

Small pilot of an online work experience programmeDecember 2023 (active December 2023 to ~June 2024)
  • I wanted to test an alternative option for supporting especially promising participants, either to create a new standalone programme with public applications or just as a follow-up support method for alumni of other programmes.

  • I only ran this with 6 participants and 5 orgs, with no public application round; this was primarily through direct invite of Changemakers Fellows.

  • (More info on the motivation here)

  • Decent interest from potential partner organisations. I just dropped a Google Form on the Charity Entrepreneurship slack and this led to 6 sign ups. From directly contacting ~5 other orgs, I had 2 sign ups.

  • Was harder than expected to get placements to actually start in earnest, which would make an organised cohort tough.

  • As of now, 1 clear ‘success’ for the intern and their org; 1 seems weakly net negative for the org; 1 paused/​stopped early; 1 never started; and 1 org (2 interns) only just starting but seems potentially promising.

  • (More thorough data collection pending.)

Online early 2024:

January to April 2024 (active March)
  • I felt surprised at the low application numbers for History to Shape History 2023 and suspected that a new programme might do better.

  • I wanted to directly test subject-specific vs cause-specific online programmes, e.g. on application quality and quantity, e.g. for apparent effects.

  • I wanted to test the potential for setting up an ecosystem of online programmes, where alumni sign up for follow-up online programmes.

  • I wanted to test the scalability of the online programmes by reusing History to Shape History with extra marketing and alumni facilitators but little change to the content.

  • I wanted to test smaller tweaks to the model, e.g. incorporating a weekly essay competition into HSH, e.g. to see if I could run AI with a much lower-effort curriculum.

  • 10x the number of applications (1,563) as the 2023 online cohort and 2x the number of the Changemakers Fellowship, due to a mix of improved product, improved website, and an alumni TikTok which received 100k views. My optimism in the online fellowships as a scalable model was restored!

  • MoM had 5x the applications that AI had, and 10x HSH. My optimism in the subject-specific model was restored!

  • Promise for an ecosystem of Leaf online programmes: 68% of AI Fellows signed up for follow up fellowships (e.g. a Leaf AISF adaptation, switching to MoM), as did 32% of MoM and HSH fellows.

  • Leaf alumni facilitators were great. I estimate that I can 1.5x the size of each fellowship each cohort without needing to hire facilitators externally.

  • Likelihood to recommend was high (9.0 average for Fellows) across programmes, despite way less effort being put into the AI curriculum; this suggests the possibility to rapidly expand our array of online options.

  • It may be difficult to run termly cohorts advertised to the same audience; there were fewer HSH applications than in late 2023, despite website improvements (but possibly due to applicants switching to MoM).

  • 32% of HSH Fellows proposed independent follow-up projects; building towards impact-focused projects is feasible alongside subject-specific fellowships, with appropriate structuring.

  • I raised £5.6k in donations from participants’ parents, and expect to be able to cover 10-40% of online programme financial costs this way going forwards.

Leaf’s next steps

I’m excited about continuing to offer cheap and scalable online fellowships, supporting top students to explore how they can use the subjects they love and skills they have to maximise their positive impact.

Some findings from the recent cohorts that make me especially excited:

  • The majority of Fellows reported changes to their career plans and the majority reported changes to their views about the world’s most important problems.

  • Feedback to date has been highly positive, with better or similar likelihood to recommend scores than EA Global, Non-Trivial, MATs, and AI Safety Fundamentals.

  • By my estimates, Leaf is roughly as cost-effective on some of the headline metrics that CEA and 80k use as those orgs are (hours of engagement, number of connections made).

So I’m planning to scale the recent online programmes model!

Specifically, within the next few weeks I’m planning to launch applications for a new cohort to take place over summer which:

  • Includes a repeat of The Mathematics of Morality, History to Shape History, and Dilemmas and Dangers in AI.

  • Includes a new subject-specific programme focused on biology and medicine.

  • Focuses primarily on the UK again, but tests out an internationalisation strategy (and avoids overfitting on the UK) by expanding to include the Republic of Ireland.

  • (If time) I might also pilot or test demand for one or two more cause-specific programmes.

I estimate that by the end of the year, we’ll have supported more students at/​on track for top universities to spend 10 hours or more exploring how they can do the most good than the entire UK EA uni group ecosystem. I’m also optimistic that the model can scale and spread to new countries more rapidly than many existing talent search and community building methods.

How you can help

If you have some form of relevant expertise* and are excited about Leaf’s mission and impact potential, I’d love to hear from you!

  • I’m always keen for advice and feedback on Leaf’s plans.

  • We need guest speakers as well as facilitators (hosting weekly discussion calls with ~5 fellows for 5 weeks).

  • I’ll likely be hiring for more substantial roles (e.g. programme leads for individual online programmes; perhaps specialists in community, marketing, or ops; perhaps someone to replace me if I adjust my views on where I’m best suited) in Autumn/​Fall this year.

If you’d like to quickly fill out an expression of interest, you can do so here. Or (my preference), you can message me on the Forum or email jamie [at] leaf.courses

* What could be relevant? Many things! Some examples, in rough order of priority:

  • If you were successful or are well-connected in olympiads, debating, competitions, or other programmes that cluster together smart, curious, ambitious, or socially motivated teenagers

  • Biology/​medicine

  • EdTech or innovative online startups

  • Talent search or community building

  • Marketing (especially TikTok?)

  • AI safety

  • Mathematics

  • History

  • Biorisk

  • Plausibly any common/​popular academic subject or any top cause area

Additionally, Leaf needs to raise funds for our 2025 plans, to enable hiring and scaling without slowing down after our summer 2024 cohort. If you’re interested in supporting Leaf financially, please reach out!

Funding has primarily come from EA Infrastructure Fund to date; renewal is likely but not guaranteed, and I’d ideally diversify our donor base. (We’ve also been supported by parents of our participants.) For brief context, our most recent cohort cost ~£30,000; ~£250 per completed Fellow (our highest level of engagement, from our most promising applicants) or £40 per completion of any level.

Thanks a lot!

(Reminder: I wrote this post quickly, adapting from an existing internal doc, so that I could do an ‘80:20’ version of sharing insights. Please message me if you’d like access to the original doc with lots more detail (it’s 50 pages, mostly of summary tables of metrics I track), evidence, and reasoning transparency. Please briefly explain who you are and why you’re interested in access.)