Thanks to Owen Cotton-Barratt, Max Daniel, Eliana Lorch, Tanya Singh and the summer fellows for reviewing and improving this post.
Summary of activities
27 fellows spent 6 weeks on a remote fellowship programme.
We got ~300 applications and interviewed ~50 people.
Backgrounds ranged from undergraduate students to postdocs.
By default, each fellow had a mentor and worked on a research project (though there were some exceptions, and many fellows also spent significant time exploring).
The mentors were researchers and research managers at FHI, CSER, OpenPhil, OpenAI, GPI, and MIRI.
Approximately, the projects related to AI governance (13 fellows), technical AI safety (3 fellows), biosecurity (4 fellows), macrostrategy broadly construed (5 fellows), and policy (2 fellows).
This fellowship grew out of the CEA summer fellowship (which CEA wanted to stop running), though was significantly different in that a) it was remote, and b) we took more fellows (in 2019, the CEA fellowship took 11 fellows).
Lessons learned
Meta: we’re stating these confidently to make clear which direction the summer fellowship has updated us in. Obviously this was just one summer programme, and in many cases there might be important differences which mean that these takeaways wouldn’t generalise. The takeaways are in rough order of importance according to us.
Remote fellowships can provide substantial value to fellows and organizers, and are worth considering where in person options are not available, or are especially costly.
Initially we put some probability on the whole fellowship being awkward and unproductive, but in fact fellows had good experiences and some good things seem to have come from the programme. Remoteness also allowed us to take more fellows, and in the particular case of Covid-19, also came with lower opportunity costs for many fellows and organisers.
In spite of remoteness, it was possible to create a positive, friendly and open culture. Many fellows remarked on this and appreciated it, and we guess that it helped to increase engagement, support those experiencing difficulties, and deepen fellows’ experiences. Modelling open and authentic communication on the part of the organisers seems to have been a major contributing factor to this culture.
That said, we still think that there are many ways in which an in person fellowship would have been better, particularly in terms of culture, networking and some kinds of logistics.
There is more interest than we expected in mentoring people amongst researchers in a range of different EA organisations. We successfully found 18 mentors for 24 fellows (3 fellows didn’t have formal mentors for various reasons), and we contacted at least a further 10 people who we think would have mentored if there had been a fellow working on the right topic. We also didn’t do a comprehensive search for mentors, and instead reached out ad hoc to people in our network, so we expect we would have found more interest if we had searched a bit harder.
There were more promising applicants for this programme than we expected. In 2019 the CEA fellowship received ~90 applications, and this fellowship received ~300. We think that the number of strong applicants was correspondingly larger.
Minor difficulties with mental health were common, and more serious ones not extremely rare (although this is confounded by remoteness and Covid-19). 1-1s and other conversations seemed to help support people with this. We think it would be good for future organisers to invest more time in thinking about how to mitigate the risks from a) imposter syndrome, and b) the open ended nature of the fellowship and the lack of clear success metrics.
The personal fit between mentor and fellow might potentially be very impactful. In the two cases where the fellowship led directly on to further remunerated collaboration between the fellow and the mentor, we selected the mentorship pairing in part because we expected the two people to get on well together personally. It’s unclear how important this was, but we think it’s worth paying attention to in future pairings.
A significant part of the value of (this) fellowship came from providing mentorship and network.
There’s significant interest in the fellowship being longer than 6 weeks, from fellows and mentors, though it’s not clear to us if this is net worth it.
Many fellows thought that a mentor with more subject expertise would have been able to support them more. However, most of these fellows also had a strong positive experience with their actual mentor, suggesting that this is a nice to have rather than an essential.
Low-input attempts to create good online office space can be net harmful. We set up a default remote office space, which hampered some more energetic attempts from fellows to create good online environments.
Putting on talks especially for fellows was high time cost and only somewhat valuable. Especially in the context of a remote programme or in the case of recorded talks, we expect that it would have been better to either not do this at all, or to invest even more time in making the series really good.
Communicate clearly to fellows before the start of the programme that there are multiple different outcomes that look like success, if this is the case. We communicated this at the start of the programme, which we think was better than not communicating it at all or communicating it later, but we think that communicating earlier than this would likely have better helped fellows to plan and orient.
The learning value of running such a programme for organisers can be very high. Rose had already co-ran a fellowship similar to this one, but still learned a lot (in part because this fellowship was remote, in part because she had a more empowered role on this fellowship). Eliana hadn’t previously run a fellowship like this and also learned a lot.
Both Rose and Eliana had a lot of vision for this fellowship, but not for a remote version of it, and were initially unexcited about a remote version. We were surprised that it went well given that we didn’t have a clear vision of how to run this remotely.
Goals of the fellowship
The main goal of the fellowship is to help young people to do better things from a longtermist perspective than they would otherwise have done.
We expect most of this change to happen through:
Changing fellows’ career plans, by putting fellows in the way of concrete opportunities, shifting fellows’ motivations, and helping fellows to test their fit.
Improving fellows’ networks, both among other fellows and with more senior people.
Improving fellows’ knowledge and skills in a range of ways, including research taste, research experience, and psychological empowerment.
We try to meet this goal by:
Selecting high potential young people where we think the fellowship is better than what they could counterfactually be spending time on.
Bringing them into a rich longtermist environment.
Giving them the affordance to work on a relevant project.
Providing them with a mentor who can give them feedback.
Introducing them to each other and relevant researchers in the space.
Detail on activities
Timeline of activities
January/February 2020: Rose and Eliana decided to take on the project together
19th February: application page goes live (deadline 22nd March)
We asked for a CV and a research proposal
The research proposal was for the purposes of evaluation; people were free to work on other things during the programme
March-April: desk rejects and interview decisions
Late April: interviews
Early May: offers
13th July: fellowship started
24th August: fellowship ended
Summer fellows
There were 27 fellows in all. Below is a list of those fellows who explicitly gave us permission to name them in this post. See here for more extended bios.
Metagenomic sequencing and public health surveillance
Evaluation of aspects of the fellowship
The fellowship as a whole
We asked fellows the following questions:
How much better or worse do you feel about the summer fellowship than about what you would have done otherwise?
Average 8.58 out of 10, min 5
If the same program (same organizers, same structure etc.) happened next year, how strongly would you recommend a friend (with similar background to you before the fellowship) to apply?
Average 8.89 out of 10, min 2 (then 7)
We think these are fairly good responses overall. The minimum responses to both questions came from the same fellow.
Length of the programme
Fellows’ opinions
We asked fellows ‘What % of the value you’ve gotten from the fellowship so far would you attribute to each week of the fellowship?’
The average responses were 15%, 17%, 16%, 15%, 14% and 16% for weeks 1,2,3,4,5 and 6
This suggests that returns weren’t diminishing over the 6 weeks
Of the 27 fellows, 5 said in the final survey that they wished the fellowship had been longer. We guess that an additional 3 might also have benefitted from a longer fellowship
Mentors’ opinions
2 mentors suggested making the fellowship longer in future
Our thoughts
It’s worth noting that the CEA summer fellowship was 6 weeks in 2018, and fellows said it was too short. The 2019 CEA fellowship was therefore lengthened to 8 weeks, and fellows again said it was too short. However, the organisers thought that the 2019 fellowship was a success, and that an additional 2 weeks would not have increased the impact of the programme much.
There’s some tension between the content of the fellowship and where we think the impact comes from. Fellows undertake a research project during the fellowship; we think that 6 weeks is a little too short for this, especially given the extra time needed to orient to a new environment. However, we don’t think that the main value of the fellowship comes from completed research outputs. Instead, we expect most of the value to come for career updates, networking and gaining knowledge and skills. 6 weeks seems enough to capture significant value here.
We think that organisers next year should seriously consider making the fellowship longer, but we don’t think it’s clear that they should.
Program components
We asked fellows ’How valuable did you find the following [programme components]?
The options were Harmful, ~Neutral, Valuable, Very Valuable, and NA
Assigning −1 to Harmful, 0 to ~Neutral, 1 to Valuable, and 2 to Very valuable, the programme components were rated in the following way:
Component
Description
Score
Comments
Mentoring
All fellows bar 3 were assigned a research mentor who met with them roughly once a week.
1.73
These seem like the two most important parts of the fellowship structure. In other sections of the survey, multiple fellows stressed how valuable these things were.
Setting up the 1-1 introductions was time intensive, but seems worth it.
Intros to FHIers
Rose facilitated around 80 1-1 introductions between fellows and (primarily) FHI researchers
1.67
1-on-1s w/ fellows
Every week, fellows were paired with one other fellow to have a 1-1
1.44
1-on-1s w/ Rose/Eliana
Rose had at least one 1-1 with every fellow, and checked in more frequently with some. Eliana had a range of 1-1s. The content of these 1-1s was usually checking in, discussing problems and progress and getting to know each other.
1.38
This seems valuable but was high time cost. Overall we would do it again, as it helped us to run the programme better and also mitigated some risks.
Socials
We organised a handful of socials at the beginning of the fellowship, and fellows then organised additional socials themselves.
1.32
This was scored relatively highly even though we didn’t put much organising effort into making socials happen. 8 fellows commented that they would have liked more socials, so it’s probable that there was more value to be had here, though we’re unsure if this would have been worth additional time investment on the part of the organisers.
Regular FHI talks and seminars
Fellows were invited to most FHI seminars and research events (1-2 per week).
1.29
These talks were happening anyway so it was very cheap to invite fellows to them.
Opening session
On the first day of the fellowship we held two 1.5 hour opening sessions to launch the fellowship (2 because time zones meant that it was not possible for all fellows to attend at the same time).
1.26
Career planning workshops
A collaborator ran an optional career planning workshop in the 5th week of the fellowship, which ~8 fellows participated in.
1.25
Worth running, though interestingly two people rated it as neutral.
Midway session
At the end of the third week, we ran two 1.5 hour midway sessions, to check in with people and provide an opportunity for fellows to reflect and reset.
1.24
The project review template
In the 5th week we shared an optional template for fellows to use to review their projects.
1.19
This seems worth it as it was such a small time cost. It’s interesting that it was so much more valuable than the project plan template.
SRF talk series
We organised around 14 talks during the fellowship, from researchers in the x-risk space. Some were pre-recorded, others were live.
1.08
This took a fair bit of organising, and possibly this isn’t a high enough rating to justify the cost. We think that putting in more effort to make the talk series really great, or not doing it at all, would have been better in our case. We also think that a large part of why this wasn’t more valuable was that we didn’t provide a good enough format for synchronous viewing of talks, and that had the fellowship been in person, the talks may well have been much more valuable without any other changes.
The SRF meta doc
Before the fellowship, we shared a meta doc with fellows describing the purpose and structure of the fellowship.
1.04
Although it’s a relatively low rating, it seems clearly important for fellows to have some doc giving context and explaining the structure.
Final presentations
In the final week of the fellowship, we organised opt-out final presentations for fellows. These lasted 10 minutes followed by 5 minutes for questions, and were open to all fellows, mentors and FHIers. 23 fellows gave final presentations.
1.00
Given the time cost to fellows and organisers, this may not be worth doing in future (or better formats might make it more valuable).
The project planning template
In the first week we shared an optional template for fellows to use to plan their projects.
0.86
This seems worth it as it was such a small time cost.
Check in group meetings
All fellows were assigned to a check in group with 2-4 other fellows, to meet once a week to discuss progress and any topics of interest. In the second week, we surveyed fellows on how their check in groups were going, hoping to be able to intervene where things weren’t working, but few fellows responded and we didn’t take any significant actions.
0.81
This score was pulled down somewhat by the responses from one group which didn’t go well (the score would be 0.96 excluding that group).
If this is repeated, it would be good to have a better way of intervening on check in groups when they are not working (e.g. by organisers attending all groups once, or checking in with every fellow individually).
In spite of the low score, we expect that given the remote nature of the fellowship and the large number of fellows, having smaller groups to talk with regularly may have been important for creating a friendly, supportive culture. At least two check in groups continued meeting after the fellowship, additionally suggesting that this was valuable for some fellows.
Lightning talks
In the second week, we ran three lightning talk sessions. Each fellow had 5 minutes for introducing their work and answering questions. The talks were open to all fellows, mentors and FHIers.
0.76
Probably not worth doing, but it’s a bit hard to tell. Fellows gave longer talks than would have been ideal, and tried to cover a lot of information. If we’d been able to better prompt fellows to give really snappy introductions to the motivation for their projects, this might have been worth doing.
Weekly slack check-in
Every Monday all fellows posted what they had done last week, what they were planning to do next week, and any problems or worries in Slack.
0.74
This was probably worth it as it was cheap, one of the only ways we created surface area between fellows in different check in groups, and wasn’t rated negatively by anyone.
The discussion group spreadsheet
A fellow set up a spreadsheet to coordinate discussions before and during the fellowship.
0.44
This didn’t really take off, but we think that it was worth it in expectation, and probably worth it overall (as 7 fellows rated it as valuable).
The default online office
In the first week, we experimented with having a default online office for one day. After the experiment we decided to implement this as a default office throughout the rest of the fellowship.
0.06
This seemed costly, as it hampered some more energetic attempts by fellows to coordinate online office spaces. We think that creating a really good online office space could be very valuable, but that cheap versions may be worse than nothing.
Mentoring
For fellows
We asked fellows the following questions:
Would you like to see your mentor mentoring other people in the future?
Average 4.77 out of 5
In what ways do you think another mentor might have been able to support you more?
8 fellows: more subject expertise
NB 7 of these fellows mostly scored the question about their mentor mentoring other people 5⁄5 (the final person rated it as 3⁄5), and 6 rated mentoring very valuable (the other 2 rated it as valuable)
4 fellows: more accountability/management/structure
2 fellows: more active mentorship in general
1 fellow each:
Feedback on written work
Different thinking style to me
More directive/forceful on research directions
More introductions
We think that subject expertise is a nice to have property in a mentor, but not strictly necessary for valuable mentoring pairs to be formed.
For mentors
We asked mentors the following questions:
How much better or worse do you feel about your mentoring than about what you would have done otherwise? (Counting both benefits to your mentee(s) and to you.)
Average 7.2 out of 10 (min 6)
If the same program (same organizers, same structure etc.) happened next year, how strongly would you recommend a friend (with similar background to you before the fellowship) to be a mentor?
Average 7.5 out of 10 (min 5)
I’ve learned a lot about how to mentor researchers.
Average 3.5 out of 5 (min 2)
I’m glad that I volunteered to be a mentor.
Average 4.3 out of 5 (min 3)
If we ran the fellowship again, what would you recommend we change?
2 mentors mentioned providing more advice and prompting on mentoring
1 mentor each mentioned
Sharing templates for mentors to request feedback from mentees
More involvement of mentors in the selection process
Nudging mentors to make introductions
Updating mentors on the career planning session we ran
In one sense, the ratings mentors gave seem quite low. However, they clear the bar of ‘worth my time overall’.
Culture
We asked fellows ‘Do you have any comments on the culture of the Summer Research Fellowship?’
12 fellows commented on the culture being quite good to fantastic in a generic way
9 fellows mentioned the culture being warm/kind/friendly
3 fellows appreciated the openness of the culture (in terms of sharing struggles and difficulties)
1 fellow mentioned each of the following as a problem with the culture:
Lack of Slack engagement
Sharing curiosities not being sufficiently common
Insufficient steps taken to mitigate imposter syndrome
Too much of an ‘in’ culture around EA/rationality
Fellows also commented on what had helped contribute to the culture
7 fellows mentioned modelling from Rose and Eliana
2 fellows mentioned the opening session
We think the culture was surprisingly good for a remote fellowship, and we guess that it helped to increase engagement, support those experiencing difficulties, and deepen fellows’ experiences.
Mental health
We asked fellows, ‘Did you experience any difficulties with mental health during the fellowship?’
15 people reported some difficulties, of whom 3 reported significant difficulties (according to us)
Of those 15 people, 4 reported that SRF contributed to the difficulties they experienced, specifically through:
Open ended commitments and unclear paths to success (2 fellows)
Feelings of imposter syndrome (1 fellow)
Logistical stuff to do with time zones (1 fellow, and we know of at least one additional fellow for whom this posed non-trivial challenges)
13 fellows reported that the fellowship supported them in their difficulties, mostly via 1-1s and other conversations
The intersection between this group of 13 and the group of 15 who reported difficulties is 12
The intersection between this group of 13 and the group of 4 where SRF contributed to their difficulties is 3
2 fellows sometimes felt unwelcome
In one case the fellow felt unwelcome in some heavily EA conversations where they had less background than the other participants
In another case the fellow felt unwelcome in some fellow-organised discussion groups
We think that given that the fellowship was remote, this rate of mental health difficulties isn’t surprising, and also that the counterfactual is unclear given the global pandemic.
We also think that anyone having difficulties with mental health that are exacerbated by the fellowship is bad, and we should endeavour to do better. In particular, we think future organisers should think more about ways of mitigating the mental health risks of the fellowship being open ended and (in its current form) having many and therefore unclear paths to success. It’s worth noting that we think that the opposite extreme (of a very clear and particular goal) is likely to be challenging for other fellows, and that the right solution will take variation among fellows into account.
Slack
We created a separate Slack workspace for the fellows and their mentors. Mentor engagement with the Slack workspace was optional, and in the end, minimal. We did not add fellows to the regular FHI Slack workspace, as it contains potentially sensitive information, and we didn’t want to flood the Slack with new members.
The Slack worked well as a coordination tool, but didn’t become a rich intellectual space. There weren’t many object-level research discussions, or much social engagement.
Not adding fellows to the FHI Slack worked well. We think we still managed to introduce fellows to FHIers in a way that was valuable.
Costs of running the fellowship
To FHI
The financial costs were all spent on stipends for fellows.
Researcher time talking to fellows. As organisers, we set up roughly 80 1-1 pairings between FHI researchers and fellows. Probably some of these never actually took place, but in addition many mentors made introductions too. Assuming that 1-1s lasted 60 minutes on average, this cost maybe 100 hours of FHI researcher time.
Admin time administering selection, setting up stipends and scheduling events (perhaps 110-130 hours).
A small number of people (2-6) gave us feedback on our plans, which cost a small amount of time (maybe 4-20 hours overall).
Around 20 FHIers spent some time reviewing some applications. We guess this cost 40-60 hours total.
To mentors
There were 18 mentors, who were researchers and research managers at FHI, CSER, OpenPhil, OpenAI, GPI and MIRI.
In addition, there were 2 informal mentors who agreed to speak more occasionally with a fellow. These informal mentors were at CSET and GPI.
All mentors were volunteers.
Most mentors met with fellows once a week for 30-60 minutes. These 18 mentors had 24 fellows between them.
Guessing that two thirds of the time mentors spent was literally in meeting fellows, this makes for a net time cost to mentors of roughly ~170 hours.
NB all mentors thought mentoring was net better than their counterfactual (average 7.2/10 compared with the counterfactual; minimum 6⁄10).
To the SRF organisers
Eliana Lorch and Rose Hadshar ran the fellowship. Neither tracked their time, but we guess that combined we spent the following amount of time on the fellowship:
Selection: maybe 3 weeks of work time
Planning the programme: maybe 1.5 weeks of work time
Running the programme: maybe 2.5 weeks of work time
We feel that this time was better spent on the fellowship than on our counterfactuals (in Rose’s case, more investment into RSP culture and programme structure; in Eliana’s case, more work on modeling).
This makes for a total of ~280 organising hours. The CEA fellowship in 2019 cost more like ~200 hours to its organisers, so this version was more time intensive. We expect that this comes in large part from the fact that we had triple the number of applications, took nearly three times the number of fellows, and needed to rethink the programme substantially to make it remote.
Organising was stressful for both Rose and Eliana at times.
To fellows
2 fellows reported feeling isolated at points during the fellowship
3 fellows reported that the fellowship exacerbated significant mental health difficulties they were facing
In one case, the fellowship contributed via pressure and feelings of imposter syndrome
In the other two cases, the fellowship contributed via unclear expectations and success criteria
Opportunity costs
1 fellow rated the fellowship as 5⁄10 compared with their counterfactual and we think had a frustrating experience of the fellowship
All other fellows rated the fellowship as net better than their counterfactual (average 8.6/10 compared with the counterfactual)
Benefits produced by the fellowship
Meta:
Unless stated otherwise, the following is based on a mix of (i) responses to feedback forms all fellows and mentors filled out at the end of the programme, (ii) Rose’s and Eliana’s impressions based on interactions with the fellows, their mentors etc.
A lot of this is subjective.
It’s also somewhat sensitive as it relates to particular people. We’ve anonymised as far as possible.
We think that most of the benefits of the fellowship won’t be visible until more time has passed. We intend to survey fellows again 6 months after the fellowship, and possibly 2 years afterwards too.
To fellows
Career updates
Concrete opportunities
1 fellow found a paid research position working for the person who mentored them on the fellowship
1 fellow received two grants to continue their fellowship project and to work on related research
2 fellows have ongoing collaboration/mentorship relationships
1 fellow became an affiliate at a longtermist research organisation
3 fellows have potential future collaborators
Motivations
4 fellows became more likely to work on things with longtermist implications
Studying
4 fellows updated towards further study (with longtermist relevance)
Research updates
4 fellows got a clearer idea of what to work on during their PhD or masters degree
2 fellows updated against research careers
2 fellows updated towards global priorities research
1 fellow shifted their idea of what to apply to do a PhD on
Meta
11 fellows have mentors who intend to speak with them in future about their career plans
In one case the mentor intends to substantially engage in discussions around choosing a PhD topic
Network
The median fellow reported knowing 5 people before the fellowship who met the following criteria:
a) have spent more time than you thinking about or working on questions of relevance to the long-term future, and
b) you’d feel comfortable reaching out to for advice or feedback
The median fellow reported knowing an additional 6 people as a result of the fellowship who met the same criteria
This represents an average increase of fellows’ long termist networks of around 100%
One fellow who had very little exposure to EA prior to the fellowship reported that it was a positive introduction to EA for them
Knowledge and skills
9 fellows gained significant knowledge on various aspects of longtermism
8 fellows gained significant research skills
In most cases, these were fellows with little prior research experience
In a few cases, skills were gained by more experienced researchers around research taste
4 fellows reported improving on other skills or traits
Agenti-ness, productivity, and confidence were named
Outputs
This list contains outputs where there was a publicly accessible link, and/or where fellows were happy to include their name and email address against their work. There were many other outputs where neither of these things applied, which we have excluded from this list.
Note that in some cases only parts of the output were produced during the fellowship. However, we think that for most if not all of these, these texts would not exist, or would have been published later or with worse quality, if not for the fellowship.
Promoting versatile vaccine development for emerging pandemics (Joshua Monrad joshua.monrad@gmail.com, with Jonas Sandbrink and Neil Cherian. Currently under review with an academic journal)
A draft LessWrong post on future trajectories for democracy and voting assistants (Wiona Glänzer, wiona.glaenzer@icloud.com)
A draft EA forum post with the working title ‘A Review of Moral Psychology Research Relevant to Moral Circle Expansion’ (Megan Kinniment, megan.k.w.12@gmail.com)
An article titled ‘National Interests or the Greater Good? Dual Moral Obligations and International Cooperation’ (Jenny Xiao, j.xiao@columbia.edu)
One fellow was able to facilitate in-depth exchange between RSP management and another research organisation on running research teams
One fellow changed the focus of their masters dissertation to AI safety
One fellow was able to better prepare for starting their job at a think tank
To mentors
1 mentor’s fellow was working on a project from the mentor which otherwise the mentor would have wanted to do themselves. The fellow’s time was a bit under 1:1 equivalent of the mentor’s time on the project, and the fellow extended the paper following on from the fellowship
3 mentors updated towards mentoring in future, significantly in 2 cases
2 mentors clarified some of their own research thinking
4 mentors learned some useful object level things
1 mentor reported improving at mentoring
Mentors updated on how promising 9 fellows were in the course of interacting with them
Comparing costs and benefits
We think that the fellowship was pretty clearly net positive, but that it also wasn’t clearly an outstanding success.
In particular, at the end of the 2019 CEA fellowship, we think there were proportionally more clear success cases to point to than there are at the same point for the 2020 FHI fellowship. This might in part be because there were fewer fellows in 2019, which might mean a) fellows had more input per head and this created more value, and b) organisers had a better sense of how individual fellows had developed, making it easier to spot successes. It might also be because the fellowship was remote and this made it less valuable. Small sample size and frequency of outliers might also mean that this is explained just by luck.
We expect more of the impact of this fellowship to become clear later as fellows progress in their careers, and intend to survey fellows again 6 months after the fellowship and possibly 2 years after.
Mistakes
We’re sure we made other mistakes too, but the ones that stand out in chronological order are:
Rose invested insufficient time in managing the person providing administrative support during the application process. This created unnecessary inefficiency, confusion and delays for organisers and applicants.
We’d make some hiring decisions differently now, and we think for some of them we could have realized this ex ante.
We didn’t communicate promptly enough with fellows about the expectations we had of them, or the structure and schedule of the programme.
In the first week the timings of a number of events were changed last minute, and this created confusion that seemed particularly costly at the beginning of the fellowship.
We instituted a default online office space, but didn’t invest any energy or thought into integrating this well with other parts of the fellowship or fellows’ initiatives. This led to a fairly inactive online office space, and probably hindered some more energetic initiatives from fellows.
We didn’t communicate clearly with fellows about how to prepare and give good lightning talks, and this made these sessions less valuable than otherwise.
We didn’t put conscious attention on how to elicit negative feedback (anonymous feedback options, options to give feedback to non-organisers). We expect this leads to a positive skew in our evaluation here.
We may consider sharing some materials and resources with organisers of similar programmes in future. If you think these materials would be useful to you, please email srf@philosophy.ox.ac.uk explaining why this would be useful, and we’ll consider your request.
Review of FHI’s Summer Research Fellowship 2020
This post reviews the Future of Humanity Institute’s Summer Research Fellowship 2020 in detail. If you’re in a hurry, we recommend reading the summary of activities, lessons learned, and comparing costs and benefits sections for a quicker take.
Thanks to Owen Cotton-Barratt, Max Daniel, Eliana Lorch, Tanya Singh and the summer fellows for reviewing and improving this post.
Summary of activities
27 fellows spent 6 weeks on a remote fellowship programme.
We got ~300 applications and interviewed ~50 people.
Backgrounds ranged from undergraduate students to postdocs.
By default, each fellow had a mentor and worked on a research project (though there were some exceptions, and many fellows also spent significant time exploring).
The mentors were researchers and research managers at FHI, CSER, OpenPhil, OpenAI, GPI, and MIRI.
Approximately, the projects related to AI governance (13 fellows), technical AI safety (3 fellows), biosecurity (4 fellows), macrostrategy broadly construed (5 fellows), and policy (2 fellows).
This fellowship grew out of the CEA summer fellowship (which CEA wanted to stop running), though was significantly different in that a) it was remote, and b) we took more fellows (in 2019, the CEA fellowship took 11 fellows).
Lessons learned
Meta: we’re stating these confidently to make clear which direction the summer fellowship has updated us in. Obviously this was just one summer programme, and in many cases there might be important differences which mean that these takeaways wouldn’t generalise. The takeaways are in rough order of importance according to us.
Remote fellowships can provide substantial value to fellows and organizers, and are worth considering where in person options are not available, or are especially costly.
Initially we put some probability on the whole fellowship being awkward and unproductive, but in fact fellows had good experiences and some good things seem to have come from the programme. Remoteness also allowed us to take more fellows, and in the particular case of Covid-19, also came with lower opportunity costs for many fellows and organisers.
In spite of remoteness, it was possible to create a positive, friendly and open culture. Many fellows remarked on this and appreciated it, and we guess that it helped to increase engagement, support those experiencing difficulties, and deepen fellows’ experiences. Modelling open and authentic communication on the part of the organisers seems to have been a major contributing factor to this culture.
That said, we still think that there are many ways in which an in person fellowship would have been better, particularly in terms of culture, networking and some kinds of logistics.
There is more interest than we expected in mentoring people amongst researchers in a range of different EA organisations. We successfully found 18 mentors for 24 fellows (3 fellows didn’t have formal mentors for various reasons), and we contacted at least a further 10 people who we think would have mentored if there had been a fellow working on the right topic. We also didn’t do a comprehensive search for mentors, and instead reached out ad hoc to people in our network, so we expect we would have found more interest if we had searched a bit harder.
There were more promising applicants for this programme than we expected. In 2019 the CEA fellowship received ~90 applications, and this fellowship received ~300. We think that the number of strong applicants was correspondingly larger.
Minor difficulties with mental health were common, and more serious ones not extremely rare (although this is confounded by remoteness and Covid-19). 1-1s and other conversations seemed to help support people with this. We think it would be good for future organisers to invest more time in thinking about how to mitigate the risks from a) imposter syndrome, and b) the open ended nature of the fellowship and the lack of clear success metrics.
The personal fit between mentor and fellow might potentially be very impactful. In the two cases where the fellowship led directly on to further remunerated collaboration between the fellow and the mentor, we selected the mentorship pairing in part because we expected the two people to get on well together personally. It’s unclear how important this was, but we think it’s worth paying attention to in future pairings.
A significant part of the value of (this) fellowship came from providing mentorship and network.
There’s significant interest in the fellowship being longer than 6 weeks, from fellows and mentors, though it’s not clear to us if this is net worth it.
Many fellows thought that a mentor with more subject expertise would have been able to support them more. However, most of these fellows also had a strong positive experience with their actual mentor, suggesting that this is a nice to have rather than an essential.
Low-input attempts to create good online office space can be net harmful. We set up a default remote office space, which hampered some more energetic attempts from fellows to create good online environments.
Putting on talks especially for fellows was high time cost and only somewhat valuable. Especially in the context of a remote programme or in the case of recorded talks, we expect that it would have been better to either not do this at all, or to invest even more time in making the series really good.
Communicate clearly to fellows before the start of the programme that there are multiple different outcomes that look like success, if this is the case. We communicated this at the start of the programme, which we think was better than not communicating it at all or communicating it later, but we think that communicating earlier than this would likely have better helped fellows to plan and orient.
The learning value of running such a programme for organisers can be very high. Rose had already co-ran a fellowship similar to this one, but still learned a lot (in part because this fellowship was remote, in part because she had a more empowered role on this fellowship). Eliana hadn’t previously run a fellowship like this and also learned a lot.
Both Rose and Eliana had a lot of vision for this fellowship, but not for a remote version of it, and were initially unexcited about a remote version. We were surprised that it went well given that we didn’t have a clear vision of how to run this remotely.
Goals of the fellowship
The main goal of the fellowship is to help young people to do better things from a longtermist perspective than they would otherwise have done.
We expect most of this change to happen through:
Changing fellows’ career plans, by putting fellows in the way of concrete opportunities, shifting fellows’ motivations, and helping fellows to test their fit.
Improving fellows’ networks, both among other fellows and with more senior people.
Improving fellows’ knowledge and skills in a range of ways, including research taste, research experience, and psychological empowerment.
We try to meet this goal by:
Selecting high potential young people where we think the fellowship is better than what they could counterfactually be spending time on.
Bringing them into a rich longtermist environment.
Giving them the affordance to work on a relevant project.
Providing them with a mentor who can give them feedback.
Introducing them to each other and relevant researchers in the space.
Detail on activities
Timeline of activities
January/February 2020: Rose and Eliana decided to take on the project together
19th February: application page goes live (deadline 22nd March)
We asked for a CV and a research proposal
The research proposal was for the purposes of evaluation; people were free to work on other things during the programme
March-April: desk rejects and interview decisions
Late April: interviews
Early May: offers
13th July: fellowship started
24th August: fellowship ended
Summer fellows
There were 27 fellows in all. Below is a list of those fellows who explicitly gave us permission to name them in this post. See here for more extended bios.
Evaluation of aspects of the fellowship
The fellowship as a whole
We asked fellows the following questions:
How much better or worse do you feel about the summer fellowship than about what you would have done otherwise?
Average 8.58 out of 10, min 5
If the same program (same organizers, same structure etc.) happened next year, how strongly would you recommend a friend (with similar background to you before the fellowship) to apply?
Average 8.89 out of 10, min 2 (then 7)
We think these are fairly good responses overall. The minimum responses to both questions came from the same fellow.
Length of the programme
Fellows’ opinions
We asked fellows ‘What % of the value you’ve gotten from the fellowship so far would you attribute to each week of the fellowship?’
The average responses were 15%, 17%, 16%, 15%, 14% and 16% for weeks 1,2,3,4,5 and 6
This suggests that returns weren’t diminishing over the 6 weeks
Of the 27 fellows, 5 said in the final survey that they wished the fellowship had been longer. We guess that an additional 3 might also have benefitted from a longer fellowship
Mentors’ opinions
2 mentors suggested making the fellowship longer in future
Our thoughts
It’s worth noting that the CEA summer fellowship was 6 weeks in 2018, and fellows said it was too short. The 2019 CEA fellowship was therefore lengthened to 8 weeks, and fellows again said it was too short. However, the organisers thought that the 2019 fellowship was a success, and that an additional 2 weeks would not have increased the impact of the programme much.
There’s some tension between the content of the fellowship and where we think the impact comes from. Fellows undertake a research project during the fellowship; we think that 6 weeks is a little too short for this, especially given the extra time needed to orient to a new environment. However, we don’t think that the main value of the fellowship comes from completed research outputs. Instead, we expect most of the value to come for career updates, networking and gaining knowledge and skills. 6 weeks seems enough to capture significant value here.
We think that organisers next year should seriously consider making the fellowship longer, but we don’t think it’s clear that they should.
Program components
We asked fellows ’How valuable did you find the following [programme components]?
The options were Harmful, ~Neutral, Valuable, Very Valuable, and NA
Assigning −1 to Harmful, 0 to ~Neutral, 1 to Valuable, and 2 to Very valuable, the programme components were rated in the following way:
1.73
These seem like the two most important parts of the fellowship structure. In other sections of the survey, multiple fellows stressed how valuable these things were.
See below for more details on mentoring.
Setting up the 1-1 introductions was time intensive, but seems worth it.
1.67
1.44
1.38
1.32
1.29
1.26
1.25
1.24
1.19
1.08
1.04
1.00
0.86
0.81
This score was pulled down somewhat by the responses from one group which didn’t go well (the score would be 0.96 excluding that group).
If this is repeated, it would be good to have a better way of intervening on check in groups when they are not working (e.g. by organisers attending all groups once, or checking in with every fellow individually).
In spite of the low score, we expect that given the remote nature of the fellowship and the large number of fellows, having smaller groups to talk with regularly may have been important for creating a friendly, supportive culture. At least two check in groups continued meeting after the fellowship, additionally suggesting that this was valuable for some fellows.
0.76
0.74
0.44
0.06
Mentoring
For fellows
We asked fellows the following questions:
Would you like to see your mentor mentoring other people in the future?
Average 4.77 out of 5
In what ways do you think another mentor might have been able to support you more?
8 fellows: more subject expertise
NB 7 of these fellows mostly scored the question about their mentor mentoring other people 5⁄5 (the final person rated it as 3⁄5), and 6 rated mentoring very valuable (the other 2 rated it as valuable)
4 fellows: more accountability/management/structure
2 fellows: more active mentorship in general
1 fellow each:
Feedback on written work
Different thinking style to me
More directive/forceful on research directions
More introductions
We think that subject expertise is a nice to have property in a mentor, but not strictly necessary for valuable mentoring pairs to be formed.
For mentors
We asked mentors the following questions:
How much better or worse do you feel about your mentoring than about what you would have done otherwise? (Counting both benefits to your mentee(s) and to you.)
Average 7.2 out of 10 (min 6)
If the same program (same organizers, same structure etc.) happened next year, how strongly would you recommend a friend (with similar background to you before the fellowship) to be a mentor?
Average 7.5 out of 10 (min 5)
I’ve learned a lot about how to mentor researchers.
Average 3.5 out of 5 (min 2)
I’m glad that I volunteered to be a mentor.
Average 4.3 out of 5 (min 3)
If we ran the fellowship again, what would you recommend we change?
2 mentors mentioned providing more advice and prompting on mentoring
1 mentor each mentioned
Sharing templates for mentors to request feedback from mentees
More involvement of mentors in the selection process
Nudging mentors to make introductions
Updating mentors on the career planning session we ran
In one sense, the ratings mentors gave seem quite low. However, they clear the bar of ‘worth my time overall’.
Culture
We asked fellows ‘Do you have any comments on the culture of the Summer Research Fellowship?’
12 fellows commented on the culture being quite good to fantastic in a generic way
9 fellows mentioned the culture being warm/kind/friendly
3 fellows appreciated the openness of the culture (in terms of sharing struggles and difficulties)
1 fellow mentioned each of the following as a problem with the culture:
Lack of Slack engagement
Sharing curiosities not being sufficiently common
Insufficient steps taken to mitigate imposter syndrome
Too much of an ‘in’ culture around EA/rationality
Fellows also commented on what had helped contribute to the culture
7 fellows mentioned modelling from Rose and Eliana
2 fellows mentioned the opening session
We think the culture was surprisingly good for a remote fellowship, and we guess that it helped to increase engagement, support those experiencing difficulties, and deepen fellows’ experiences.
Mental health
We asked fellows, ‘Did you experience any difficulties with mental health during the fellowship?’
15 people reported some difficulties, of whom 3 reported significant difficulties (according to us)
Of those 15 people, 4 reported that SRF contributed to the difficulties they experienced, specifically through:
Open ended commitments and unclear paths to success (2 fellows)
Feelings of imposter syndrome (1 fellow)
Logistical stuff to do with time zones (1 fellow, and we know of at least one additional fellow for whom this posed non-trivial challenges)
13 fellows reported that the fellowship supported them in their difficulties, mostly via 1-1s and other conversations
The intersection between this group of 13 and the group of 15 who reported difficulties is 12
The intersection between this group of 13 and the group of 4 where SRF contributed to their difficulties is 3
2 fellows sometimes felt unwelcome
In one case the fellow felt unwelcome in some heavily EA conversations where they had less background than the other participants
In another case the fellow felt unwelcome in some fellow-organised discussion groups
We think that given that the fellowship was remote, this rate of mental health difficulties isn’t surprising, and also that the counterfactual is unclear given the global pandemic.
We also think that anyone having difficulties with mental health that are exacerbated by the fellowship is bad, and we should endeavour to do better. In particular, we think future organisers should think more about ways of mitigating the mental health risks of the fellowship being open ended and (in its current form) having many and therefore unclear paths to success. It’s worth noting that we think that the opposite extreme (of a very clear and particular goal) is likely to be challenging for other fellows, and that the right solution will take variation among fellows into account.
Slack
We created a separate Slack workspace for the fellows and their mentors. Mentor engagement with the Slack workspace was optional, and in the end, minimal. We did not add fellows to the regular FHI Slack workspace, as it contains potentially sensitive information, and we didn’t want to flood the Slack with new members.
The Slack worked well as a coordination tool, but didn’t become a rich intellectual space. There weren’t many object-level research discussions, or much social engagement.
Not adding fellows to the FHI Slack worked well. We think we still managed to introduce fellows to FHIers in a way that was valuable.
Costs of running the fellowship
To FHI
The financial costs were all spent on stipends for fellows.
Researcher time talking to fellows. As organisers, we set up roughly 80 1-1 pairings between FHI researchers and fellows. Probably some of these never actually took place, but in addition many mentors made introductions too. Assuming that 1-1s lasted 60 minutes on average, this cost maybe 100 hours of FHI researcher time.
Admin time administering selection, setting up stipends and scheduling events (perhaps 110-130 hours).
A small number of people (2-6) gave us feedback on our plans, which cost a small amount of time (maybe 4-20 hours overall).
Around 20 FHIers spent some time reviewing some applications. We guess this cost 40-60 hours total.
To mentors
There were 18 mentors, who were researchers and research managers at FHI, CSER, OpenPhil, OpenAI, GPI and MIRI.
In addition, there were 2 informal mentors who agreed to speak more occasionally with a fellow. These informal mentors were at CSET and GPI.
All mentors were volunteers.
Most mentors met with fellows once a week for 30-60 minutes. These 18 mentors had 24 fellows between them.
Guessing that two thirds of the time mentors spent was literally in meeting fellows, this makes for a net time cost to mentors of roughly ~170 hours.
NB all mentors thought mentoring was net better than their counterfactual (average 7.2/10 compared with the counterfactual; minimum 6⁄10).
To the SRF organisers
Eliana Lorch and Rose Hadshar ran the fellowship. Neither tracked their time, but we guess that combined we spent the following amount of time on the fellowship:
Selection: maybe 3 weeks of work time
Planning the programme: maybe 1.5 weeks of work time
Running the programme: maybe 2.5 weeks of work time
We feel that this time was better spent on the fellowship than on our counterfactuals (in Rose’s case, more investment into RSP culture and programme structure; in Eliana’s case, more work on modeling).
This makes for a total of ~280 organising hours. The CEA fellowship in 2019 cost more like ~200 hours to its organisers, so this version was more time intensive. We expect that this comes in large part from the fact that we had triple the number of applications, took nearly three times the number of fellows, and needed to rethink the programme substantially to make it remote.
Organising was stressful for both Rose and Eliana at times.
To fellows
2 fellows reported feeling isolated at points during the fellowship
3 fellows reported that the fellowship exacerbated significant mental health difficulties they were facing
In one case, the fellowship contributed via pressure and feelings of imposter syndrome
In the other two cases, the fellowship contributed via unclear expectations and success criteria
Opportunity costs
1 fellow rated the fellowship as 5⁄10 compared with their counterfactual and we think had a frustrating experience of the fellowship
All other fellows rated the fellowship as net better than their counterfactual (average 8.6/10 compared with the counterfactual)
Benefits produced by the fellowship
Meta:
Unless stated otherwise, the following is based on a mix of (i) responses to feedback forms all fellows and mentors filled out at the end of the programme, (ii) Rose’s and Eliana’s impressions based on interactions with the fellows, their mentors etc.
A lot of this is subjective.
It’s also somewhat sensitive as it relates to particular people. We’ve anonymised as far as possible.
We think that most of the benefits of the fellowship won’t be visible until more time has passed. We intend to survey fellows again 6 months after the fellowship, and possibly 2 years afterwards too.
To fellows
Career updates
Concrete opportunities
1 fellow found a paid research position working for the person who mentored them on the fellowship
1 fellow received two grants to continue their fellowship project and to work on related research
2 fellows have ongoing collaboration/mentorship relationships
1 fellow became an affiliate at a longtermist research organisation
3 fellows have potential future collaborators
Motivations
4 fellows became more likely to work on things with longtermist implications
Studying
4 fellows updated towards further study (with longtermist relevance)
Research updates
4 fellows got a clearer idea of what to work on during their PhD or masters degree
2 fellows updated against research careers
2 fellows updated towards global priorities research
1 fellow shifted their idea of what to apply to do a PhD on
Meta
11 fellows have mentors who intend to speak with them in future about their career plans
In one case the mentor intends to substantially engage in discussions around choosing a PhD topic
Network
The median fellow reported knowing 5 people before the fellowship who met the following criteria:
a) have spent more time than you thinking about or working on questions of relevance to the long-term future, and
b) you’d feel comfortable reaching out to for advice or feedback
The median fellow reported knowing an additional 6 people as a result of the fellowship who met the same criteria
This represents an average increase of fellows’ long termist networks of around 100%
One fellow who had very little exposure to EA prior to the fellowship reported that it was a positive introduction to EA for them
Knowledge and skills
9 fellows gained significant knowledge on various aspects of longtermism
8 fellows gained significant research skills
In most cases, these were fellows with little prior research experience
In a few cases, skills were gained by more experienced researchers around research taste
4 fellows reported improving on other skills or traits
Agenti-ness, productivity, and confidence were named
Outputs
This list contains outputs where there was a publicly accessible link, and/or where fellows were happy to include their name and email address against their work. There were many other outputs where neither of these things applied, which we have excluded from this list.
Note that in some cases only parts of the output were produced during the fellowship. However, we think that for most if not all of these, these texts would not exist, or would have been published later or with worse quality, if not for the fellowship.
Completed outputs:
Clarifying “What failure looks like” (part 1)
Notable works in progress include:
Draft papers on:
Labor, Capital, and the Optimal Growth of Social Movements
Promoting versatile vaccine development for emerging pandemics (Joshua Monrad joshua.monrad@gmail.com, with Jonas Sandbrink and Neil Cherian. Currently under review with an academic journal)
Metagenomic sequencing and public health surveillance (Sid Sharma, sidsharma1125@gmail.com)
A discussion on the statistical nature of threats to historical civilizations (Nithin Sivadas, nithin@bu.edu)
A draft LessWrong post on future trajectories for democracy and voting assistants (Wiona Glänzer, wiona.glaenzer@icloud.com)
A draft EA forum post with the working title ‘A Review of Moral Psychology Research Relevant to Moral Circle Expansion’ (Megan Kinniment, megan.k.w.12@gmail.com)
An article titled ‘National Interests or the Greater Good? Dual Moral Obligations and International Cooperation’ (Jenny Xiao, j.xiao@columbia.edu)
Other effects
Two fellows and a third collaborator won the 2020 Next Generation for Biosecurity Competition with a joint paper, Widening the Framework for Regulation of Dual-Use Research in the Wake of the COVID-19 Pandemic. This collaboration would not have happened without the collaborators meeting through SRF
One fellow was able to facilitate in-depth exchange between RSP management and another research organisation on running research teams
One fellow changed the focus of their masters dissertation to AI safety
One fellow was able to better prepare for starting their job at a think tank
To mentors
1 mentor’s fellow was working on a project from the mentor which otherwise the mentor would have wanted to do themselves. The fellow’s time was a bit under 1:1 equivalent of the mentor’s time on the project, and the fellow extended the paper following on from the fellowship
3 mentors updated towards mentoring in future, significantly in 2 cases
2 mentors clarified some of their own research thinking
4 mentors learned some useful object level things
1 mentor reported improving at mentoring
Mentors updated on how promising 9 fellows were in the course of interacting with them
Comparing costs and benefits
We think that the fellowship was pretty clearly net positive, but that it also wasn’t clearly an outstanding success.
In particular, at the end of the 2019 CEA fellowship, we think there were proportionally more clear success cases to point to than there are at the same point for the 2020 FHI fellowship. This might in part be because there were fewer fellows in 2019, which might mean a) fellows had more input per head and this created more value, and b) organisers had a better sense of how individual fellows had developed, making it easier to spot successes. It might also be because the fellowship was remote and this made it less valuable. Small sample size and frequency of outliers might also mean that this is explained just by luck.
We expect more of the impact of this fellowship to become clear later as fellows progress in their careers, and intend to survey fellows again 6 months after the fellowship and possibly 2 years after.
Mistakes
We’re sure we made other mistakes too, but the ones that stand out in chronological order are:
Rose invested insufficient time in managing the person providing administrative support during the application process. This created unnecessary inefficiency, confusion and delays for organisers and applicants.
We’d make some hiring decisions differently now, and we think for some of them we could have realized this ex ante.
We didn’t communicate promptly enough with fellows about the expectations we had of them, or the structure and schedule of the programme.
In the first week the timings of a number of events were changed last minute, and this created confusion that seemed particularly costly at the beginning of the fellowship.
We instituted a default online office space, but didn’t invest any energy or thought into integrating this well with other parts of the fellowship or fellows’ initiatives. This led to a fairly inactive online office space, and probably hindered some more energetic initiatives from fellows.
We didn’t communicate clearly with fellows about how to prepare and give good lightning talks, and this made these sessions less valuable than otherwise.
We didn’t put conscious attention on how to elicit negative feedback (anonymous feedback options, options to give feedback to non-organisers). We expect this leads to a positive skew in our evaluation here.
We may consider sharing some materials and resources with organisers of similar programmes in future. If you think these materials would be useful to you, please email srf@philosophy.ox.ac.uk explaining why this would be useful, and we’ll consider your request.