This post is part of the “Insights from the 2023 summer research fellowship of the Existential Risk Alliance (ERA)” sequence, outlining lessons learned from running the ERA Fellowship programme in 2023. For more information, see the post introducing this sequence.
We decided to publish this post summarising lessons from the 2023 ERA fellowship because we believe there is value in transparently sharing our experiences running one of the largest existential risk summer programs. While we made an effort to survey participants and learn from their feedback, we recognise that others may identify additional strengths or weaknesses based on a high-level summary. By putting these reflections out into the community, we hope to catalyse discussion that will help us and others improve future programming aimed at building existential risk research capacity.
Executive Summary
At ERA, we tried to create a culture where fellows felt able to explore diverse x-risk ideas openly and collaboratively
Our programming included 34 speaker events and 4 discussion groups. We contained a number of speaker events in our 2 day long intro workshop and our day trip to Trajan House.
The intro workshop and the reading groups particularly helped fellows gain knowledge; the mean satisfaction with knowledge acquisition was 8.4/10.
Events exposed fellows to many new ideas and cause areas from a diverse range of perspectives, and survey responses suggest that this was a key contributor to a truth-seeking atmosphere during the Fellowship.
Social connections and professional collaboration between fellows proved very important, as many rated these components as one of the most important aspects of the programme.
Areas for improvement include organising events earlier, providing more research methodology training and networking events as well as sharing events more broadly (including with other fellowship programmes).
Introduction
During the 2023 ERA Cambridge summer research fellowship, we attempted to create a culture where participants felt able to explore a variety of different ideas related to existential risk (x-risk) - we were intent on creating a space where questioning and exploration of different approaches was encouraged, and wanted to avoid making participants feel like they had to agree with our views on x-risk, but rather to upskill into strong x-risk researchers who could develop important new ideas. Essentially, we wanted to create a space where discovery and debate was encouraged in a collaborative rather than tense way.
ERA’s Culture
In general, the culture around the office felt exceptionally friendly, with discussion between fellows of different cause areas being a common occurrence.
Discussion on a multitude of key issues around existential risk happened a lot, with a variety of fellows taking different views.
It was very clear that there wasn’t an ‘orthodoxy’ on ways to approach this, which seemed to be very useful in allowing for exploration of a variety of approaches. We took active steps to communicate this to fellows (one example of this can be found in the Appendix).
Participants seemed not only excited about exploring ideas around x-risk that seem common in EA, but also a much broader range of ideas including those that may be much less commonly discussed in EA spaces.
Participants seemed willing and interested to talk about and explore other cause areas; almost every event (including reading groups) had participants from a variety of cause areas.
Fellows’ professional connections with each other seemed really crucial. Many participants rated interactions with other fellows to be the most important aspect of the fellowship for them.
Fellows also developed strong social connections with each other away from work settings as well. This likely helped in creating a positive work environment that allowed debates, disagreements and divergences to be fostered in a friendly way, and we expect this to aid our efforts to build a sustainable and collaboratively x-risk research community even after the Fellowship ends.
Overview of Programming
During the 8 week fellowship, we ran 34 ‘speaker events’ (including talks, Ask Me Anythings and discussions with external speakers), and hosted an existential risks discussion group with an external facilitator (APOCALYPSE), facilitated 2 cause-specific discussion groups in Cambridge (AI and Nuclear) and joined an existing cause-specific discussion group (GCBR). Fellows also organiseda one-off discussion group (Climate + Meta) and ran weekly reflection activities. A full list of speaker events is in the appendix.
We began the fellowship with a 2-day introductory workshop on Tuesday and Wednesday of week 1, followed by a visit from Michael Aird on Monday and Tuesday of week 2. The introductory workshop had the following talks:
Gideon Futerman (ERA): Key Concepts in Existential Risk Studies
Luke Kemp (at the time CSER): Agents of Doom
Lalitha Sundaram (CSER): Governance in Existential Risk Studies
SJ Beard (CSER): History and Ethics in Existential Risk Studies
Michael Aird—recording (Rethink Priorities): Theory of Change Workshop
Lara Mani and Luke Kemp (CSER): Futures and Foresight Methods
Lara Mani (CSER): How to do a literature review
SJ Beard (CSER): Methods in Existential Risk Studies
These were meant to be accessible for those with no experience in x-risk (as some of our fellows had never engaged explicitly with x-risk before), but also provide very useful context and content for those heavily involved in x-risk discussions. In general, they were focused on ‘big questions’ of x-risk, and attempted to give an overview of the breadth of the field.
Fellows highly valued this set of workshops, with many explicitly mentioning it in their post-fellowship feedback[1]. Those participants with little x-risk experience seemed to get the most out of it, with it seemingly bringing everyone ‘up to speed’ rather quickly. A few fellows already involved in x-risk found it slightly less useful, though most with existing experience still found it informative. The initial workshops influenced the research questions for some fellows, and, according to others, may have changed the direction of their projects if questions had not already been decided upon.
Theory of Change for the Programming
Note: This was written before the programme started.
Key objectives:
Help inform the work of the fellows during the fellowship, including research skills and other useful skills they will directly use in the fellowship
This should directly improve their research output, which is a useful x-risk end in itself. It also will make the research experience more useful and easier, meaning retention in x-risk research is likely higher
Provide fellows with skills, knowledge and concepts useful to specific cause areas and x-risk in general.
This is about training them to be good x-risk researchers, giving them a good basis for further work and awareness of the field as a whole to help with research and prioritisation
Provide fellows with networking opportunities and understanding of the social organisation of their cause area and x-risk in general
This should allow them to be better involved in the core discussions going on in the field, allowing them to be exposed to more ideas via diffusion, opportunities and connections, as well as allowing their research to in turn be communicated to these audiences
We want fellows to leave the fellowship not just with an understanding of what they researched, but a better grasp of their cause area and x-risk as a whole. This means programming should be intentionally broad to expose fellows to as wide a range of concepts as possible, whilst also in depth enough to provide them with a solid basis to explore these concepts further if they so wish. Equipping fellows with this diverse conceptual and skills toolkit is a key outcome of the fellowship that is operationalised through the programming.
The programming character should change through the fellowship. The start should focus on skills and concepts directly relevant, or could be directly relevant, to their projects, and on building strong social connections between the fellows. As time goes on, the focus should shift towards networking and broader concepts and skills that may be useful to the fellows in the future.
This is essentially based on what is necessary to make the fellows as useful additions to x-risk as possible: equipping them with skills and knowledge to do novel and useful research towards reducing x-risk, and a network and awareness of the field that will allow them to operationalise this into useful and impactful research.
Did the programming fulfil the Theory of Change:
Our programming contributed to different fellows’ research in different ways. In general, for more narrowly scoped projects, the impact tended to be limited, whilst broader projects tended to benefit more. Whether better focused programming could have actually assisted with the more specific projects over such a short window is unclear. Many participants noted the lack of programming for specific research skills that may have been useful specifically for their projects. This may be reflected in the lower score (mean) given for gain in research skills and competencies when compared to knowledge acquisition (7.1 compared to 8.4).
The programming was clearly very useful at helping improve fellows’ knowledge of x-risk in general, and for their cause area in particular. The first of these two aspects seemed to be stronger than the latter. The mean satisfaction of knowledge acquisition was 8.37, with the programming, particularly the reading group and initial workshops being mentioned in responses a large number of times. (Note the mean satisfaction with events was 7.7, although most negative comments were related to the last-minute organisation; indeed one fellow rated the quality of events at 9 and the organisation of them at 2).
It also somewhat contributed to understanding the social organisation of the cause area, and some commented on how useful the connections with speakers were (e.g. one person has said it has helped them consider a job at the speaker’s institution). One speaker explicitly offered attendants to reach out to them afterwards, an offer that many attendants of that event picked them up on, leading to multiple calls with the speaker that we have reason to believe were counterfactually very unlikely to happen. Nonetheless, other factors such as discussion with mentors, research managers and other fellows seemed overall more important in connecting people and giving fellows the awareness of what their cause area actually looks like than the events.
The shift towards networking[2] didn’t really occur. Fellows did visit other institutions (some attended the Cooperative AI Summer School, and there were visits to Conjecture and Trajan House) which seemed to be quite valuable. Research managers and mentors also put fellows in contact with relevant researchers, and fellows did establish connections with speakers. However, as the programme went on, fellows wanted to spend more time on their research, and because of holidays August is a hard time to organise events. Nonetheless, it is likely that networking opportunities could be improved in the future.
Generally, we feel the programming gave participants greater awareness of key concepts and ideas across x-risk in a lot of depth, helping skill them up to be active participants in the x-risk field.
How can programming contribute to culture and overall aims
The initial workshops (and Michael Aird’s week 2 visit) seemed to immediately signal to participants encouragement for exploration of a variety of viewpoints, and that disagreement would be encouraged, as long as it was truth-seeking (medium confidence).
This is because speakers openly presented different and disagreeing perspectives about core issues in x-risk. Some speakers were part of the EA community, some ambivalent and some critical of the EA community.
The initial workshop and the APOCALYPSE reading group encouraged participants to continually explore ‘big questions’ about x-risk, how to understand it and how to work on it (medium confidence).
There are compounding factors here, in that a number of the projects were engaging with many big questions of how to conceptualise x-risk.
The reading and discussion groups encouraged participants to learn and discuss x-risk topics with people in other cause areas (high confidence). Having ‘general’ events, and having specific events for every cause area encouraged an ‘equality’ of engagement where people from each cause area learnt things from other cause areas (weak confidence), and led to broader discussion about ‘big questions’ in XRisk (medium confidence).
In Week 3, we introduced the ERA Research Circles, cause area-specific reading and discussion groups to facilitate knowledge exchange amongst the fellows. The idea is that Fellows can meet each week to discuss research papers relevant to their cause areas, or wider existential risks research. To encourage the formation of these groups, we supported fellows by covering dinner for the evenings of such discussions. This led to the formation of 2 fellow-led discussion groups. These Research Circles were not restricted specifically to fellows from specific cause areas and helped contribute to a culture of open inquiry and collaboration. Fellows from different cause areas joined each group, exposing themselves to new perspectives and allowing for lively debate that tested assumptions on all sides.
What we ought to improve
Organisation: Clearly signal when events are (rather than just putting them in a calendar). This was the most common complaint of the event organisation.
Nonetheless, the flexibility that the last minute organising gave to speakers
Some participants desired more Q&As or Ask-Me-Anything type sessions. This may have been more useful, as it is also easier for speakers generally. However, it also may somewhat limit the range of ideas explored and that the participants are exposed to.
More programming focused on research skills and methodologies. This is something we attempted, however, we feel we did this rather unsuccessfully. Partially this is because such research skills may be specific to the project (the skills needed for a technical AI Agent Foundation Project may be very different to those needed in a project analysing tail-risk treaties). Working out how to do appropriately tailored programme to research skills and methods will be a challenge, but would potentially be very useful to certain fellows. Moreover, there does generally seem to be a lack of training that can be provided to early career researchers on what good methodologies for x- risk research looks like.
Sharing more! The initial workshops had some people from the GCR community in Latin America and from ILINA join, and some of the later programming had participants from CHERI and XLab join. The quality of these sessions seemed no worse when more people joined. Given how valuable they were, and given how many sessions ERA managed to get (35 over 8 weeks), it seems sharing these workshops more may have created more value for others in the x-risk space as well, whilst keeping the same value for the ERA fellows.
More networking events. Networking and socialising with others in the X-Risk space can be difficult, and is a key element that ERA brings by bringing together 24 researchers together. We had intended to organise specific networking events with other x-risk researchers, but this broadly failed to materialise. Partially, this is due to the fellowship being over the summer, particularly August, which makes this harder. However, the networking fellows had with external people e.g. through their mentors or with speakers seemed very valuable, so more of this may be useful. Making general networking events more tailored to researchers interests may be challenging, and the model of visiting other spaces (e.g. the Trajan visit) may be more fruitful. One potentially good opportunity for networking and sharing which was limited was external people attending the research symposium.
How replicable is all of this?
It is hard to know how easy it will be to replicate the culture. Active culture forming efforts, including programming, clearly can help, however, many other factors, including simply who the fellows (and research managers) were, contributed to such a truth-seeking culture where diverse opinions are encouraged. Setting the tone for this early, as the initial workshops did, would certainly be important in the future; whether it would be a workshop, or more spread out over the first few weeks, remains to be seen.
Organising so many events was only possible due to the generosity of time of all our speakers, and the more fellowships are run, the fewer time speakers will likely have to talk. Nonetheless, many x-risk researchers seem keen to communicate with promising young researchers, and so it seems organising so many events in the future may be possible. Whilst the diversity of speakers seemed noticeably larger than previous years of ERA (then CERI), this seems to be the fact we reached out to a greater diversity of speakers from our networks. Certainly, having strong networks in different areas of x-risk made it easier to reach out to people.
Many of the events were remote, and so could be organised with essentially zero cost. As such, it may be possible to make similar quality events much more accessible to a much broader audience of people interested in existential risk. Whilst it is questionable that participants would gain the same value outside of the fellowship context where conversations and ideas were able to be fostered (although remote fellows did seem to gain a lot from events), this may be a high impact activity to continue and broaden to a much larger audience.
Credits
This post was written by Gideon Futerman and Nandini Shiralkar, part of the Existential Risk Alliance (ERA) team, a fiscally sponsored project of Rethink Priorities. We are especially grateful to Tilman Räukers, Oscar Delaney and Moritz von Knebel for their insights and constructive feedback on this post. However, it’s important to note that their feedback doesn’t imply complete endorsement of every viewpoint expressed in this post. Any inaccuracies, errors, or omissions are solely our responsibility. We encourage readers to engage critically with the content and look forward to incorporating further feedback as we refine our understanding and approach.
Appendix
Message to ERA Fellows on Slack
We just wanted to clarify a few things about ERA, the speakers we invite and our epistemic culture:
We value disagreement and diverse views, and strongly encourage truth seeking norms
We have no ‘Official ERA line’ on most issues within XRisk, and indeed likely have substantial disagreements and divergences in the ERA team over important xrisk related ideas
We acknowledge in quite intense intellectual spaces like these people can feel it is hard to think in certain ways that may go against group consensus. We want this space to allow for as truth-seeking and creative thinking as possible, so please let us know if you’re feeling this way at any point in the programme. We’re passionate that people don’t feel pressured to view XRisk a certain way. Such pressure can be felt by one person and not others within the environment, so let us know even if you can’t put your finger on exactly why
The speakers are chosen because we think they provide useful and interesting perspectives. We do not encourage nor expect you all to agree with them on everything, and encourage those who wish to challenge speakers in Q&As. The same will apply to the reading group as well
There is no one way to do xrisk research or one provably right set of conclusions we expect you to come to. The point of the fellowship is to help you improve as xrisk researchers; thus the process is more important than the destination
As we explained at the start of the programme, some people on this programme are strongly affiliated with the Effective Altruism community, and some are not, and attitudes towards that community surely range amongst all of you as they do amongst us research managers. Speakers will also likely espouse a variety of views on the principles and community, which we are happy for them to share with you. There is no expectation that fellows take any particular position with respects to the Effective Altruism community, either supportive or critical, although we do expect everyone remains respectful of each other
Overview of all events
Event Title
Speaker
Affiliation
Week
Cause Area
Key Concepts in Existential Risk Studies
Gideon Futerman
ERA
1
Existential Risk
Agents of Doom and Deep Systems Analysis
Luke Kemp
CSER
1
Existential Risk
Governance in Existential Risk
Lalitha Sundaram
CSER
1
Existential Risk
History and Ethics of Existential Risk Studies
SJ Beard
CSER
1
Existential Risk
Theory of Change Workshop
Michael Aird (Recording)
Rethink Priorities
1
Existential Risk
Foresight and Futures methods
Luke Kemp and Lara Mani
CSER
1
Existential Risk
How to do a literature review
Lara Mani
CSER
1
Existential Risk
Methodologies in Existential Risk Studies
SJ Beard
CSER
1
Existential Risk
AI Governance (incl. AMA)
Michael Aird
Rethink Priorities
2
AI Governance
EA Funding workshop or open Q&A session
Michael Aird
Rethink Priorities
2
Research Funding
Biosecurity projects and careers AMA
Tessa Alexanian
Independent
2
Biosecurity
Writing Skills Workshop
Ashwin Acharya
Rethink Priorities
3
Research Skills
Trip to Oxford (events below)
3
Fellowship Activity
Chat with Tushant Jha
Tushant Jha
FHI
3
AI Technical
Approaches to AI Alignment
Joar Skalse
FHI
3
AI Technical
XRisk and FHI AMA
Matthew Van der Merwe
FHI
3
Existential Risk
Comprehensive AI Services and Q&A
Eric Drexler
FHI
3
AI Technical
Biosecurity Discussion
Nadia Montazeri and Shrestha Rath
EV
3
Biosecurity
Biosecurity Discussion
Gregory Lewis
Independent
3
Biosecurity
Careers Workshop/Talk
Matt Reardon
80000 Hours
4
Research Careers
Climate Change, Collapse and Cascading Risk
Constantin Arnscheidt
CSER
4
Climate Change
AI and Biosecurity
Jonas Sandbrink
UK Cabinet Office
4
AI, Biosecurity
Obsidian workshop
Ximena Barker Huesca
ERA
5
Fellowship Activity
Initial Alignment Plan (Talk+Q&A)
Buck Shlegeris
Redwood Research
5
AI
Solar Radiation Modification and Existential Risk
Aaron Tang and Gideon Futerman
CSER, ERA
5
Climate Change
Self-care workshop
Megan Nelson
6
Forecasting workshop
Nuño Sempere
Samotsvety, Independent
6
Pausing AI? The Ethics, History, Epistemology and Strategy of Technological Restraint
Matthijs Maas
CSER, Legal Priorities Project
7
AI
Engaging governments trading global nuclear catastrophic risk: a personal journey, 40 years in
Paul Ingram
CSER
7
Nuclear Risk
Is climate change ungovernable?
Paul N Edwards
SERI, Stanford University
7
Climate Change
Overcoming Adversity in Negotiations
Rose Gottemoeller
NATO
7
Nuclear Risk
DeepMind Q&A / Fireside Chat
Sébastien Krier
DeepMind
7
AI Governance/Meta
Why do we future, creating manifestos for existential risk’ workshop
We expect these results to be slightly biased by potential primacy and recency effects, but have reasonable confidence in there being some signal within the noise.
When we refer to “networking”, we mean deliberate attempts to have conversations with experts and practitioners about matters directly or indirectly related to the respective fellows’ research or interests rather than the general notion of “meeting person X at a conference to have a friendly chat” or the kind of political networking that is more common in DC and Brussels. We recognize that the term “networking” is more commonly used for these settings.
Culture and Programming Retrospective: ERA Fellowship 2023
This post is part of the “Insights from the 2023 summer research fellowship of the Existential Risk Alliance (ERA)” sequence, outlining lessons learned from running the ERA Fellowship programme in 2023. For more information, see the post introducing this sequence.
We decided to publish this post summarising lessons from the 2023 ERA fellowship because we believe there is value in transparently sharing our experiences running one of the largest existential risk summer programs. While we made an effort to survey participants and learn from their feedback, we recognise that others may identify additional strengths or weaknesses based on a high-level summary. By putting these reflections out into the community, we hope to catalyse discussion that will help us and others improve future programming aimed at building existential risk research capacity.
Executive Summary
At ERA, we tried to create a culture where fellows felt able to explore diverse x-risk ideas openly and collaboratively
Our programming included 34 speaker events and 4 discussion groups. We contained a number of speaker events in our 2 day long intro workshop and our day trip to Trajan House.
The intro workshop and the reading groups particularly helped fellows gain knowledge; the mean satisfaction with knowledge acquisition was 8.4/10.
Events exposed fellows to many new ideas and cause areas from a diverse range of perspectives, and survey responses suggest that this was a key contributor to a truth-seeking atmosphere during the Fellowship.
Social connections and professional collaboration between fellows proved very important, as many rated these components as one of the most important aspects of the programme.
Areas for improvement include organising events earlier, providing more research methodology training and networking events as well as sharing events more broadly (including with other fellowship programmes).
Introduction
During the 2023 ERA Cambridge summer research fellowship, we attempted to create a culture where participants felt able to explore a variety of different ideas related to existential risk (x-risk) - we were intent on creating a space where questioning and exploration of different approaches was encouraged, and wanted to avoid making participants feel like they had to agree with our views on x-risk, but rather to upskill into strong x-risk researchers who could develop important new ideas. Essentially, we wanted to create a space where discovery and debate was encouraged in a collaborative rather than tense way.
ERA’s Culture
In general, the culture around the office felt exceptionally friendly, with discussion between fellows of different cause areas being a common occurrence.
Discussion on a multitude of key issues around existential risk happened a lot, with a variety of fellows taking different views.
It was very clear that there wasn’t an ‘orthodoxy’ on ways to approach this, which seemed to be very useful in allowing for exploration of a variety of approaches. We took active steps to communicate this to fellows (one example of this can be found in the Appendix).
Participants seemed not only excited about exploring ideas around x-risk that seem common in EA, but also a much broader range of ideas including those that may be much less commonly discussed in EA spaces.
Participants seemed willing and interested to talk about and explore other cause areas; almost every event (including reading groups) had participants from a variety of cause areas.
Fellows’ professional connections with each other seemed really crucial. Many participants rated interactions with other fellows to be the most important aspect of the fellowship for them.
Fellows also developed strong social connections with each other away from work settings as well. This likely helped in creating a positive work environment that allowed debates, disagreements and divergences to be fostered in a friendly way, and we expect this to aid our efforts to build a sustainable and collaboratively x-risk research community even after the Fellowship ends.
Overview of Programming
During the 8 week fellowship, we ran 34 ‘speaker events’ (including talks, Ask Me Anythings and discussions with external speakers), and hosted an existential risks discussion group with an external facilitator (APOCALYPSE), facilitated 2 cause-specific discussion groups in Cambridge (AI and Nuclear) and joined an existing cause-specific discussion group (GCBR). Fellows also organised a one-off discussion group (Climate + Meta) and ran weekly reflection activities. A full list of speaker events is in the appendix.
We began the fellowship with a 2-day introductory workshop on Tuesday and Wednesday of week 1, followed by a visit from Michael Aird on Monday and Tuesday of week 2. The introductory workshop had the following talks:
Gideon Futerman (ERA): Key Concepts in Existential Risk Studies
Luke Kemp (at the time CSER): Agents of Doom
Lalitha Sundaram (CSER): Governance in Existential Risk Studies
SJ Beard (CSER): History and Ethics in Existential Risk Studies
Michael Aird—recording (Rethink Priorities): Theory of Change Workshop
Lara Mani and Luke Kemp (CSER): Futures and Foresight Methods
Lara Mani (CSER): How to do a literature review
SJ Beard (CSER): Methods in Existential Risk Studies
These were meant to be accessible for those with no experience in x-risk (as some of our fellows had never engaged explicitly with x-risk before), but also provide very useful context and content for those heavily involved in x-risk discussions. In general, they were focused on ‘big questions’ of x-risk, and attempted to give an overview of the breadth of the field.
Fellows highly valued this set of workshops, with many explicitly mentioning it in their post-fellowship feedback[1]. Those participants with little x-risk experience seemed to get the most out of it, with it seemingly bringing everyone ‘up to speed’ rather quickly. A few fellows already involved in x-risk found it slightly less useful, though most with existing experience still found it informative. The initial workshops influenced the research questions for some fellows, and, according to others, may have changed the direction of their projects if questions had not already been decided upon.
Theory of Change for the Programming
Note: This was written before the programme started.
Key objectives:
Help inform the work of the fellows during the fellowship, including research skills and other useful skills they will directly use in the fellowship
This should directly improve their research output, which is a useful x-risk end in itself. It also will make the research experience more useful and easier, meaning retention in x-risk research is likely higher
Provide fellows with skills, knowledge and concepts useful to specific cause areas and x-risk in general.
This is about training them to be good x-risk researchers, giving them a good basis for further work and awareness of the field as a whole to help with research and prioritisation
Provide fellows with networking opportunities and understanding of the social organisation of their cause area and x-risk in general
This should allow them to be better involved in the core discussions going on in the field, allowing them to be exposed to more ideas via diffusion, opportunities and connections, as well as allowing their research to in turn be communicated to these audiences
We want fellows to leave the fellowship not just with an understanding of what they researched, but a better grasp of their cause area and x-risk as a whole. This means programming should be intentionally broad to expose fellows to as wide a range of concepts as possible, whilst also in depth enough to provide them with a solid basis to explore these concepts further if they so wish. Equipping fellows with this diverse conceptual and skills toolkit is a key outcome of the fellowship that is operationalised through the programming.
The programming character should change through the fellowship. The start should focus on skills and concepts directly relevant, or could be directly relevant, to their projects, and on building strong social connections between the fellows. As time goes on, the focus should shift towards networking and broader concepts and skills that may be useful to the fellows in the future.
This is essentially based on what is necessary to make the fellows as useful additions to x-risk as possible: equipping them with skills and knowledge to do novel and useful research towards reducing x-risk, and a network and awareness of the field that will allow them to operationalise this into useful and impactful research.
Did the programming fulfil the Theory of Change:
Our programming contributed to different fellows’ research in different ways. In general, for more narrowly scoped projects, the impact tended to be limited, whilst broader projects tended to benefit more. Whether better focused programming could have actually assisted with the more specific projects over such a short window is unclear. Many participants noted the lack of programming for specific research skills that may have been useful specifically for their projects. This may be reflected in the lower score (mean) given for gain in research skills and competencies when compared to knowledge acquisition (7.1 compared to 8.4).
The programming was clearly very useful at helping improve fellows’ knowledge of x-risk in general, and for their cause area in particular. The first of these two aspects seemed to be stronger than the latter. The mean satisfaction of knowledge acquisition was 8.37, with the programming, particularly the reading group and initial workshops being mentioned in responses a large number of times. (Note the mean satisfaction with events was 7.7, although most negative comments were related to the last-minute organisation; indeed one fellow rated the quality of events at 9 and the organisation of them at 2).
It also somewhat contributed to understanding the social organisation of the cause area, and some commented on how useful the connections with speakers were (e.g. one person has said it has helped them consider a job at the speaker’s institution). One speaker explicitly offered attendants to reach out to them afterwards, an offer that many attendants of that event picked them up on, leading to multiple calls with the speaker that we have reason to believe were counterfactually very unlikely to happen. Nonetheless, other factors such as discussion with mentors, research managers and other fellows seemed overall more important in connecting people and giving fellows the awareness of what their cause area actually looks like than the events.
The shift towards networking[2] didn’t really occur. Fellows did visit other institutions (some attended the Cooperative AI Summer School, and there were visits to Conjecture and Trajan House) which seemed to be quite valuable. Research managers and mentors also put fellows in contact with relevant researchers, and fellows did establish connections with speakers. However, as the programme went on, fellows wanted to spend more time on their research, and because of holidays August is a hard time to organise events. Nonetheless, it is likely that networking opportunities could be improved in the future.
Generally, we feel the programming gave participants greater awareness of key concepts and ideas across x-risk in a lot of depth, helping skill them up to be active participants in the x-risk field.
How can programming contribute to culture and overall aims
The initial workshops (and Michael Aird’s week 2 visit) seemed to immediately signal to participants encouragement for exploration of a variety of viewpoints, and that disagreement would be encouraged, as long as it was truth-seeking (medium confidence).
This is because speakers openly presented different and disagreeing perspectives about core issues in x-risk. Some speakers were part of the EA community, some ambivalent and some critical of the EA community.
The initial workshop and the APOCALYPSE reading group encouraged participants to continually explore ‘big questions’ about x-risk, how to understand it and how to work on it (medium confidence).
There are compounding factors here, in that a number of the projects were engaging with many big questions of how to conceptualise x-risk.
The reading and discussion groups encouraged participants to learn and discuss x-risk topics with people in other cause areas (high confidence). Having ‘general’ events, and having specific events for every cause area encouraged an ‘equality’ of engagement where people from each cause area learnt things from other cause areas (weak confidence), and led to broader discussion about ‘big questions’ in XRisk (medium confidence).
In Week 3, we introduced the ERA Research Circles, cause area-specific reading and discussion groups to facilitate knowledge exchange amongst the fellows. The idea is that Fellows can meet each week to discuss research papers relevant to their cause areas, or wider existential risks research. To encourage the formation of these groups, we supported fellows by covering dinner for the evenings of such discussions. This led to the formation of 2 fellow-led discussion groups. These Research Circles were not restricted specifically to fellows from specific cause areas and helped contribute to a culture of open inquiry and collaboration. Fellows from different cause areas joined each group, exposing themselves to new perspectives and allowing for lively debate that tested assumptions on all sides.
What we ought to improve
Organisation: Clearly signal when events are (rather than just putting them in a calendar). This was the most common complaint of the event organisation.
Nonetheless, the flexibility that the last minute organising gave to speakers
Some participants desired more Q&As or Ask-Me-Anything type sessions. This may have been more useful, as it is also easier for speakers generally. However, it also may somewhat limit the range of ideas explored and that the participants are exposed to.
More programming focused on research skills and methodologies. This is something we attempted, however, we feel we did this rather unsuccessfully. Partially this is because such research skills may be specific to the project (the skills needed for a technical AI Agent Foundation Project may be very different to those needed in a project analysing tail-risk treaties). Working out how to do appropriately tailored programme to research skills and methods will be a challenge, but would potentially be very useful to certain fellows. Moreover, there does generally seem to be a lack of training that can be provided to early career researchers on what good methodologies for x- risk research looks like.
Sharing more! The initial workshops had some people from the GCR community in Latin America and from ILINA join, and some of the later programming had participants from CHERI and XLab join. The quality of these sessions seemed no worse when more people joined. Given how valuable they were, and given how many sessions ERA managed to get (35 over 8 weeks), it seems sharing these workshops more may have created more value for others in the x-risk space as well, whilst keeping the same value for the ERA fellows.
More networking events. Networking and socialising with others in the X-Risk space can be difficult, and is a key element that ERA brings by bringing together 24 researchers together. We had intended to organise specific networking events with other x-risk researchers, but this broadly failed to materialise. Partially, this is due to the fellowship being over the summer, particularly August, which makes this harder. However, the networking fellows had with external people e.g. through their mentors or with speakers seemed very valuable, so more of this may be useful. Making general networking events more tailored to researchers interests may be challenging, and the model of visiting other spaces (e.g. the Trajan visit) may be more fruitful. One potentially good opportunity for networking and sharing which was limited was external people attending the research symposium.
How replicable is all of this?
It is hard to know how easy it will be to replicate the culture. Active culture forming efforts, including programming, clearly can help, however, many other factors, including simply who the fellows (and research managers) were, contributed to such a truth-seeking culture where diverse opinions are encouraged. Setting the tone for this early, as the initial workshops did, would certainly be important in the future; whether it would be a workshop, or more spread out over the first few weeks, remains to be seen.
Organising so many events was only possible due to the generosity of time of all our speakers, and the more fellowships are run, the fewer time speakers will likely have to talk. Nonetheless, many x-risk researchers seem keen to communicate with promising young researchers, and so it seems organising so many events in the future may be possible. Whilst the diversity of speakers seemed noticeably larger than previous years of ERA (then CERI), this seems to be the fact we reached out to a greater diversity of speakers from our networks. Certainly, having strong networks in different areas of x-risk made it easier to reach out to people.
Many of the events were remote, and so could be organised with essentially zero cost. As such, it may be possible to make similar quality events much more accessible to a much broader audience of people interested in existential risk. Whilst it is questionable that participants would gain the same value outside of the fellowship context where conversations and ideas were able to be fostered (although remote fellows did seem to gain a lot from events), this may be a high impact activity to continue and broaden to a much larger audience.
Credits
This post was written by Gideon Futerman and Nandini Shiralkar, part of the Existential Risk Alliance (ERA) team, a fiscally sponsored project of Rethink Priorities. We are especially grateful to Tilman Räukers, Oscar Delaney and Moritz von Knebel for their insights and constructive feedback on this post. However, it’s important to note that their feedback doesn’t imply complete endorsement of every viewpoint expressed in this post. Any inaccuracies, errors, or omissions are solely our responsibility. We encourage readers to engage critically with the content and look forward to incorporating further feedback as we refine our understanding and approach.
Appendix
Message to ERA Fellows on Slack
We just wanted to clarify a few things about ERA, the speakers we invite and our epistemic culture:
We value disagreement and diverse views, and strongly encourage truth seeking norms
We have no ‘Official ERA line’ on most issues within XRisk, and indeed likely have substantial disagreements and divergences in the ERA team over important xrisk related ideas
We acknowledge in quite intense intellectual spaces like these people can feel it is hard to think in certain ways that may go against group consensus. We want this space to allow for as truth-seeking and creative thinking as possible, so please let us know if you’re feeling this way at any point in the programme. We’re passionate that people don’t feel pressured to view XRisk a certain way. Such pressure can be felt by one person and not others within the environment, so let us know even if you can’t put your finger on exactly why
The speakers are chosen because we think they provide useful and interesting perspectives. We do not encourage nor expect you all to agree with them on everything, and encourage those who wish to challenge speakers in Q&As. The same will apply to the reading group as well
There is no one way to do xrisk research or one provably right set of conclusions we expect you to come to. The point of the fellowship is to help you improve as xrisk researchers; thus the process is more important than the destination
As we explained at the start of the programme, some people on this programme are strongly affiliated with the Effective Altruism community, and some are not, and attitudes towards that community surely range amongst all of you as they do amongst us research managers. Speakers will also likely espouse a variety of views on the principles and community, which we are happy for them to share with you. There is no expectation that fellows take any particular position with respects to the Effective Altruism community, either supportive or critical, although we do expect everyone remains respectful of each other
Overview of all events
1
1
1
1
1
1
1
1
2
2
2
3
3
3
3
3
3
3
3
4
4
4
5
5
5
6
6
7
7
7
7
7
7
We expect these results to be slightly biased by potential primacy and recency effects, but have reasonable confidence in there being some signal within the noise.
When we refer to “networking”, we mean deliberate attempts to have conversations with experts and practitioners about matters directly or indirectly related to the respective fellows’ research or interests rather than the general notion of “meeting person X at a conference to have a friendly chat” or the kind of political networking that is more common in DC and Brussels. We recognize that the term “networking” is more commonly used for these settings.