Effective Thesis—Activities and Impact Evaluation [June 2021-June 2022]
This impact report covers the period of 1st June 2021 − 1st June 2022.
This report was written by me, with significant contributions from many of our team members (especially Adela Novotna and Sophie Kirkham).
We are grateful to Michael Aird, Rose Hadshar and Rossa O’Keeffe-O’Donovan for their helpful comments. Any errors in the report remain our own.
Summary
What we did: (more)
Our mission is to support university students to begin and progress in research careers that significantly improve the world
Before June 2021, our main activities were building profiles of impactful research directions, listing of relevant research agendas, and providing individual topic choice coaching.
We’re now continuing the same activities, and in addition, we’ve broadened our portfolio of activities significantly, to meet other needs aspiring researchers have when building their research careers. We now provide help with finding a supervisor; a newsletter with early-career research-related opportunities; a database of PhD funding sources; an online community; and the Exceptional Research Award.
In the past 12 months, we have spent approx 160,000 USD and used 2.65 FTE years of labour. We are now a team of 6 paid staff (~4 FTE), as well as 4 volunteers (~0.25 FTE).
Our impact: (more)
In the past 12 months, we have attracted 501 new unique applicants for our services (a 188% increase compared to the whole of 2020, i.e. almost tripling). Our website has been visited by 18,723 new unique visitors (81% increase compared to the previous 12 months, i.e. almost doubling) and approximately 26% of these visitors spent more than 1 minute on the website.
The application data we collect suggests we are acting primarily as an internal service for the EA community (around 60 % of our users are considerably engaged with EA at the time of application). It also suggests that we might be supporting demographic diversification of the EA community via attracting people from countries that are less represented in the EA community, more non-male people, and perhaps also people from less represented disciplines, though this is uncertain because of a high proportion of missing data. Moving forward, we hope to increase our ability to (a) attract newcomers to the EA community and (b) encourage people unfamiliar with EA to work on EA-aligned topics, regardless of whether they go on to feel engaged with EA.
This document provides info on our “high impact cases”: the people who we think we’ve most increased the expected impact of (in absolute terms—not necessarily as a fraction of the impact they would have had otherwise). We added 16 new high-impact cases this year (compared to 11 last year), but our estimate adjusted for reporting delays and response rate suggests that our actions in the period from 1st June 2021 to 1st June 2022 produced as many as 134 short-term effect high impact cases, out of which 71 will experience long term effects on their trajectory 12 or more months later. We think these high-impact cases capture most of the traceable impact we produced.
We also very likely have some untraceable impact, in the form of people who used our website content to make decisions but didn’t sign up for any of our services or contact us. We know of at least a few cases where our content influenced someone because they reached out to tell us at a later date. We haven’t tried to estimate the size of our untraceable impact.
According to feedback surveys, people find our coaching useful for career advice and expanding their network of relevant connections, but only to a lesser extent for research topic choice.
Our funding database has helped at least 4 (and possibly 7+) people to get funding from a source the person says they wouldn’t otherwise have known about or applied for funding from.
Our supervisors database helped 2 people find PhD and general supervision, which shows room for improvement. The database was also used by organisers of EA-aligned research training programmes (e.g., CHERI, CERI, SERI) to find mentors for their participants, which shows an alternative use case for the database.
More than 50% of feedback form respondents who used our Early Career Research Opportunities Newsletter (ECRON) applied for at least one opportunity we shared. As a result, some of them were accepted to selective fellowships or research jobs or received funding for their projects. 67% of respondents said that ECRON was the only source from which they heard about most of the opportunities we shared, suggesting we are filling an important niche.
People value being members of our online community and being aware of others working on similar research but are not yet making many new connections via our online community platform, showing room for improvement.
We discuss 3 ways how we could potentially have a negative impact and how we try to mitigate these risks. Specifically, we discuss the risks of giving suboptimal advice, crowding out similar initiatives, and reputational harm to the EA community.
Our plans and funding needs: (more)
We have a funding gap of about 300,000-1,366,000 USD for the next year, depending on how rapidly we scale up our activities and how many of our new project ideas we try out.
On the side of scale-up, we think it would be good to grow our outreach and content capacities.
On the side of trying out new projects, we are most excited about developing and running new research skills workshops; start supporting our mentors and experts in various ways such as helping them become better at mentoring, or arranging more networking opportunities for them; and distributing mini-grants for students who are already working on aligned research topics to help them conduct higher quality research for their theses/projects.
What we did
Our mission
Effective Thesis (ET) is an organisation that supports university students to begin and progress in research careers that significantly improve the world. We do this primarily by helping students identify important problems where further research could have a big impact.
ET aims to do this by steering students who will become researchers towards EA-aligned research directions and supporting them to develop specific research questions, as well as finding other support such as supervision or funding. The ultimate goal of ET is not to influence the focus of individual student’s theses, but whole research careers. We aim to influence career trajectories by influencing thesis topic choice, since this seems to be a crucial point on research career trajectories (see some evidence that researchers tend not to diverge much from their initial topic choice e.g. in Jia, Wang & Szymanski, 2017).
The main focus of the project is on students who are likely to become researchers anyway, not on adding more researchers to the system. The chances are that these students will become researchers within academia, rather than within EA research orgs (because of the sheer number of positions available in academia vs in EA), which carries the promise of scalability of this intervention.
Aside from our direct impact on research careers, we also think our activities will create some additional positive effects for the EA ecosystem, e.g. by allowing students to better test their fit for research jobs at EA orgs and bringing a new and more diverse set of research-minded people to EA.
Effective Thesis thus aims to create a world where there are many more researchers focusing on existential risks and longtermism, animal sentience and welfare/alt protein, global development and other priorities that people in the EA ecosystem consider important, such that “not enough knowledge/understanding” or “not knowing how to proceed in solving these problems” is a relatively smaller obstacle compared to today. The world we strive for is also a world where interest in these problems within academia (and the broader research community) has become more mainstream and carries a lot of credibility.
Our activities
In June 2021, our main activities were building more content about impactful research directions and how aspiring researchers can approach them, and providing 1:1 coaching giving students personalised help with the process of choosing their thesis/PhD topic. Since then, we have broadened our portfolio of activities significantly, to account for other needs aspiring researchers have when building their research careers. Specifically, we developed:
Help with finding a supervisor—general advice as well as a database of recommended supervisors for the impactful research directions we promote
Newsletter with early-career research-related opportunities—to help aspiring researchers get a chance to test their fit with research careers and build relevant credentials
Database of PhD funding sources—to help people find sources that specifically fund PhD students in the research directions we recommend on our website
Online community—to facilitate connections between people doing research in the research directions we recommend and provide additional support beyond coaching during the students’ research process
Exceptional Research Award—to encourage and recognize promising research by students that has the potential to significantly improve the world; to stimulate more interest in the research directions we recommend; and to create more credentials for people doing this kind of research
On the content side, we have also developed a number of pieces advising aspiring researchers on topics relevant to building their research careers.
To learn more about these activities, read this EA forum post.
We also did some experimentation with outreach and marketing and built significant capacities in this domain, as we believe there is value in experimenting with messaging to attract people with large potential to improve the world who have not been attracted by the current messaging of the EA community so far. We also believe that our services are well placed to potentially attract such people and help them have a more positive impact on the world.
Specifically, we did interviews with community builders and researchers from 26 non-English speaking countries to identify country-specific institutions and channels to connect with our target group (i.e. very talented undergrad, master and PhD students who consider research as a career). That generated a number of potential outreach channels and a database of almost 500 contacts to relevant people, institutions and programmes which could help us reach our target group. We can potentially share a more detailed list of all the outreach contacts with people running sufficiently aligned projects, so if you run one of these, feel free to get in touch at david.janku@effectivethesis.org and we can discuss it. We have tested most of these contact and channels and summarized our lessons learn from this here (covering the period June 2021- January 2022). Since then, we built more capacities by hiring an online marketing professional with several years of experience in the higher education sector and a breadth of experience working with various marketing tools for our communications director.
Our organisational capacities
In June 2021, we made our first three hires for the head of coaching, content manager and second project manager roles, adding up to approx. 2 FTE combined. In July 2021 we received an EAIF grant that allowed us to professionalize and scale up our capacities to a total of approx 3 FTEs consisting of: organisational director; communications director; content manager; head of coaching; community manager and manager of our ECRON service.
Our team has a mixture of disciplines, backgrounds and interests. Several members of our team have experience pursuing research careers (often splitting their time between Effective Thesis and their own PhD or Masters studies), with some of them even being past users of Effective Thesis services. Our culture is start-up-like, friendly and collaborative, based around personal agency.
From July 1st 2021 - July 1st 2022 we have spent approx 160,000 USD, with 143,367 USD spent on salaries and 15,545 USD spent on other expenses (travelling, office equipment budget, web services, lawyer and accounting services, etc…).[1] During that year-long period, we had on average 2.65 FTE of paid staff. Both spending and staff capacity gradually increased toward the end of that period.
We have also created a formal non-profit legal entity for our organisation.
Our impact
The Effective Thesis evaluation was conducted in June and July 2022, and it combined three different methodological approaches: survey, interviews and desk search.
For reference, here is the impact report of Effective Thesis covering summer 2020 to summer 2021.
The main objective of the evaluation was to evaluate the impact we had on people who were in touch with us over the last 12 months. It was designed to explore the impact of our whole organisation, as well as its individual services that were established in the past year. Those services are the Coaching, Funding database, Supervisors database, Early-Career Opportunities Newsletter and Community.
The evaluation survey (see here) was distributed to people who used any of the Effective Thesis services between 1st June 2021 and 1st June 2022 (e.g. applied for a coaching session or requested access to the database of supervisors). Students who applied for coaching before 1st June 2021 but completed their thesis after 1st June 2021 were also included. The total sample was 549 unique people and the response rate was around 18 % (for more detail see the ‘Response rate’ section below).
The following first summarizes who we attracted to apply to one or more of our services, and then what impact our services had on them.
Applications (who we attracted)
In this section, we present some descriptives of the applications that we attracted. This aim is to help you understand what kinds of people are interested in our services and who we reached with our outreach activities. This section does not describe how our services influenced those people, it only describes the kinds of people who were attracted enough to apply.
Summary of Applications section
Profiles of applicants look similar across all of our services—we are sourcing applicants in a higher proportion from countries that are less represented in the EA community, mostly from people who are already fairly engaged with EA ideas (with about one-third of people who are not engaged or only mildly engaged), mostly from Master’s and PhD students, and people who applied have often heard about Effective Thesis from EA-related sources. There is likely a larger proportion of non-male applicants compared to the EA community and the distribution across study disciplines is fairly broad, encompassing many disciplines.
While for some services this seems aligned with their theory of change (for example, we appreciate that we are attracting more Master’s and PhD students for coaching and the supervisors database compared to undergraduates; and more EA engaged people for the supervisors database), for other services, our hopes were a bit different (for example, we hoped to attract a larger percentage of people with no or mild engagement with EA to apply for our coaching or our Early Career Research Opportunities Newsletter). So far, the structure of our applications suggests we are acting as an internal service for the EA community, and supporting diversification of the EA community in some sense (e.g. via attracting people from less traditional countries, more non-male people, and perhaps also people from less traditional disciplines). We have yet to see whether we can (a) become an outreach tool and attract relative newcomers to the EA community, or (b) encourage people to work on EA-aligned topics whether or not they go on to feel engaged with EA.
Coaching
Between 1st June 2021 and 1st June 2022, 294 students applied for a coaching session at Effective Thesis. This is a 69% increase compared to the previous year (174 applicants between June 2020 and June 2021).
These applications came primarily from: people in countries already well-represented in the EA ecosystem/community; people who are already fairly engaged with EA ideas (with one-third of people who are not at all engaged with EA or only mildly engaged); masters and PhD students; and people who heard about Effective Thesis from EA-related sources.
Country of origin or study:
This data shows that about 60% of applicants are coming from the EU, UK or US which is similar with the 2020 applicants (see Impact Evaluation 2021). Compared to the distribution of people currently involved in Effective Altruism (Rethink Priorities, 2020), we are sourcing significantly fewer people from the US, slightly more from the UK, and generally significantly more from outside the top 5 countries that are most represented in the EA community (US, UK, Germany, Australia, Canada; we are sourcing 45% from these countries, whereas in the EA community 69% of people come from these countries).
Gender (based on choice of pronouns)
We do not track gender on our application forms, but we ask about pronouns, which is a good proxy.
The percentage of our coaching students with pronouns she/her and they/them is similar to the people involved in EA Community, but the percentage of people with the pronouns he/him is noticeably lower (Rethink Priorities, 2020). Anyway, due to 30% of missing data, we can expect all those values to be slightly higher, and as a result, more people with pronouns she/her and they/them compared to the EA community, and fewer people with pronouns he/him.
Research interests at time of application
Applicants are asked which cause areas they are currently interested in pursuing research in when they apply for coaching. Note that applicants could pick more than one area, in which case each choice has been counted. To better understand our classification of cause areas, see this chart with links to our profiles of each cause area.
Study discipline
The coaching applicants have a background in almost 50 different disciplines. The most frequent are displayed in the figure below.
Compared to the 2020 applicants, there is a noticeable increase in students with a computer science background (6% of 2020 applicants), and a decrease in philosophy students (10% of 2020 applicants).
Prior engagement with EA:
*Explanation of levels of engagement:
Considerable engagement: have engaged extensively with EA content and community, spending significant time on that
Mild engagement: have engaged with a few articles, videos, podcasts, discussions or events on effective altruism, but overall spent only a few hours on that
No engagement: have not heard of effective altruism before, or even if I did, I do not engage with related content or ideas at all
This suggests that our coaching mostly attracts already considerably engaged people. About one-third of applicants have no or mild prior engagement with EA. This distribution is similar to the engagement distribution of 2020 applicants, with slightly fewer applications in the “no engagement” category and slightly more in the “mild engagement” category (see Impact Evaluation 2021).
Degree level
(Note: This is about what degree respondents are currently doing, not the highest degree they’ve already achieved.)
This suggests we are attracting mostly Master’s and PhD students, which is positive according to our theory of change: we believe that influencing people’s PhD topics have the most path-dependent effects for the rest of their careers, and many people choose their PhD topic based on their Master’s thesis. This data shows we attracted 23% more applications from Master’s or PhD students or students thinking about starting graduate studies compared to the 2020 applicants.
Source of learning about Effective Thesis
(Note: people could select only one option)
This suggests that EA channels (such as EA conferences, 80000 hours, local/student groups and EA Forum) are the source of at least half our applications, suggesting we have not yet managed to attract applications from other sources in high numbers.
Students of Top 100 Universities
24% of the applicants study at one of the top 100 universities based on the 2022 QS World University Ranking. This is a slight decrease compared to the previous year when 29% of the applicants studied at one of the top 100 universities.
Funding database
The funding database is openly accessible on the Effective Thesis website, thus we do not collect any information about people who visit it. At the moment it contains 53 funding opportunities. Overall, the listed opportunities received 251 clicks, and the most clicked opportunities were: Early-career funding for individuals interested in improving the long-term future, EA funds—Long-term future fund, Center for Reducing Suffering, Emergent Ventures | Mercatus Center.
Supervisors database
Effective Thesis created the supervisor’s database in October 2021. Access to this database is automatically provided for students who attend a coaching session with an Effective Thesis coach or can be requested independently via an application form on our website. Out of 87 people who requested the access independently, 59 were given the access and 28 were denied or asked for more information regarding their request (which happened primarily when people didn’t seem to be interested in the research directions we build the supervisors database around).
These applications were coming mostly from people who are already fairly engaged with EA ideas, mostly from masters and PhD students (see the figure below) and about half of the time from people who have heard about Effective Thesis from EA-related sources. This is consistent with the expected target group of the database—people usually look for PhD supervisors while studying for a Master’s or starting a PhD and the database should serve as a resource for people who are already fairly involved in EA and who already know what impactful research topic they want to focus on, with the lack of a supervisor being the limiting factor preventing them from working on that topic.
Early career research opportunities newsletter (ECRON)
The Early Career Research Opportunities Newsletter was also created in October 2021 and the current number of subscribers is 293 people.
These applications came mostly from people who are already fairly engaged with EA ideas (see the figure below), and about half of the people who applied have heard about Effective Thesis from EA-related sources. Since research opportunities should be interesting for a wide group of students even outside of EA, there is room for improvement in our outreach toward them.
Community
The online community platform was released in March 2022 and currently has 168 active members. Access to the community is provided automatically for the students who attend a coaching session with an Effective Thesis coach, or can be requested independently via application form on our website. Out of 20 applications from people who requested access independently, 18 were granted and 2 were declined access. Below we analyze information only about people who applied independently (i.e. people who applied only for access to the community, but not for coaching).
These applications came mainly from countries already well-represented in EA, with slightly more people coming from the US and Europe, and with around half of the applicants using he/him pronouns. The majority of the applicants were already fairly engaged with EA ideas, were mostly studying for Master’s degrees and had often heard about Effective Thesis from EA-related sources.
Website
The Effective Thesis website from 1st June 2021 − 1st June 2022 had 18,723 new visitors. This compared to 10,294 visitors in the previous year, from 1st June 2020 − 1st June 2021 is an 81% increase in new users. The average number of visitors to the website increased from 858 per month in the previous period to 1,560 in the current period. Though it is difficult to evaluate the impact from website traffic alone, it is a good indicator for “top-of-the-funnel” metrics to indicate awareness and reach. Approximately 26 % of these visitors spent more than 1 minute on the website.
Results of Effective Thesis support
In this section, you can learn more about the effects of our services on people who used them.
High impact cases
Based on our previous analyses, we formed an assumption that most of our impact will come from a small number of cases (Note we refer to these as both ‘high impact cases’ and ‘cases’ for brevity in the rest of this report). Summary list of our high impact cases should thus be a good proxy for most of the impact we produce. We have nominated high-impact cases based on
how much we influenced them (e.g. them reporting the high counterfactual value of our services in the survey) and
how promising their trajectory seems to us (that is some combination of how impactful the topic they work on is and how good of a fit they seem to be for a research career).
We sourced these cases from:
our evaluation survey (see below), focusing on people who have reported getting large counterfactual value from Effective Thesis;
people we met via personal connections who told us Effective Thesis significantly influenced them;
looking at the lists of prize-winners, fellows for various fellowships, people who received various sources of funding we shared, and cross-checking these names with the names of people in our database (i.e. people who signed up for any of our services) and asking them how much influence Effective Thesis had on their decisions and actions.
We categorise all high-impact cases in 2 buckets:
direct, short-term effects: describes the immediate effects we had on the given person, e.g. helping them switch their research focus, connecting them with some research opportunity, mentor, or funding;
long-term effects: describes the effects on the given person’s trajectory 9 or more months after their graduation, which usually means 12 or more months from the time we helped them and from the time the direct, short term effects of our help occurred. In this bucket, we only list the stories of people whose trajectory still seems to be influenced by our actions 12 or more months ago.
We don’t list people who changed their trajectories significantly such that there is likely no impact our initial help could have had on them (e.g. people who refocused on different causes, people who switched to non-research careers not relevant to the original cause they were interested in, people who report very small effects of our help on their career decisions in our long term feedback form, etc…).
We think it’s likely that the people we influenced the most were particularly likely to stay in contact in some way (and perhaps fill in our evaluation form), though there is still some uncertainty about the extent to which our high-impact cases capture the majority of the impact we had. Specifically, one alternative scenario in which we don’t track the majority of impact via these cases is if many people used our website content to make decisions but never signed up for any of our services or got in touch with us in other ways. We anecdotally know that this happened at least a few times (e.g. Karolina Sarek found out about Charity Science Health via our website and later became Co-founder and Director of Programs at Charity Entrepreneurship).
For the full list of the high-impact cases that Effective Thesis has produced, see this document.
Some examples:
Student 1
This student applied to ET while doing a Masters in Mathematical Logic & Computer Science. Signs of promise include a scholarship awarded to the top 0.5% of German students nationally, a Don Lavoie Fellowship and currently 4.0 GPA in their Master. Although they were interested in AI alignment before coaching, they described ET as having really ‘encouraged [them] to pursue AI alignment as they felt there was a very supportive community out there.’ They also applied to the PIBBSS fellowship and got in after it was suggested to them by their coach. They rated the value of Effective Thesis as 10⁄10 for providing useful career advice with an 80% chance they would not receive such value elsewhere, 8⁄10 for expanding their network with a 90% chance they wouldn’t otherwise have found this value, and 10⁄10 at helping them find new research opportunities, with an 80% chance they wouldn’t otherwise have felt this value.
Jin Qian
Jin connected with Effective Thesis wanting feedback on her PhD proposal. Her coach also helped her prepare for her PhD interview. She received a scholarship for her PhD, which is centred on wildlife practices in China’s food systems, from her university. She additionally received funding from the Animal Welfare Fund for an independent project on wild animal advocacy in China, after being told about the fund by her coach. She said: ‘My coach has helped with pointing to resources, scoping, and reviewing my PhD pitch presentation. On the networking and funding side he helped a lot, too. Though I’m also in the EA circle, I probably would not have heard of the Animal Welfare Fund if it was not because of him’ and ‘the proposal helped me get a scholarship at our graduate school.’ She’s still studying for her PhD.
Koen Schoenmakers
Koen found the GPI research agenda on the Effective Thesis website and decided to write his undergraduate philosophy thesis on a topic in the agenda. He then did a Master’s in Health and Social Psychology, and during this became a graduate affiliate with Megan Palmer’s lab at Stanford. He said, ‘I read about Megan Palmer’s lab after exploring the website of ET. When an internship opportunity came up at that lab, I already had some familiarity with their lab and the topics they study thanks to ET. The topic I ended up working on was only indirectly biosecurity related…ET likely helped me score some extra points in the selection interviews, but from an impact perspective it is hard to evaluate, as I don’t know which candidate I beat out. Later I did end up working on biosecurity [with the same Stanford researchers].’ Koen also received funding from Open Philanthropy for early career biosecurity research. He’s now switched research areas, having accepted a PhD offer from Harvard working on moral psychology with Joshua Greene, although he thinks getting the research position working with Stanford really helped with getting into Harvard. Koen is also sharing open biosecurity research questions raised by the work he did at Stanford, so possibly his biosecurity research will have an additional impact through this.
Year-to-year progress and cost-effectiveness
Summary
We added 36% more high-impact cases than last year (15 in total). Taking into account the cases we cannot see e.g. due to incomplete response rate or due to delays in reporting, we estimate our actions over the last 12 months produced up to 135 short-term high-impact cases, which we expect to translate into 71 cases showing long-term effects. Putting together this estimate and our spending, we further estimate that we generated one short-term case for approx 1,2k USD, and one long-term case for approx 2,25k USD.
Number of new high-impact cases added this year
Since June 2021 (when we did our last collection of high-impact cases), we have added 15 new cases—an increase of 36% compared to the number of cases we added last year (n=11). We tried to stick with the same entry bar for the new cases as we used the previous year, by having multiple team members check the previous list and the current list and marking whether there were any new cases which seemed worse than any cases from the previous year. We put a bit more effort into searching for high-impact cases this year by also explicitly collecting stories from people who have not filled out our form but who we met via our networks and who had some notable public success (like receiving an award or funding).
| Category | Number of high-impact cases |
| Total number of new short-term high-impact cases | 11 |
| Total number of new long-term high-impact cases | 19 |
| Long-term cases that are based on short-term cases from previous reports | 15 of 31 previous short-term high-impact cases |
| Total number of new cases (both short term and long term) not included in previous reports | 15 |
How do high-impact cases added this year relate to the impact of our actions in the past 12 months
It is also good to keep in mind that the number of new cases we added does not neatly track the impact produced in the past 12 months, because of delays in reporting. For example, in the table below, you can see we have recorded 11 short-term cases in the past 12 months. For a high-impact case to be reported, the person must fill in our evaluation form, which is sent to them at the time of their thesis submission deadline. 67% of people we have helped in the past 12 months have their thesis submission deadline after June 1st 2022, and thus we will only be able to record them as high-impact cases in our impact report next year. Similarly, as you can see in the table below, 2 of the cases we report this year applied for our services and were helped more than 12 months ago, which says nothing about the impact of our actions in the past 12 months. Therefore, the more correct interpretation of the number of cases reported above is that it tracks part of the impact produced in the past 12 months and some part of the impact produced in the previous years.high-impact
| Application year | Number of short-term high-impact cases in 2022 report |
| June 2021 - June 2022 | 9 |
| June 2020 - June 2021 | 1 |
| June 2019 - June 2020 | 1 |
Out of the 33% of people who had a chance to fill out our feedback form (and thus be considered as high-impact cases), only 21,6 % filled out the form (here we are using the ‘engaged response rate’ that should account for the fact that people who were more engaged were more likely to respond). This suggests that to get a real number of potential high-impact cases, we should take the number of the high-impact cases from the past 12 months (9), take only those that we learnt about via our feedback form (still 9), and adjust it for the response rate by multiplying it by 5 (as the response rate was approx ⅕), and adjust it for the people who haven’t had a chance to fill out the form yet by further multiplying by 3 (since only ⅓ of people had a chance to fill out the form because they had thesis submission deadline in the past 12 months). That puts the estimated number of (short-term) high-impact cases at 135. This is perhaps still a bit inflated estimate since even though we based it on engaged response rate rather than total response rate, it is still likely that people who we helped are more likely to respond. Anyway, it was hard for us to get a sense of how much lower it should be because of this consideration (and we don’t think this really undermines the bigger picture we’re presenting here), so we leave it up to the reader’s judgement.
The illustration below shows the estimation of the total number of high-impact cases from students who applied for coaching in the last 12 months.
Conversion of short-term effects into long-term effects
How many short-term high-impact cases will turn into long-term high-impact cases (i.e. how many short-term effects have repercussions in the long term)? Out of 36 short-term effects (31 short-term cases from previous reports + 5 newly found long-term cases which were not reported as short-term cases previously), in total 19 turned into long-term cases. This is approx a 53 % conversion rate.
If we apply this conversion rate to the estimated number of short-term cases our actions produced in the past 12 months (135), we get an estimate of 71 long-term cases created.
Hint at cost-effectiveness
This can also give us some hint about cost-effectiveness. With our spending of 160k USD over the past 12 months, this means we have generated one short-term case for approx 1,2k USD, and one long-term case for approx 2,25k USD. Since “long-term case” often means influencing a person’s career trajectory 12 and more months down the line, we might assume we have generated a couple of extra years of research on the pressing global problem by each long-term case. This could be ideally compared to the value of such research work estimated by various EA leaders, or in a much more conservative manner, it could be compared to the value of such research work expressed by the salaries of these researchers. We won’t get into further details here, but overall the value seems to be likely order(s) of magnitude greater than the cost.
These figures are also assuming that all our impact comes from selected high-impact cases, which is probably not true—our real impact will be a bit higher since we can’t trace it all. For example, we don’t have any way to trace the impact on people who used our website content to make decisions but never signed up for any of our services or got in touch with us in other ways. We anecdotally know that this happened at least a few times. Thus the costs per long-term case are a bit lower, but it is hard to estimate how much lower. These costs per long-term case figures should thus be interpreted as the upper bound.
Descriptives of the high-impact cases
Types of value generated by ET for our new high-impact cases
Of the 15 new high-impact cases (each case study could be mentioned in several categories):
8 mentioned value generated by ET in terms of changing their research direction/helping them figure out the research topic
8 mentioned value generated by ET in terms of connecting them with internships/jobs and other research opportunities
4 mentioned value generated by ET in terms of connecting them with funding
Evaluation of specific services
Response rate
We have collected information about the effects of our services by sending out a survey (see here) to all people who have used our services in the past 12 months, or people who used our services before that but had their thesis submission deadline in the past 12 months.
The survey was sent to 549 students and we received 97 responses, which represents approx 18 % response rate. To reduce the potential response bias suggesting that more engaged people (or people who we helped more significantly) are more likely to respond, we have calculated the ‘engaged response rate’, which came at 21.6 %.
We calculated the engaged response rates in the following ways:
Coaching: We included coaching applicants if they were connected with at least one expert (which applies to 24 % of students we coached), so the engaged response rate for coaching is the percentage of these applicants who also filled in feedback forms.
ECRON: We considered newsletter subscribers engaged if they opened at least one link in the newsletter between February and June 2022 (as this is the earliest we have data, and it seems reasonable to say that people who haven’t clicked links in the newsletter in several months are no longer engaged). This applies to 54 % of all ECRON users. The engaged response rate for the newsletter is the percentage of these people who filled in feedback forms.
Community: We considered community members engaged if they made at least one post on the community platform (which applies to 36 % of community members), so the engaged response rate is the percentage of these people who also filled in feedback forms.
Supervisors: We didn’t directly track how much different applicants engaged with the supervisor database.
Below is the response rate for each of our services:
*This number combines students who applied for coaching over the last 12 months (294), and students who had thesis submission deadlines over the last 12 months (159). Those two groups partially overlap (78).
Coaching
72 students (19% of the coaching applicants) filled in our evaluation form. Out of those respondents, 32 (9% of the coaching applicants) had already submitted their thesis.
The following figure shows the distribution of students based on the perceived help Effective Thesis provided in the following areas.
Overall, the results suggest that coaching provides the highest value for students by expanding their network of people to talk to and giving them useful career advice, which is consistent with the design of the coaching process. Surprisingly, students seemed to experience these types of effects more than effects related to refining a research topic or choosing a new topic, which is also an important part of the coaching process. Support with finding a supervisor or funding is valued by students rather occasionally, which is okay since Effective Thesis offers different services to help with those specific issues, but still a bit surprising since these are touched upon in coaching sessions frequently.
The following figure shows the distribution of students that reported that Effective Thesis was very helpful based on the counterfactual value of Effective Thesis (thus how likely they are to gain the same value some other way in case Effective Thesis would not exists).
Funding database
8 respondents applied to some of the funding opportunities listed on the Effective Thesis website.
The results of the applications are displayed in the table below.
The probability of learning about the given funding opportunity elsewhere in case our database would not exist is displayed in the figure below. The colours represent the results of the funding application displayed in the table above.
Note: Data from one respondent is missing.
That suggests that among our respondents, the funding database did not provide much counterfactual value since most applicants feel like they would learn about those opportunities even if the Effective Thesis did not exist. Anyway, it is important to note that the sample we surveyed is not representative of people who have used the database, since we provided access to the database publicly and did not collect any applications for access, so we ultimately have very limited information about who ended up using the database.
We have also reached out to funding sources that received the most clicks through our database (which is our only measure of usage of the database). We have learnt that at least 7 people who applied to these sources of funding learnt about them from Effective Thesis website, and 3 of them received funding (one from Early-career funding for individuals interested in improving long-term future’ and two from Long-term future fund).
Supervisors database
25 students that used the supervisors’ database (6% of the applicants) filled up the evaluation form and 2 of them managed to find a supervisor/mentor in the database. One was a PhD supervision, the second was general non-thesis-related mentorship.
Overall, respondents find the database moderately useful.
The following figure shows how helpful students who used the supervisors database judged Effective Thesis to be in the following areas.
The following figure focuses only on students who used the supervisors database and reported Effective Thesis was “very helpful (scores 7-10)”. It shows how much of the value they received they judged to be counterfactual (i.e. how likely they were to gain the same value some other way if Effective Thesis did not exist).
The results suggest a low counterfactual impact of Effective Thesis on help with finding the thesis/PhD supervisor which is striking since this is the purpose of the database. It also seems that the highest value Effective Thesis provided for respondents who signed up for the supervisors database was giving them useful career advice and expanding the network of people to talk to. There is a causal mechanism for how access to our database could have caused such effects—as part of giving people access, we also send them advice on how to choose a supervisor and how to communicate with them which could be perceived as career advice. Similarly, getting access to the database and seeing all the contacts to potential supervisors can be perceived as “expanding one’s network of relevant people to talk to”. However, these effects could also be caused by these respondents being signed up for some other of our services and getting these types of value from them, rather than from the supervisors database.
We have also shared access to our database with organisers of many research training programmes and fellowships, such as CERI, CHERI and SERI, who then used it for finding mentors for their applicants. They report that it helped them find between 5-30 % of the mentors that they needed.
Early Career Research Opportunities Newsletter (ECRON)
47 students (16% of the subscribers) filled in the evaluation form. About 50% of those respondents did make use of some of the opportunities we listed in the newsletter and as a result, for example, got accepted to the SERI or CHERI fellowship, received Open Philanthropy Early Career Funding for their master’s program or other types of funding, secured a job at the Global Priorities Institute or discovered a research group where they will complete their thesis.
Overall, the respondents who subscribed to the newsletter stated that Effective Thesis helped them with finding new research opportunities that they would not have found elsewhere, which is a desirable output.
This is relatively surprising as we expected people would hear about many of these opportunities from other newsletters. However, it justifies the existence of this newsletter and suggests that our choice of focus (specifically early-career research opportunities) might be filling an important niche.
The respondents find the newsletter useful. The reasons why respondents find the newsletter useful (scores between 6 and 10) were that it keeps them informed about opportunities in their focus area and saves time with job searching, it is well-organized, is sent at a good frequency, is well-targeted and provides a good selection of opportunities with an easy overview. The reasons why respondents did not find the newsletter useful (scores between 1 and 5) were that they didn’t find opportunities interesting enough to apply or don’t see themselves as a good fit for the opportunities.
Community
37 students (22% of the community members) filled in the evaluation form. Those respondents find the community slightly useful.
Respondents reported that the highest counterfactual value of our community was receiving useful career advice and expanding their networks of people to talk to, which is consistent with the objectives of the community.
On the other hand, the majority of the respondents did not make any new connections through the community platform, which suggests that they value being members of the community and being aware of others even without direct contact with them (see the figure below). This also reflects room for improvement in how our community works.
The reasons respondents found the community useful (scored between 6 and 10) were because it made them feel less lonely, gave them inspiration, because they are excited about the idea and concept and because they had great interactions with others on the platform. The reasons why respondents did not find the community useful (scored between 1 and 5) were that they didn’t engage much with the community, they think there are not enough people and activity or that the community did not provide any valuable exchange.
The respondents moderately agreed that the Effective Thesis community keeps them motivated to make their research more impactful and makes them feel less alone with their topic and desire to do impactful research.
How we could potentially have a negative impact
For a taxonomy of potential accidental harms, see this 80000 Hours post. Below, we have highlighted three ways of potentially creating a negative impact that seem most likely to us.
Giving suboptimal advice
We try to prevent this by inviting external expert feedback on each of our content pieces (especially research direction profiles). If you think that some of our content is misleading or could be improved in any way, please do reach out to our Content manager Sophie (sophie.kirkham@effectivethesis.org). This is always very appreciated!
We also vet the domain experts that we connect our coaching students with in various ways, to ensure that students will get good advice.
Our Coaching, Content, and Outreach managers also have a list of domain advisors that they try to keep in regular touch with.
Going forward we also plan to invest more in our staff education so each staff member has the capacity to keep up with evolving research landscapes.
We also plan to explicitly measure the potential negative effects we might have had via directly asking about that in our next evaluation form.
Crowding out similar initiatives
We are actively trying to engage with people who are running similar programs or think about starting to run something similar. We will also be publishing a standalone post specifying what we are doing and where we think additional initiatives might be most useful.
Reputational harm
We try to prevent potential risks to EA community by refraining from using explicit EA labelling and rather focusing attention on specific research directions. We offer opportunities to get more involved with the EA community here and there via most of our services, but this is treated as strictly optional and framed as an additional resource that students can make use of and where they can potentially meet other people interested in the similar research directions, not as an umbrella under which students operate from the beginning. This should prevent risks of diluting the EA brand or harming its credibility by low quality or otherwise harmful work.
Our plans and funding needs
We plan to do a similar evaluation using the same methods every 12 months, and some limited version of that every 6 months for internal purposes.
Next plans
Keep developing all the existing services
We think that there is a significant value to be gained by keeping improving and scaling up our existing activities and services.
Outreach—we think Effective Thesis can attract talented (especially research-minded) individuals who have not yet heard about EA thanks to its value proposition (“help with figuring out a thesis topic”), and thus create an independent pipeline of people excited to make progress on the research directions/cause areas we promote. We think our messaging and communications strategy could offset some of the limitations of the existing EA community communications and offer a viable alternative for people who would not resonate with broader EA messaging or were previously discouraged by it for some reason.
To unlock the outreach potential, we have recently hired an experienced communications director and can scale up to hire 2 more FTEs focused on marketing/communications, or alternatively provide our communications director with an increased budget.
Content—many people we meet refer to the Effective Thesis website and its content as a great and unique resource they benefited from and direct other people to. We think there is a lot of value in keeping developing our content and making Effective Thesis a go-to place for thinking about impact of research as well as actions to take to make your research more impactful.
We could achieve our current content plans faster if we had 2 more FTEs focused on it (either in a form of permanent team members or in the form of one-off collaborators)
Coaching—if we manage to scale up our outreach it’s likely we will need to hire 1-2 FTEs for coaching. We are also considering testing out more intensive and hands-on coaching models.
Community—we are planning to test out new engagement strategies for online community building to improve the experiences on our existing community platform. It’s likely we will not need any additional FTEs for that.
Early Career Research Opportunities Newsletter—it’s likely we will not need any additional FTEs for this.
Exceptional Research Award—it’s likely we will not need any additional FTEs for this.
Trying out new projects
We are also considering:
Start running research skills workshops—we are now testing this with participants of CERI/SERI/CHERI Summer Research Fellowships and the members of our own Effective Thesis community. The topics we would like to create workshops for are mostly focused on how to make research more impactful rather than how to make research more efficient. We selected them based on the suggestions of researchers, Summer Research Fellowship organizers and students’ feedback as well as our experience. These topics cover: Reasoning Transparency (already developed and being tested); Asking decision-relevant questions & Theory of Change for research (partly developed by Michael Aird); Orienting yourself in a new field/literature reviews; Developing interesting research questions of appropriate scope. These workshops would be online or in-person, depending on the context.
In the future, we envision running these workshops not only for our community but also for various research fellowships (such as CERI/CHERI/SERI, etc..) as well as research EA organizations. We might also consider promoting this as an independent product (some kind of “fellowship/bootcamp”, perhaps in-person, selective and prestigious-looking) since we think this might attract a different audience than our current services (especially people not yet familiar with EA ideas or people who got a bad initial impression of the EA community).
We have already started developing this and it is likely to happen in some form. If you would like to help out with that (or know someone who could be a good fit), please reach out. If we get extra funding, we can scale this up and realize this in a more ambitious form.
Create an internship programme, where we would pair up the existing supervisors/experts we have in our database with selected students and have them collaborate on some short-term (3-6 months) project with a specific output. We could also arrange funding for either the mentor, the student or both, and support the student throughout the process by coaching, teaching research skills, etc… We could also facilitate regular meetings of all students in this programme to help them create connections with their peers and give them a sense of community. This would perhaps effectively make it a research fellowship similar to the one that is run by SERI/CERI/CHERI, but we would likely be focused on a broader range of research directions, not just existential risks. “Having more research internships” also came up in our survey as the most frequent answer to “What research opportunities would you like to see more of”, so we assume there is still a demand for some additional programmes.
This is currently in the phase of deliberating, we haven’t done any active steps towards this and are unsure whether to put priority on this compared to the other plans mentioned here.
Supporting our mentors and experts, helping them become better at mentoring and arranging some networking opportunities for them—basically start focusing on our database of potential mentors (contains >250 researchers) and have them become beneficiaries of our work.
This is currently in the phase of deliberating, we haven’t done any active steps towards this, though we think we are likely to realize this in some form. If you would like to help out with that (or know someone who could be a good fit), please reach out.
Mini-grants for students who are already working on aligned research topics to help them conduct higher quality research for their theses/projects (e.g. reimbursing expenses for participant rewards or software, not reimbursing their own time) - this could potentially raise the profile of the research directions we recommend and provide additional incentive to focus on them (similarly to what we expect from our Exceptional Research Award).
This is currently in the phase of deliberating, we haven’t done any active steps towards this (aside from previously arranging with Elicit, AI research assistant tool, to give access to all our coachees). We are unsure whether we will end up doing this.
Creating and administering ‘hamming questions’ survey to researchers active in our recommended research directions (i.e. asking them ‘what are the 5 most important problems/research questions in your area from the perspective of improving the world’, ‘what research questions do you think others should work on/should receive more research attention,’ etc..) and collate and publish the results. This would hopefully give us some further concrete recommendations for the research we should support and direct aspiring researchers to. Moreover, this could also support the perception that “Effective Thesis is a go-to place when it comes to thinking about impactful research and learning about which problems are worth focusing on”, which in turn would raise the profile of the other research directions we promote.
This is currently in the phase of deliberating, we haven’t done any active steps towards this and are unsure whether to put priority on this compared to the other plans mentioned here.
Funding needs
To keep our services running while following our current trajectory (which means continuing to develop our existing services at the current pace and running the research skills workshops), we are likely to spend 316,200 USD per year (266,200 USD salaries, 30,000 other expenses, 20.000 marketing budget) and 4 FTE of capacities in labour (we already have these capacities being deployed currently by our paid staff).
If we were to maximally scale up all our existing activities and services, our budget needs would rise to approx 650,000 USD and 9 FTEs.
If we were to start working on all of the additional projects proposed above but not scale up our existing activities, our budget would rise to 416,000 − 716,000 USD and capacities to 5 − 6 FTE (with the 1-2 needed to be hired). Together with maximally scaling the existing activities, it would come to 1,066,000 − 1,366,000 USD and 10-11 FTEs in capacities.
We are happy to accept partial contributions to our budget or grants only for specific activities we plan to do. Please reach out to david.janku@effectivethesis.org if you’d be interested in funding us.
- ^
This figure is relatively low compared to our current spending, since for the first 3 months, we operated on a fairly constrained budget, with low salaries and no benefits. Since we received new funding, aside from hiring new people, we have also invested into improving the working conditions (higher salaries, offering some benefits) to create a competitive working environment and attract better talent.
Thanks for the update, really cool to see how fast you’re growing! Cheers to another great year!
Thanks!