Effective Thesis project: second-year update
What is Effective Thesis?
Effective Thesis is a project that directs students’ research towards EA causes by offering them EA-related topics for their final dissertations and theses. The aim is to deliver three valuable outcomes:
Changing student trajectories at a particularly crucial juncture in their lives and directing some of their research careers to more impactful paths.
Generating additional research in EA cause areas, with little cost to EA funding sources.
Making students more knowledgeable in a specific EA cause and overall more involved in EA.
The hope is that this could be potentially high-leverage, even more so than usual career coaching, as we are focusing on a particularly crucial juncture. Other potential beneficial effects of the project are diversifying the scope of people involved in EA by bringing some from non-mainstream backgrounds into the community (e.g. mechanical engineers, archaeologists, history students, etc.); making current academics (students’ supervisors and committees) more familiar with the EA perspective through students’ work; and potentially creating a gateway for academic-minded students interested in improving the world but not knowing about or not willing to identify with EA directly.
More generally, in the goal of influencing research, this project focuses more on the junior side of research career trajectories, trying to create new junior researchers who are not necessarily dependent on the EA funding. For other approaches like funded fellowships or engagement with senior academics, there are likely better-suited organisations like Global Priorities Institute or Forethought Foundation.
I founded the project in June 2017 supported by the Czech EA Association and received an EA Grant in August 2018 to test the new project design—thesis coaching. Thesis coaching is meant to provide more tailored advice to students via connecting them with experienced coaches (usually, researchers from the EA community). Coaches participate on a voluntary basis and usually have only one call with the student (devoting about 1 hour per student). The hypothesis was this would lead to a higher number of interested students and a higher quality of students’ project choice decisions than the original design, where there was a list of specific topics on the website and students were expected to match themselves with a topic. Read here to get more details on the background of the project and why we have decided to change the design.
After the student applied, was matched and connected with a coach and received the coaching, I followed up with them to have an impact interview and explore in which ways Effective Thesis influenced them. Here are the questions I used. I plan to do another follow-up after they finish their theses (approx. 6-12 months after they received the coaching) to get a better sense of the long term impact and to test some of the hypotheses mentioned above.
Results
Thesis Coaching
The advice we were able to give students seems higher quality. Students report getting reasonably high value from the project: about 7.3/10 of usefulness on average and about 50 % of counterfactual impact in their decision-making regarding their study/research/career plans. However, sources of this value are mixed: 1⁄3 of students appreciated general advice on research direction, 1⁄3 appreciated academic career-related advice and 1⁄3 appreciated guidance in the topic they came up with themselves.
The main source of counterfactual impact seems to be in getting specific topic ideas; getting sources (readings, etc..) to start with; getting the general direction to focus on within their study discipline (e.g. biosecurity for maths students); and being introduced to the main idea that they can do EA-related research in their thesis. On the other hand, the main hindrance for counterfactual impact seems to be not getting a specific-enough topic suggestion from the coach.
Promotion of the project
In contrast with the first year when the project was promoted mainly via online ads and directed outwards from the EA community, this year I focused on promoting it in the EA community via various channels: presenting it at the EAG London and EAGxNordics, informing student group leaders about the project in cooperation with LEAN; and 80,000 Hours mentioning Effective Thesis in one of their posts.
This strategy, together with the new thesis coaching design, attracted more students. There were 111 applications in the first 5 months, out of which I wasn’t able to help (or had very low-quality applications from) about 1⁄3 of people, another 1⁄3 stopped communicating during the process and about 1⁄3 used the coaching advice and had an impact interview with me in the end.
I have collected data on 5 descriptive variables from the people who applied for coaching: Whether someone considers research career as their top career option, how much involved in EA they are, which country they are from, what degree level and their university ranking. Under this outreach strategy, the project attracts mostly students whose career plan is to become researchers—roughly 2⁄3 of people who applied for coaching stated they want to take careers with a large research component. About 1⁄3 of people who applied were very involved in EA, 1⁄3 people we moderately involved, 1⁄3 people were not familiar with EA. Dropout was mainly among those who were not involved in EA and those who didn’t plan to become researchers.
People from 22 countries applied for coaching with most people applying from the UK (16 %) and the rest of Europe (41 %), US (15 %) and Australia (7 %).
Most of the students who applied studied masters degree (36 %), then undergraduate degree (24 %) and then PhD degree (19 %), with 17 % not reporting their degree levels.
28 % of students who applied were from top 100 university rated by this website. Dropout was close to even on these variables and didn’t show any significant trend.
Impact
When considering the final impact of the project, I estimated the impact of the project on each student’s decisions based on all the information I had. Impact assessed this way had two components: 1) how good is the change that the student has done; and, 2) how much of that change was counterfactually caused by Effective Thesis. When I tried to plot all student cases, I came to the conclusion that the distribution is likely log-normal, with few best cases accounting for most of the project’s impact. These best cases were students who planned to become researchers in the long-term and have substantially changed their focuses mainly thanks to Effective Thesis. I expect this trend to continue and expect that the main value that Effective Thesis will bring will be in influencing few people who will become top researchers and shift their research focus towards more important problems.
Forthcoming plans
Since the last update, I have also made some changes to the content—I’ve stopped promoting specific topics and moved towards promoting general high-impact research paths for each study discipline (see e.g. biology). Also, I have created a new page Agendas showing all research agendas in the EA community, and the page Finished showing finished as well as in-progress theses of students we advised.
As discussed in the Results section, prescribing students a specific topic doesn’t work very well in the current setting. On the other hand, most students benefited from being introduced into a general direction for their discipline and getting general academic career advice. This suggests either that coaching is not yet designed well-enough to deliver specific topics; or that the coaching is not the optimal strategy for delivering specific topics; or that offering specific enough topics is just too difficult and intractable. However, based on my impression from impact interviews, it seems that offering specific topics would still be the most important factor in raising the counterfactual impact of Effective Thesis. Therefore, it might be worth it to afford more thinking to the question of how to deliver students specific enough topics and continue experimenting with coaching design in order to learn how to do it.
My other plans are to focus on redesigning the website to make it appear more credible, trustworthy, official and prestigious; and communicate more clearly that what we are doing is important. I also plan to focus on improving the coaching process to generate even more counterfactual value, providing students with recommendations on future courses and supervisors and summarizing the advice on how to choose a thesis topic in a couple of blog posts. Recently, I have also launched an online community of students to enable them to connect with each other, solve some of their problems without my assistance and to enable me to communicate relevant job/grant offers, calls for papers and updates on thinking regarding how to start with high-impact research more readily.
In the long-term, I’d like to cooperate more closely with EA research orgs and get some further feedback on the project, since visiting Oxford was very helpful for setting direction on future development.
Regarding the promotion of the project, I would like to keep focusing on EA community members since they seem to benefit most from the coaching. Still, I’d like to devote a small portion of resources to experiment with promoting the project to non-EAs, since this has large expected value if done right.
If you have any feedback on this project, I’ll be happy to read it in the comments or at david@effectivethesis.com. If you have a good way to promote this project or have ideas on how to, please reach out as well.
Acknowledgements
I would like to thank all coaches involved who allowed this project to happen, CZEA for constant support, especially Jan Kulveit who inspired the project and came up with the idea to change design to coaching and Dan Hnyk who helps with programming, CEA for their financial support and together with EAGxNordics organisers, LEAN and 80,000 Hours for promoting the project and many community members who provided valuable feedback and ideas for future development of the project.
- Effective Thesis seeks a project manager by (25 Nov 2020 15:47 UTC; 73 points)
- Effective Thesis: updates from 2019 and call for collaborators by (18 Jun 2020 12:39 UTC; 62 points)
- Effective Thesis is hiring project manager and content manager by (13 May 2021 12:53 UTC; 30 points)
- Latest EA Updates for May 2019 by (31 May 2019 16:36 UTC; 21 points)
Thanks for writing up the project! I liked the idea of Effective Thesis when I first heard about it, but I wasn’t sure how well a small team would be able to advise a large number of students with varying backgrounds and majors. It sounds as though the results have exceeded my expectations. (Also, the “Agendas” page of the website has been a really handy resource for me.)
This is an interesting statistic. I’d have thought that nearly all the students would mostly benefit from “general advice on research direction”, since specialized EA knowledge is something Effective Thesis has that professors and career offices don’t.
1. Can you give an example of what “guidance in the topic they came up with themselves” might look like? Particularly in a case where the coach isn’t an expert on the topic?
2. Do you have any general observations of where your applications came from? I’d be interested in both the country/regional breakdown and a breakdown of applicant school rankings (e.g. “1/3 from schools in or around the top 100 of this list, most of the rest from other private American/European schools, a few from other continents”).
--
Finally, thanks for linking to your interview questions; it helps put the answers you got in context, and I like being able to see the back-end infrastructure of EA projects.
That would be my guess as well. Maybe the average advice they would get would not be that good or they would get as good advice anyway from professors and career offices but they got it first from the Effective Thesis and therefore attributed the value to Effective Thesis. However, when trying to estimate the overall counterfactual impact, I have usually rated cases which referred “career advice” or “help with topic they came up with themselves” as the main sources of value they got from Effective Thesis as being less impacted by Effective Thesis than cases which appreciated and used “general advice on research direction”.
Usually I’ve tried to match student with a coach who would be expert on the broader domain of the topic student came up with. Example could be a philosophy student who came up with 3 different topic ideas related to global priorities research. The coaching helped him prioritise between these 3 topics and choose the one which would be best in student’s situation. Another student came up with a broader topic and coaching helped him take a perspective of the topic, use a specific methodology and/or focus on a specific issue within the topic.
I’ve asked people “Where did you learn about the Effective Thesis from” in the application form but the results show no specific trend and there are too few cases to draw conclusions from. 5 most frequent channels were word of mouth (14 people), university referral (12), non EA online blog post—Thesis Whisperer (11), facebook referral (11), personal connections/individual outreach (10) and all produced candidates of similar qualities in terms of chance of getting through the funnel, desire to continue in research career and involvement in EA.
I haven’t done breakdown by country and school rankings systematically yet, but I will take a closer look and update the post soon. My impression was that people are applying from many countries around the world, even those where there is not visibly large or active EA group and I haven’t noticed any strong trend (e.g. that half of the people would be from the UK or something like that). Regarding school rankings, prevalence of candidates from top rated schools is much higher than chance which is in my opinion the result of sampling people from the EA community, however, I need to crunch the numbers to get a more specific information.
Thanks for the comment!
The update is on, in the section Promotion of the project.
Has any of the students’ research been used?
Most students haven’t finished their projects yet, but I will make another post about long-term changes in students and the impact of their theses after I make impact interview with those who are finishing their theses now (so the post will come out probably during the summer). What specifically do you mean by “being used”? Is it being published in a journal, having reference from some current researchers that it helped them in some way / showed them something new or something else?
Being referenced by a current researcher, policymaker, charity or organisation. I thought the point of Effective Thesis was for students to write something that would actually be used by someone.
Thanks for clarification! I think I have promoted it that way in the early stages, but I have deprioritised the direct impact that theses may create in the past year. The reason is that the value of long-term goals like influencing someone’s (research) career focus or building skills/learning more about the issue in order to become better able to work on it afterwards seems much higher than direct impact produced by theses. Also, many organisations don’t seem ready to take this kind of help from students and the process of soliciting the results from students is pretty long (students work on their theses from 3 months to 2 years, usually about 7 months) which makes usefulness of such help a bit lower. However, I plan to keep track of the direct impact of theses as well and will update this assumption based on the data I receive.
Thanks for writing this up, I found it very interesting, and it seems like a great project.
Quick question: did you mean to write “biology students” here below, or is there a tighter link between biosecurity and maths than I expected?
Would the team ever consider expanding to effective final projects?
What do you specifically? The Effective Thesis is focused on helping students with their final thesis projects, there were just a few cases when we helped students with their module thesis or something smaller than capstone project, but I usually deprioritise these cases in favour of final capstone projects.
I was thinking of a capstone projects that were around equivalent to a thesis