Do AI safety talent programmes work? Nobody has run the study that would tell us.
Headline finding: I audited 17 AI Safety Talent programmes. Zero of 17 have published any comparison group, rejected-applicant follow-up, matched control or randomisation. Not one. Every programme that mentions a counterfactual does it by asking participants to self-report.
Background
At least $70 million[1] has gone into AI safety talent programmes so far, but we still can’t really say much about their impact. Notably, Kairos has just raised $50M on a self-described “shallow retrospective” that looked at “around 80 people”. I decided to do a deep dive into how AI Safety Talent programmes self-evaluate their impact.
Disclaimer, these programmes are small, fast, and generally run by people with no evaluation training and no slack. The oldest one was only established in 2018 (AI Safety Camp, as far as I can tell). Most last less than 6 months. When timelines are genuinely short and urgent, then time spent evaluating is time that could be spent building. I am not criticising any individual programme.
Having said that, there are retrospective evaluation designs that could be run cheaply without slowing anything down. And a field that funds talent pipelines at this scale without knowing whether they work is not moving fast, it is moving blind.
A quick word on who I am: I am an evaluation expert, but I am new to AI safety. I have no stake in any of the programmes below and I am not an alumnus of any of them.
The audit
I audited 17 AI Safety Talent programmes: MATS, ARENA, LASR Labs, PIBBSS, SPAR, Pathfinder, Global Challenges Project, Kairos (org level), BlueDot Impact, AI Safety Camp, Apart Research, Talos Fellowship, Astra (Constellation), Anthropic Fellows, MARS (CAISH), ERA Cambridge, Horizon Fellowship. I used AI to initially search and extract findings, then checked every figure against the live page. Full findings here[2].
| Out of 17 programmes: | |
| Publish any comparison group, control or rejected-applicant follow-up | 0 |
| Publish a response rate for the survey behind their outcome claims | 2 |
| Publish a fully loaded cost per outcome | 1 |
| Pre-specified their outcome metrics before a cohort ran | 0 |
| Publish methodological caveats alongside their headline figure | 2 |
(Because it bears repeating) Zero of 17 have published any comparison group, rejected-applicant follow-up, matched control or randomisation. Not one. Every programme that mentions a counterfactual does it by asking participants to self-report. And every one of them rejects a large majority of applicants, scores them numerically, and holds their contact details, so each is sitting on a ready-made near-miss comparison group it has never used.
Only two of the 17 publish a response rate for the survey behind their outcome claims. Everyone else with a survey publishes a percentage without telling you how many people it came from.
Rigour appears to run inversely to funding. AI Safety Camp operates on $60,000 to $300,000 a year, has never held a major-funder grant, and is the only programme here to have commissioned an external assessment and published it in full (including the estimate least flattering to itself). BlueDot Impact has raised $35 million and publishes “25% of our graduates land impactful roles within six months” with no definition of impactful roles, no denominator and no method. This appeared to be a common pattern: the organisations with the most money shared the fewest details.
The strongest numbers are presented in the weakest documents. LASR Labs’s “90% of alumni have gone on to work in AI safety/security” appears only in recruitment posts, while Talos’s “70% of our alumni (58/83)” appears only in a donation appeal. Neither organisation has published a retrospective or an impact report at all. The programmes that do publish reports (like MATS and ARENA), report more modest figures alongside limitations.
Multiple organisations contradict themselves across their comms. I won’t name names here, but figures on how many fellows get a job offer can differ between blogs and job postings, or placement rates can differ between accomplishments and recruitment pages. I’m sure there’s nothing malicious here, I know from experience with a big org it’s easy for multiple figures to exist as evidence is updated! It does make it harder to do fair comparisons however.
Thoughts on the counterfactual impact of talent programmes
The counterfactual impact is what would have happened if your intervention didn’t exist. Chris Clay assembled the largest alumni dataset in the field, 600+ profiles across nine fellowships, and labelled counterfactual impact “The Golden Egg”. He asked publicly for people to work on “What proportion of people would have entered AI Safety without doing a given fellowship?” That was January 2024, and as far as I can tell nobody has.
MATS was one of the most transparent programmes in this audit. They publish a 46% response rate alumni survey[3] with “possibly” and “probably” pooled into one counterfactual category, and no comparison group. In their report is possibly the most significant finding to me: 30% of alumni say they would have participated for no stipend at all. People motivated enough to do a research fellowship unpaid are precisely the people most likely to have found their way into alignment research anyway. The programme may be measuring its selection function and reporting it as its treatment effect.
This echoes a question that frequently occurs to me about career programmes. Unless a field is significantly talent constrained/requires very specific hard-to-get skills, it’s unclear to me how valuable the counterfactual impact of that programme was. For example, say Org A is hiring a researcher, while Org B has a career programme. Org B runs a fellowship/accelerator/has 1-2-1 coaching sessions, and an alumnus gets the job at Org A. The counterfactual impact relies on both 1) would the alumnus have applied for and gotten the job without Org B?, and 2) would Org A have been able to hire another person who is equally successful in the role?
A placement is only field growth if the person would not otherwise have got there and the role would not otherwise have been filled by someone equally good. If either leg fails, the programme has moved a person around rather than adding one. Self-report surveys, regression discontinuity designs etc, attempt to answer the first question, but none of the programmes audited have published anything on the second.
In fairness, the second question is harder, because answering it means asking employers rather than participants. 80,000 Hours is the only organisation I found that goes near it, noting of its headhunting placements that “these 15 cases involved an organisation hiring a candidate who they weren’t otherwise exploring”.
Whether that matters depends on where the binding constraint sits. Weronika Żurek’s review of talent constraints suggests the AI Safety field is short of senior mentors, experienced operators, fieldbuilders and grantmakers, and she estimates 2,000 to 2,500 fellows will graduate from research programmes in 2026 alone. If that is roughly right, junior research talent is the part of the pipeline least likely to be constrained, and a placement rate from relevant talent programmes is less relevant.
To be clear, getting the right person into the right role faster is worth money, and so is the counterfactual for someone who would never have found the field at all. But those are different claims with different evidence requirements, and a placement percentage cannot tell them apart.
Jamie Bernardi, who ran AI Safety Fundamentals at BlueDot, wrote in 2024 that “people take your course because they’re already interested in the topic – so you’ll never know the counterfactual of which steps they took because of your course.” The external assessment of AI Safety Camp says the same thing about that programme’s own headline number: “AISC draws from people already interested in AIS, so researchers who appear to have a step-change in participation may not have needed AISC to break into AIS research in the first place.”
Proposed design
Regression discontinuity design (RDD) is a quasi-experimental pretest–posttest design. It uses a specific cutoff score on a continuous assignment variable to estimate the causal effect of an intervention. In the case of AI Safety Talent Programmes, we know that they are massively oversubscribed and the cutoff for being accepted is relatively arbitrary. Applicants who were close to being accepted but didn’t quite make it are probably not materially poorer researchers than those who did just make the grade.
LawrenceC collected a senior researcher’s retrospective assessment of 60 junior AI safety researchers. The mean change between initial and later ratings of promise was 0.05 on a ten-point scale with a standard deviation of 1.37, and the two strongest performers had initially been rated “somewhat above average” and “below average”. Żurek puts it more plainly: “even the best candidate assessment methods are just moderately correlated with job performance.” That doesn’t mean selection is worthless, it just means the ordering is informative in the large and close to a coin toss within a few points of the threshold (exactly the condition a discontinuity design needs).
A RDD design first needs to define what success looks like. Working full-time on AI safety at eighteen months post-decision is an obvious primary, with publication as a secondary. Both are observable from the outside and do not depend on anyone answering a survey.
The minimum viable version is retrospective. Application scores already exist in an Airtable at every one of these organisations. Pull the scores, pull (or search[4])the outcomes for admits and near-misses, done. No surveys needed, no cohort disrupted.
(Quick caveat, application scores were collected to make a selection decision, and studying career outcomes is a different purpose, so a programme shouldn’t simply repurpose them. It needs either fresh consent from past applicants or a documented legitimate-interests assessment, with pseudonymised analysis and no individual reported. Going forward, I’d recommend a checkbox at the application stage for the use of data for research purposes.)
Now, it’s worth noting that these cohorts are small. MATS has 60 scholars, LASR 12 to 16, ARENA 30. Regression discontinuity needs a decent density of applicants near the threshold, so most of these programmes would be individually underpowered. Ideally we would need a pooled analysis across programmes. Failing that, we’d need to simply compare admits against near-misses without the discontinuity machinery.
In the academic literature regression discontinuity is a standard design. E.g., see Bol, de Vaan and van de Rijt (2018, PNAS) on early-career grant thresholds (a near-miss design on a competitive fellowship, which is structurally what these programmes are), or Gonzalez-Uribe and Leatherbee (2018, RFS) on a start-up accelerator (a competitive cohort programme with numeric application scores).
In the EA field, the closest example I can find is Fodor & Tidmarsh (2024), who conducted a case-control survey of EAGx attendees with a difference-in-differences design. Note they had an admissions process available as an assignment mechanism but did not use it.
Conclusion
In a recent retrospective on their Effective Careers RFP, Coefficient Giving said “we saw a wide range of how applicants monitor their outcomes, and we’re considering an internal project to help set and share clearer standards.” Given the small cohort numbers, I agree that a joined up approach across the field, with standardised metrics, would be valuable. Failing that, it would be great to see one of the bigger organisations take this seriously. I’m happy to help design a study, or put people in touch with interested researchers.
- ^
likely a massive understatement—based on the published accounts of just four organisations.
- ^
I did check these findings, but if I got anything wrong, please let me know!
- ^
I know filling out surveys is annoying but c’mon guys, if you get given $$$ to study something cool, do the decent thing and fill out a form for your organisation after.
- ^
Less onerous than it sounds, because the outcome side is largely public. Whether someone works at a safety organisation, and whether they have published, is visible on LinkedIn and in conference proceedings. Kairos is already building Talent Commons as a “consent-first database for sharing information about program participants and job applicants”, which is most of the infrastructure this would need.
Regarding “the organizations with the most money share the fewest details”:
I believe that AI Safety Camp commissioned that independent analysis because their fundraising was largely unsuccessful. The better-funded organizations didn’t have a strong incentive to commission an independent analysis! This is a clear selection effect.
I don’t think that an independent analysis is necessarily better than the default. Coefficient Giving has conducted several highly detailed internal investigations into MATS’ impact, for example, though these were never shared publicly. I expect that cG did the same for their other large grantees, which contributed to their confidence.
Note that MATS has the most money of these programs and we try to share a lot of data. We could definitely improve, though! Let me know what data would be interesting/useful.
This has a strong flavour of something I see a lot from AI safety grantees. It’s something along the lines of ‘something about our field/org is unique and therefore the usual rules of the road don’t apply’.
For example, it is almost universally accepted that independent analyses are better than in-house analyses, for obvious reasons. I’m very surprised to see you make the opposite claim, which runs counter to widely accepted wisdom. The author points out many ways in which the current monitoring is weak, prone to bias, incomplete etc.
Importantly, Coefficient Giving is a grantmaker and UHNW advisor, not an impact evaluator. Their teams aim to assess the best grants of the available options in order to achieve a goal. This is related to but distinct from evaluating the counterfactual impact of an organisation. (As an analogy, GiveWell assesses grant opportunities including ORSZ distribution, but it doesn’t measure the impact of ORSZ distribution. It relies on specialist measurement and evaluation of ORSZ distribution.)
I think the author laid out what would be interesting and useful. It would be reassuring to have more confidence in the counterfactual value of tens of millions of dollars of grants.
To be clear, I think that an independent analysis from a capable team is obviously to be preferred over an internal or grantmaker-led analysis. A GiveWell for AI safety would be great! However, my point was that this would likely require hiring serious AI safety strategy expertise (not just general data science and impact analysis experience), which might be quite hard in the current job market, where AI safety grantmakers are really struggling to hire. Why would AI safety be exceptional in this way? Because the bulk of downstream impact of AI safety interventions is rooted in the reduction of existential risk, which is unobservable directly in the near-term, and near-term proxies might be highly misleading (plenty to discuss here). None of this means that we shouldn’t try! I endorse a “GiveWell for AI safety” and I would fund this, if not for the fact that there would be an obvious COI.
“GiveWell for AI safety” is a subject I’ve discussed a lot for Manifund and have tried to do very very very briefly (see https://manifund.substack.com/p/how-cost-effective-are-ai-safety)
I would love to do more of this ourselves (or, fund someone to do this), but as you mention the talent required for it is scarce. I wish, to start, that CG would just publish their own internal analyses about their $10m+ grants, because I expect they spend a fair amount of their own grantmaker time creating those botecs, and having them published would be so much better than nothing.
“I don’t think that an independent analysis is necessarily better than the default.” What do you mean by this exactly?
I mean, the default case (cG conducting in-depth evaluations) might be better than an “independent analysis”, particularly if cG are vastly better at impact analysis than the independent evaluators. I suspect this is probably true, as cG employ the lion’s share of top grantmakers and evaluators for AI safety and I suspect that evaluating AI safety program impact is highly contingent on very detailed technical understanding of emerging research paradigms and macrostrategy considerations. To evidence the last point, there is serious disagreement in the community about whether RLHF is good or bad for AI safety and “pessimization” effects are strong. Sure, a simple analysis of talent placements in select orgs/teams, papers published with citation count, orgs founded with funds raised is useful, but it’s very hard to see the counterfactual here.
Gotcha. Have no idea how good cG are at impact analysis—I’m not sure I’ve seen one publicised by them, I think they have decided against publicising things in general. I don’t think that AI safety people are necessarily that good at impact analysis though—very few would have trained at all in that field. Being good at impact analysis and understanding what is best for AI safety are completely different and underated things.
I get that people have different ideas about what is a “good” outcome, but CoGi at least must have a criteria if they are doing an assessment at all. The criteria could be very complex and nuanced, that’s fine—as long the criteria is clear before you start the study.
I would argue if you don’t know what is a good outcome, there’s probably no point in doing any work at all.
Valid points, all. Re. your last point, if one thinks the default case is doom, once might want to fund a lot of blue sky interventions as the upside potential is stronger than the downside risk.
Agreed here
There’s only really no point in doing any work at all if you are confident the outcome is bad.
It seems like there would be a lot of benefits to having public impact assessments.
It verifies it’s taken place, assures applicants a given talent programme is worth their time, gives talent programme orgs visibility on who’s most effective and why, can be seen by other potential donors, and would allow the impact assessment process to be scrutinised and improved upon.
Maybe cG can’t do this as making how they make donation decisions might corrupt their process. In this case the independent analysis fulfills a different purpose rather than being better or worse.
Note that the main goal of MATS is to accelerate talent into AI safety careers as fast as possible, not to be entirely counterfactual for getting people hired (though there are many such cases among our alumni, e.g. Sara Price at Anthropic). This is why we deliberately select the strongest talent in our applicant pool and give them all the resources they need to maximally succeed (e.g., mentor, cohort, compute, support staff). Presumably, if the best talent had anywhere better to go (e.g., getting hired immediately), they would do this instead! MATS doesn’t pay that much and is less prestigious than a full-time offer from an AI safety org like UK AISI, Redwood, METR, Apollo, etc.
Regarding how counterfactual we are in terms of publications or selected organizations, here are some stats from 2 years back. We are working on an updated impact analysis at the moment. Some select quotes from the linked report:
78% of respondents said their publication “possibly” or “probably” would not have happened without MATS. 10% of alumni reported that MATS accelerated publication by more than 6 months; 14% said 1-6 months.
Marius Hobbhahn: “Apollo Research would counterfactually not exist without MATS.”
Jesse Hoogland: “Something like [the developmental interpretability] agenda [at Timaeus] would have probably come into existence, but we probably accelerated it by more than a year.”
@Ryan Kidd this is great feedback you are getting and the surveys seem high quality, but just surveying participants will always be among the lowest forms of evidence. Surveys are useful (I use them a lot) but in the global health field we would want to move up the evidence ladder to get stronger evidence before rolling million plus dollar interventions out. People’s Opinions and perceptions are not necesarily true (we’ve seen this many times in global health) while good observational and experimental studies can answer questions more conclusively and discover things we don’t expect.
There’s no real counterfactual or experimentation if you are doing a survey. To get better data you could at least do a form of Case-control study (with rejected participants) or a prospective cohort study with similar groups of people. Doing an RCT with applicants seems harsh on the best participants and is a bit unfair for sure (see thread with @simon below), but maybe running one rigorous RCT could be a good idea especially if people know what they are signing up for before they applied.
Running one MATS application round where you randomised entry among the best participants would get you insanely good data not only about how much your program helps, but also the specific ways it helps. This can help you iterate your program in future.
If you’re looking at accelerated career progress you might need to follow people for 3-5 years after finishing the course to measure the effects. This length of time isn’t unusual in global health studies.
I like the author’s design too, but I have an innate dislike of retrospective studies. There’s just so much room for fiddling results and decisions that favour the org being researched, even publishing bias. Tiny data manipulations before data handovers can rig the study really easily. For example MATS could just copy paste [deleted user] paragraph into Claude Fable, feed in their applicant data and just run the study themselves right now. IF the results were favourable they could hand you the data, if not they could just ignore.
1.
I think it could be useful to get data on whether MATS is optimally doing selection via the randomized process you describe, but I don’t think it would be particularly high impact. Here’s why:
We are racing against time. We only have a few years, on median, until we achieve AGI and every MATS cohort counts. If we select marginally worse fellows now, particularly for the highest-impact mentors, this could have massive negative implications in the longterm (particularly via wasting top mentors’ time).
Mentors, not fellows, are MATS’ main client. We don’t pay mentors; they volunteer their time. Top mentors don’t want to work with anyone except the best talent. Mentors are the arbiters of fellow selection, not MATS staff; we just help them with selection. We don’t have a mechanism to force mentors to work with fellows they don’t like and if we did, this would dissatisfy our most important client. Strong mentors seem to be much rarer than strong fellows. (Note that we do select a bunch of mentor “diversity picks”, which varies each round and use fellow outcomes and feedback for subsequent rounds of mentor selections.)
We don’t seem to have a good mechanism for measuring the counterfactual on a useful timescale. MATS’ most impactful outputs, such as new nonprofit research orgs founded, placements in leading roles at AI safety orgs/teams, and field-defining research, typically take 6-18 months post-MATS. This is a long time to wait for results!
Steelmanning the case for RCTs at MATS:
Maybe we are actively causing harm by selecting fellows suboptimally and could improve fast and cheaply.
Maybe we can show mentors convincing evidence that helps them select fellows better without risking their support of MATS.
Maybe there are strong near-term proxies for impact, like a panel of experts assessing a fellow’s end-of-program research output.
Maybe we could just assemble a talent database of everyone working in AI safety and chart their journeys in a way that usefully indicates whether MATS is reliably counterfactual.
It feels like we are constantly in “wartime mentality”, given the arguably civilization-level terminal risks of near-term AGI. Maybe there are cheap interventions that would help us rapidly improve mentor/fellow selection, like the equivalent of challenge trials for COVID-19 vaccines mid-pandemic? How would you suggest we conduct challenge trials, given the wartime considerations and the fact mentors are volunteers with very selective preferences in fellows?
I think the wartime framing actually argues for the retrospective design—you wouldn’t need to randomise anything, ask a mentor to take a fellow they did not choose, or make a single cohort worse.
[deleted user] I think he’s arguing against my idea not yours. I agree 100% I think there’s no reason at all not to do the retrospective design (AI could do most of it anyway), I’m just saying its still a pretty low level of research really. I agree a big improvenet though.
(Note that challenge trials is probably the wrong framing because the risk is not to MATS participants, but to broader civilization.)
I think you misunderstood me here. I wasn’t suggesting you research your selection criteria, I was suggesting a basic Randomised controlled trial design possibliity for one intake to assess the impact of the program. An RCT woudln’t help you much (if at all) with selection criteria). The candidates who were selected and not selected would then be followed over time to see if the MATS program made a difference to their trajectories.
”every MATS cohort counts” might be true, but if the difference in fellows isn’t that big why not randomise one round at least? Not sure how big your cohorts are though. If they are too small this wouldn’t work at all.
Ah, makes sense. In the world where MATS randomizes over applicants for a typical 100+ fellow cohort and we learn after 6-18 months how impactful MATS is, what are the main upsides? My guess:
Might help understand how useful fellowship programs are vs. other AI safety interventions, like academic grants or mass-messaging (most of which are also saturated with funding at the moment).
Might help better price how cost-effective AI safety is vs. other EA cause areas.
Might help MATS staff (e.g., me) work on higher-impact interventions.
Yep that’s the one!
I don’t think it would really help vs other interventions much (unless they are also doing rigorous research, which I doubt).
Because you’ll get very specifid information about the differences, It might help you understand in which ways the program helps and doesn’t as well so you could adjust programming as well. Like you might find it helps people get more jobs but not better quality jobs than those who don’t do the program. Or you might find less people enter capabilities rather than safety etc.
I would definitely recommend pre-registering the outcome definitions and the analysis before anyone sees the data, timestamped somewhere public. Ideally the outcome coding would also be done by someone blind whether participants were part of the cohort or not.
And here is some data from 1 year ago about MATS alumni careers, based on scraping LinkedIn profiles and personal websites.
Thanks Ryan, that’s useful context on the acceleration framing, and it actually might make a threshold design easier! Time to first role is continuous and observable, so your 10% and 14% figures could be compared to the “untreated” group.
It’s also great to know that Coefficient’s has an unpublished internal analysis. I do think making it public would be helpful so programmes can learn from each other. (And so MEL nerds like me can read it!)
If this is even mostly true, from a Global health perspective at least this is borderline neglectful lack of Monitoring and Evaluation. Why do these orgs almost assume their program works, or state vague non-counterfactual data as evidence?
I find the BlueDot example almost hard to believe. BlueDot is such a flagship AI safety program, I would have thought they would have had much more rigorous impact evidence and follow up published publicly. As much to improve their program as well as demonstrate impact.
”BlueDot Impact has raised $35 million and publishes “25% of our graduates land impactful roles within six months” with no definition of impactful roles, no denominator and no method.” doesn’t tell you much at all especially without a counterfactual
Sometimes we give orgs like GiveWell a hard time for poor follow up of orgs they fund, which has seemed a bit rough and a bit of a stretch for me. They copped a bit of flack after the Evidence Action Water Dispenser fail. And that’s not even the org itself, but the funder getting a hard time This here seems like another level.
IT would be so easy to look at a counterfactual here—its not perfect as they are rejected applicants but at least you need to show that your people are doing far better than the next teir down of rejected applicants who didn’t even do the program. Surely at least you could pay the rejected people all $50 to fill in a survey and say what they are doing now and get most of the info you need. Basic counterfactual comparison like this seems like it should be the baseline, with some more rigorous studies done as well.
I know programs might hate this, but randomly assigning applicants to one program or another then comparing their progress after the course might be a great way to compare courses as well.
I hope we’re missing something here because this seems pretty neglectful on a first pass...
I think this is looking too narrowly at the talent programmes’ potential impact. To my understanding their overarching goal is getting more high-quality talent working on AI Safety faster. Counterfactually getting people hired is not the only action that can contribute to this. Off the top of my head I can think of:
Getting people hired counterfactually faster (as called out by @Ryan Kidd below)
Pre-filtering applicants for orgs with limited recruiting manpower (someone getting into a competitive programme is a decent signal, in the same way as a prestigious university or work at FAANG)
Reaching out to talent outside of the ecosystem who wouldn’t even consider AI Safety otherwise
Connecting people who then go on to build new organizations
Getting people hired into counterfactually higher-impact positions than they would otherwise go for (e.g. many people seem to default to technical research even while they have transferrable skills to work in a more senior role in ops or governance)
This is a very apposite and useful post (it correlates with a small set of survey results from Future Matters here—https://future-matters.org/updates/legibility-in-the-coming-wave-anonymous-philanthropists-barriers-to-funding-ai-governance/ )
I wrote up some of my experiences trying to assess AI safety and governance programmes here: https://fundinganthropalypse.com/p/donors-dont-check-whether-their-grants
In general, the standard of M&E is very low, and independent views like this are very helpful. As you point out, it would be relatively low lift to produce better quality data. I suspect that short timelines lead to a lot of leadership and grantmaking on trust, as people think something like ‘given the stakes, we should just throw money and time at our best bets’.
Thanks for writing it.
Good post, thank you!
Regarding “In the case of AI Safety Talent Programmes, we know that they are massively oversubscribed and the cutoff for being accepted is relatively arbitrary.” This is not always accurate, according to my work with ML4Good. There is at any given time a certain amount of people a program can reach with its standard marketing, in a given region, and a program can decide to scale their cohort sizes based on the observed ratings.
So instead of being “we take the top 10%, and this lands somewhere based on how many good applicants we got”, it can be “we observed 100 applications, found that 15 passed an absolute bar and ran the program for 15 people, but could have ran it for 25 if 25 passed the bar”. In those cases the cutoff isn’t arbitrary but based on some further criteria like expected productivity of participants etc.
I don’t think this impacts much of the rest of the analysis, because I do expect the difference between 15 and 16th candidate to be relatively small (based on my experience rating hundreds of candidates for AI Safety upskilling programs), and I do think it’d be cool to contact some number of people on the negative side of the cutoff and follow their trajectories.
Nevertheless, I’m not sure this will measure some of the important things I care for. Our 15th candidate is about as marginal as our 16th, but mostly the program is ran for the 3 best participants, as we expect there to be long tails of impact and for the top three to often have more impact than the other 12 combined (roughly, vaguely, circumstantial). This research would never compare one of top 3 candidates with someone comparable who couldn’t make it, and in fact it is mostly those that we care about accelerating. (We include 15 or more notably because we don’t know perfectly who would be in the top three and there are economies of scale, but I don’t think it’s ever been the case that someone in the bottom 10% of rated candidates that gets in surprises us to perform in 10% best during and after a program)
Just noting that MATS is hiring an Impact Analyst (deadline: Sep 21). Please apply if this sort of thing interests you!
In the spirit of the OP’s post, I propose a new post: does hiring Impact Analysts work? You might want to commission a study of impact analyst effectiveness in real-world situations that involve small and messy data with long durations to outcomes.
While an eager Impact Analyst might nerd out on all sorts of fun academic tools and theoretical proposals, in industry and at this stage of your growth, nobody should really care about measuring if your programs work and everyone should share about making them work as well as possible. Spoiler: they work. If you don’t accept that, then you have to accept that your programs certainly aren’t hurting anyone and shouldn’t be prematurely derailed because someone is skeptical about whether or how well they work.
IMHO, define a short-term performance metric and put this budget toward a relentless and talented PM who is incentivized by the metric. They should be able to set up the basic data gathering (you don’t need anything fancy), listen to customer feedback, and iterate on your product quickly.
Why don’t funders require randomisation? Eg pick the 2n best applications, randomly admit the n best, track outcomes etc.
This can be ethically a bit savage and unfair, especially for a flagship program at a criticall point in someone’s career. You could do this, but it has downsides.
From my perspective, I would be happy with them at least comparing with the applicants closest to the bar. This comparison favours the course for sure, but at least it gives you clear info. I would want them to state the study protocol and make sure they really compared vs. the best rejections though,
Yeah I guess it’s ethically questionable if you’re certain about the rank ordering.
My guess is that the relative ranking between the 2n best applicants is largely noise in any case, if you’ve got many good applicants (it’s hard to measure and predict skills etc). So as long as you make this clear up front, having a lottery component in a process that’s very noisy should be ethically fine?
I generally agree with you that there’s a lot of candidate noise in rank ordering, but many EAs seem to think there are huge differences between first and second tier candidates in candidate selection in these situatoins. I don’t think a lottery process is implausible here, but it still does feel unfair that you could be the top person selected for a course and not get in because of randomisation.
I agree with your point etc.
Just on a purely philosophical level: it could also feel less unfair in expectation, if you randomise and state the rules and your lack of conviction in ranking etc up front in the sense that it can net reduce biases and people get exactly what they agree too.
You could even issue a certificate saying “was in the selected cohort, but didn’t win the lottery” or similar to reduce some of the signalling effect.
Should the reduced search and assessment costs on the side of employers also count towards impact? The selection by talent programs creates a signal about candidate quality that could save potential employers effort and resources they would otherwise have to invest in identifying and assessing the right candidates.
In that sense, some of what looks like a selection effect from the participant side may still actually be a benefit created by the program.
Ah, very fair point! Screening is a real service to employers, and even if programmes produced no counterfactual placements at all they may still save the ecosystem a lot of search cost. It is measurable too, at least crudely, by asking hiring managers what they would have spent finding the same person.
I really like that you are digging into this! 2 quick questions:
I could imagine the impact of talent interventions could be quite long tailed (e.g. perhaps 1 person in a cohort of 60 contributes ~half of the impact), might that make your proposed RDD approach understate impact, since it really just tells you the impact on the marginal accepted applicant (rather than the super impressive applicant who was a slam dunk)?
I agree that the application scores (assume that is indeed a thing) make the RDD approach straightforward, but how would you measure impact post-program, especially for the comparison group that might be hard to follow up with? I’m imagining here that, say, an AIS bootcamp greatly increases someone’s AIS knowledge but not in a legible way that would show up in, say, their resume—developing an assessment tool that captures this impact might be difficult (but also not something I’ve given serious thought to, and perhaps you have!)
Lastly just a statistical power flag: you link to LawrenceC’s piece on variation in application assessments. That variation would suggest we’d need quite a good sample size to allow for precision in the detectable effect, which might be hard. I guess that’s just a note to make sure some sample calculation is done once you have clarity on your outcome of interest to make sure you have a path to measuring the kind of effect that would make these programs cost-effective. The worst case scenario would be a study with 100 people looking to detect an impact of, say, 0.05 SD, when you are in fact only powered to detect 0.3 SD and basically doomed to concluding the program doesn’t work, when perhaps it does.
These are intended as genuine questions to see if your ideas could work, if there is indeed a path to a good study here I’d love to see something proposed to, say, the EA Infrastructure Fund, your ideas seem valuable! (Disclaimer, I work at CEA, which contains the EAIF, but I am not involved in EAIF decisions and these are personal takes)
Thanks, great questions!
RE long tails: I’d push back slightly here, because I think the tail argument actually cuts the other way. If impact is heavily long-tailed and the tail sits with the most impressive applicants, those are also the people most likely to have found their way into an AI safety role without the fellowship. The slam-dunk applicant has high absolute impact, but low counterfactual impact. So the margin may be where the programme’s causal contribution is largest, even though expected impact per person there is smaller.
But I agree the RDD doesn’t give us an average treatment effect across the cohort. It gives us a local estimate at the threshold. This is probably the most decision-relevant number for funders, (e.g., helps answer whether to accept 70 people instead of 60), but not necessarily the best number for “should this programme exist at all”.
RE measuring outcomes: I think each programme’s theory of change has to come first here. I’ve assumed it is roughly getting talented people into AI safety roles in a timely manner. (MATS reports the share of alumni “working in AI alignment”, LASR reports 90% “gone on to work in AI safety/security”, BlueDot reports graduates landing “impactful roles”.) Public observation (e.g., LinkedIn) gets us a long way for this.
If a programme’s real theory of change is latent capability that would never show up in a career trajectory, this does become more difficult!
RE power: completely agree. This is the main reason I think we need a joined-up approach. Pooling across programmes should give us a decent sample size (power calculations still depending), though at the cost being able to isolate any individual programme.
That’s not what you should conclude from a failure to reject the null at (say) an 0.05 significance level. You would need an equivalence test to ~conclude a program (probably) does not work.
You can make meaningful statistical inference for decision making with a lower level of statistical power, especially within a Bayesian framework.
Nice! Though I guess my concern that someone would conclude the program hadn’t work, after failing to reject the null, is still there. But if the proposed study goes ahead, I’m sure your quick input could be very helpful!
The study design could explicitly include a planned equivalence test etc., to avoid anchoring this.
Big fan of equivalence testing, agreed it would be important here!
Nice argument! I do agree that the impact assessment tools from development, health, and education should be effectively applied to assess the outcomes of the AI safety talent pipelines.
But the issue in the AI safety talent field is that the supply of talent is new (fellowships), while demand for talent is predictable but not necessarily actual.
In higher education, graduates are expected to get jobs after graduation partially because employment opportunities exist before enrollment. The historical employment opportunities allows the impact assessment to have a good counterfactual comparison.
However, in AI safety, many jobs and positions are being created until the fellows graduate from the program. This makes the impact assessment and the counterfactual evaluation quite different. This is because we don’t know if it’s the AI safety talent program that created the opportunity, or if the AI safety talent program helped talent get into the next job opportunity.
But still an exciting analysis! I’d love to learn more about your research and help with ways to best assess the impact of the talent programs! I have a background in education assessment as well!
What would you recommend for evaluating general high-impact career courses?
Say, the course aims to bring new, qualified people to hard-to-fill high-impact roles (or founder paths) and improve their chances of getting hired.
The program’s target is framed as:
(1) “qualified people start pursuing HI roles who otherwise wouldn’t have, or make this transition faster”
(2) “these candidates are ready to be hired”
(1) More/faster HI transitions: we have quite weak evidence—self-reported “I wasn’t aware of these roles or founder paths / wasn’t trying these before the program.” But people have multiple sources of information and might have found these paths anyway.
(2) I’d like to look at this from the employer side.
additional qualified candidates → better/faster hiring → value for HI orgs
On better/faster hiring for hard-to-fill roles, I’d want to estimate something like:
months saved + quality uplift of the hire
- value of months saved = HR costs saved + the new hire starts to bring value earlier (“value of the role” defined by the hiring orgs).
- quality uplift of the hire = the new hire brings more value (defined by the hiring orgs, again)
Let’s say we have historical time-to-fill & hire quality data for similar roles, and we could identify the trade-offs made in the past (a less-qualified candidate was eventually hired).
But how to measure the program impact?
- We could gather data on the alumni hired for roles identified as “hard-to-fill”. If we have all qualified/near-missess applications & timestamps, we could check whether the hiring manager had another equally qualified candidate in this group at the target time. And we could check for trade-offs—there should be fewer if the program works.
That seems doable but costly (get data & gather additional info from hiring orgs).
- Or we could try to analyze all applications for “hard-to-fill” roles (not necessarily filled by the program alumni) - to check whether the program expanded the pool of qualified candidates (if program alumni were equally qualified/near-misses). So we’ll see whether the program actually adds qualified candidates to the funnel. It’s even more costly.
But it still doesn’t answer whether the program saves time / improves hire quality compared to the existing candidate pool.
I guess we could see whether we could save time-to-hire: if the program’s alumni appear in the funnel earlier than equally qualified candidates. But if they weren’t eventually hired, it seems harder to argue that this time saving was actually valuable to the hiring org.
So how should we answer this question—is the pool enlargement worth the money?
Sorry, rough thoughts, and I have no proper MEL experience.
Nice to see the causal chain out laid out!
The counterfactual impact of (1) is that individuals “do good” they otherwise wouldn’t have, but not necessarily that more good is done in the world overall. That relies on the pool being talent-constrained, so jobs would either sit empty, or be filled by demonstrably less effective candidates.
I would lean to the employer-side evaluation. Might be the costly option, but it’s more informative, and could potentially be done via asking hiring managers at the point of hire, who else was in the final set and what they would have done if this candidate had not applied. That’s close to a one-question survey.
Also worth surveying hiring managers on what are the biggest blockers—e.g., contextual knowledge? Specific skill sets? Could be useful for lighter touch evaluations of the programme down the road (i.e., are participants actually leaving with better contextual knowledge?)
I’m less sure about putting a money value on months saved early on—quite a few assumptions would be involved.
Re. “MATS has 60 scholars”, this was the size of the average cohort in 2023, but our cohorts are now 100+ fellows and we run three per year.
“A placement is only field growth if the person would not otherwise have got there and the role would not otherwise have been filled by someone equally good.”
The field is already very competitive. Are any research roles really difficult to fill, perhaps competing with high-paying jobs at a frontier lab? I’m wondering where the need really is? In talent or in organisational growth (i.e. actually creating new research streams and teams that lead to concrete impact)
I keep coming back to this question too. The whole case for spending heavily on talent pipelines rests on the field being talent constrained rather than absorption constrained, and I haven’t seen public tests of that assumption. I think it would make another super interesting study, and is very testable! Vacancy durations, re-advertisement rates, applicant to hire ratios by role type, and hiring managers asked directly why a search failed would get you a long way.
I’m tempted to look at this properly as a follow up, so if anyone reading has hiring data they would be willing to share, or strong priors in either direction, I’d love to hear from you!
I want to add one thing from the hiring side:
As I understand it, what’s proposed here relies on application scores, and those scores are generated by panels. In my experience, panels can quite easily use the same label to score different criteria without anyone noticing. Definitions can drift over time, and new panellists don’t come with the tacit knowledge predecessors built. A while ago I ran a technical hiring round which failed: when running the post-mortem, it was clear panellists had different ideas of what a strong candidate looked like, which the final score hid. Before running another round, the panel documented the criteria explicitly and agreed how much each one mattered, using pairwise comparison. The discussion turned out to be more useful than the scores.
This was in a large company, with a stable panel and written processes. I’d expect this to be a bigger issue in orgs using volunteer mentors with different research interests, and panels that change between rounds, e.g. due to availability.
There could be a quick way to test the above: if an application was scored by more than one panellist, look to see how far apart the scores were. Also, ask a recent round’s panellists to write down, independently, what they were scoring against and what mattered most, and review how well those match. Neither needs any data on what applicants went on to do.
Note, my background is hiring and managing technical people in industry, not AI safety, so YMMV.
I would like to propose a related study:
Null Hypothesis: AI Safety talent programs are harmful.
Result: Null hypothesis rejected.
Study complete.