Give me feedback! :)
Co-Director at ML Alignment & Theory Scholars Program (2022-current)
Co-Founder & Executive Board Member at London Initiative for Safe AI (2023-current)
Manifund Regrantor (2023-current)
Ph.D. in Physics from the University of Queensland (2017-2022)
Group organizer at Effective Altruism UQ (2018-2021)
Copying over the Facebook comments I just made.
Response to Kat, intended as a devil’s advocate stance:
As Tyler said, funders can already query other funders regarding projects they think might have been rejected. I think the unilateralist’s curse argument holds if the funding platform has at least one risk-averse big spender. I’m particularly scared about random entrepreneurs with typical entrepreneurial risk tolerance entering this space and throwing money at projects without concern for downsides, not about e.g. Open Phil + LTFF + Longview accessing a central database (though such a database should be administered by those orgs, probably).
I’m very open to hearing solutions to the risk of a miscommunication-induced unilateralist’s curse. I think a better solution than a centralized funding database would be a centralized query database, where any risk-averse funders can submit a request for information to a trusted third party, who knows every proposal that was rejected by all parties and can connect the prospective funders with the funders who rejected the proposal for more information. This reduces the chances that potentially risk-tolerant funders get pinged with every grant proposal but increases the chances that risk-averse funders request information that might help them reject too-risky proposals. I know it’s complicated, but it seems like a much better mechanism design if one is risk-averse.
It seems pretty unlikely that small projects will get noticed or critiqued on the EA Forum, but low-quality small projects might be bad en masse. Future Fund gave a lot of money to projects and people that got low visibility but might have contributed to “mass movement building concerns” around “diluted epistemics”, “counterfactually driving the wheel of AI hype and progress,” and “burning bridges for effective outreach.”
Open-sourcing funding analysis is a trade-off between false positives and false negatives for downside risk. Currently, I’m much more convinced that Open Phil and other funders are catching the large downside projects than an open source model would avoid these. False alarms seem safer than downside risk to me too, but this might be because I have a particularly low opinion of entrepreneurial risk tolerance and feel particularly concerned about “doing movement building right” (happy to discuss MATS’ role in this, btw).
A few key background claims:
AI safety is hard to do right, even for the experts. Doing it wrong but looking successful at it just makes AI products more marketable but doesn’t avert AGI tail-risks (the scary ones).
The market doesn’t solve AI risk by default and probably makes it worse, even if composed of (ineffectively) altruistic entrepreneurs. Silicon Valley’s optimism bias can be antithetical to a “security mindset.” “Deploy MVP + iterate” fails if we have to get it right first on the first real try. Market forces cannot distinguish between AI “saints” and “sycophants” unaided.
Big AI x-risk funders are generally anchoring on the “sign-value” of impact rather than “penny-pinching” when rejecting projects. Projects might sometimes get submaximal funding because the risk/benefit ratio increases with scale.
Speaking on behalf of MATS, we offered support to the following AI governance/strategy mentors in Summer 2023: Alex Gray, Daniel Kokotajlo, Jack Clark, Jesse Clifton, Lennart Heim, Richard Ngo, and Yonadav Shavit. Of these people, only Daniel and Jesse decided to be included in our program. After reviewing the applicant pool, Jesse took on three scholars and Daniel took on zero.
MATS has the following features that might be worth considering:
Empowerment: Emphasis on empowering scholars to develop as future “research leads” (think accelerated PhD-style program rather than a traditional internship), including research strategy workshops, significant opportunities for scholar project ownership (though the extent of this varies between mentors), and a 4-month extension program;
Diversity: Emphasis on a broad portfolio of AI safety research agendas and perspectives with a large, diverse cohort (50-60) and comprehensive seminar program;
Support: Dedicated and experienced scholar support + research coach/manager staff and infrastructure;
Network: Large and supportive alumni network that regularly sparks research collaborations and AI safety start-ups (e.g., Apollo, Leap Labs, Timaeus, Cadenza, CAIP);
Experience: Have run successful research cohorts with 30, 58, 60 scholars, plus three extension programs with about half as many participants.
I wrote a short blog post a little while ago on preventing low back pain with exercise. I think your problem area report might have missed several important meta-analyses on low back pain. In particular, Huang et al., 2018 and Shiri, Coggon and Hassani, 2017 seem to supersede Steffens et al., 2017, and Lin et al., 2018 seems broader and more recent than NICE, 2016. I think your assessment of the quality of evidence in favour of exercise interventions for low back pain might reasonably update with respect to these references.
Application deadlines have been extended to May 22! Feel free message me or Victor if you have any questions.
Hi Viktoria! I’m sorry; we dropped the ball on emailing all the applicants who had previously submitted. In hindsight, this was an obvious first thing to do. We did post the extension on the EA Forum and LessWrong posts, and a host of Slack workplaces and Facebook groups, but we should also have sent that email.
Open again for 20k people at $50 each.
Buck Shlegeris, Ethan Perez, Evan Hubinger, and Owain Evans are mentoring in both programs. The links show their MATS projects, “personal fit” for applicants, and (where applicable) applicant selection questions, designed to mimic the research experience.
Astra seems like an obviously better choice for applicants principally interested in:
AI governance: MATS has no AI governance mentors in the Winter 2023-24 Program, whereas Astra has Daniel Kokotajlo, Richard Ngo, and associated staff at ARC Evals and Open Phil;
Worldview investigations: Astra has Ajeya Cotra, Tom Davidson, and Lukas Finnvedan, whereas MATS has no Open Phil mentors;
ARC Evals: While both programs feature mentors working on evals, only Astra is working with ARC Evals;
AI ethics: Astra is working with Rob Long.
Regarding Magnus’ post, which you linked, I partly wrote this article as a response. The evidence base for preventing low back pain with exercise seems much greater than that for adjusting posture, stretching and using ergonomic furniture, which his post also recommends. I wanted to emphasise the importance of exercise as the primary intervention.
Regarding global health, the Happier Lives Institute produced a report on pain in 2020 that identified low back pain as a focus area, but missed my references [3-5]. In particular, my reference [4] seems to supersede their reference (Steffens et al., 2017) and my reference [3] seems broader and more recent than their reference (NICE, 2016). I think their recommendations might reasonably update with respect to these references. https://forum.effectivealtruism.org/posts/3MiMJYwYrhPmNcNBM/problem-area-report-pain-1
Cheers, Nick! We decided to change the title to “retrospective” based on this and some LessWrong comments.
I think that one’s level of risk aversion in grantmaking should depend on the upside and the downside risk of grantees’ action space. I see a potentially high upside to AI safety standards or compute governance projects that are specific, achievable, and verifiable and are rigorously determined by AI safety and policy experts. I see a potentially high downside to low-context and high-bandwidth efforts to slow down AI development that are unspecific, unachievable, or unverifiable and generate controversy or opposition that could negatively affect later, better efforts.
One might say, “If the default is pretty bad, surely there are more ways to improve the world than harm it, and we should fund a broad swathe of projects!” I think that the current projects to determine specific, achievable, and verifiable safety standards and compute governance levers are actually on track to be quite good, and we have a lot to lose through high-bandwith, low-context campaigns.
We broadened our advertising approach for the Summer 2023 Cohort, including a Twitter post and a shout-out on Rob Miles’ YouTube and TikTok channels. We expected some lowering of average applicant quality as a result but have yet to see a massive influx of applicants from these sources. We additionally focused more on targeted advertising to AI safety student groups, given their recent growth. We will publish updated applicant statistics after our applications close.
In addition to applicant selection and curriculum elements, our Scholar Support staff, introduced in the Winter 2022-23 Cohort, supplement the mentorship experience by providing 1-1 research strategy and unblocking support for scholars. This program feature aims to:
Supplement and augment mentorship with 1-1 debugging, planning, and unblocking;
Allow air-gapping of evaluation and support, improving scholar outcomes by resolving issues they would not take to their mentor;
Solve scholars’ problems, giving more time for research.
Defining “good alignment research” is very complicated and merits a post of its own (or two, if you also include the theories of change that MATS endorses). We are currently developing scholar research ability through curriculum elements focused on breadth, depth, and epistemology (the “T-model of research”):
Breadth-first search (literature reviews, building a “toolbox” of knowledge, noticing gaps);
Depth-first search (forming testable hypotheses, project-specific skills, executing research, recursing appropriately, using checkpoints);
Epistemology (identifying threat models, backchaining to local search, applying builder/breaker methodology, babble and prune, “infinite-compute/time” style problem decompositions, etc.).
Our Alumni Spotlight includes an incomplete list of projects we highlight. Many more past scholar projects seem promising to us but have yet to meet our criteria for inclusion here. Watch this space.
Since Summer 2022, MATS has explicitly been trying to parallelize the field of AI safety as much as is prudent, given the available mentorship and scholarly talent. In longer-timeline worlds, more careful serial research seems prudent, as growing the field rapidly is a risk for the reasons outlined in the above article. We believe that MATS’ goals have grown more important from the perspective of timelines shortening (though MATS management has not updated on timelines much as they were already fairly short in our estimation).
MATS would love to support senior research talent interested in transitioning into AI safety! Our scholars generally comprise 10% Postdocs, and we would like this number to rise. Currently, our advertising strategy is contingent on the AI safety community adequately targeting these populations (which seems false) and might change for future cohorts.
I think the distinction you make is real. In the language of this post, I consider the first type of evaluation you mention as a form of “support.” Whether someone desires comfort or criticism, they might prefer this to be decoupled from evaluation that might disadvantage them.
We hope to hold another cohort starting in Nov. However, applying for the summer cohort might be good practice, and if the mentor is willing, you could just defer to winter!
I’m not advocating a stock HR department with my comment. I used “HR” as a shorthand for “community health agent who is focused on support over evaluation.” This is why I didn’t refer to HR departments in my post. Corporate HR seems flawed in obvious ways, though I think it’s probably usually better than nothing, at least for tail risks.
Yeah, I think that EA is far better at encouraging and supporting disclosure to evaluators than, for example, private industry. I also think EAs are more likely to genuinely report their failures (and I take pride in doing this myself, to the extent I’m able). However, I feel that there is still room for more support in the EA community that is decoupled from evaluation, for individuals that might benefit from this.
Thanks Joseph! Adding to this, our ideal applicant has:
an understanding of the AI alignment research landscape equivalent to having completed the AGI Safety Fundamentals course;
previous experience with technical research (e.g. ML, CS, maths, physics, neuroscience, etc.), ideally at a postgraduate level;
strong motivation to pursue a career in AI alignment research, particularly to reduce global catastrophic risk.
MATS alumni have gone on to publish safety research (LW posts here), join alignment research teams (including at Anthropic and MIRI), and found alignment research organizations (including a MIRI team, Leap Labs, and Apollo Research). Our alumni spotlight is here.
In my management role, I have to juggle these responsibilities. I think a HR department should generally exist, even if management is really fair and only wants the best for the world, we promise (not bad faith, just humour).
Thanks for publishing this, Arb! I have some thoughts, mostly pertaining to MATS:
MATS believes a large part of our impact comes via accelerating researchers who might still enter AI safety, but would otherwise take significantly longer to spin up as competent researchers, rather than converting people into AIS researchers. MATS highly recommends that applicants have already completed AI Safety Fundamentals and most of our applicants come from personal recommendations or AISF alumni (though we are considering better targeted advertising to professional engineers and established academics). Here is a simplified model of the AI safety technical research pipeline as we see it.
Why do we emphasize acceleration over conversion? Because we think that producing a researcher takes a long time (with a high drop-out rate), often requires apprenticeship (including illegible knowledge transfer) with a scarce group of mentors (with high barrier to entry), and benefits substantially from factors such as community support and curriculum. Additionally, MATS’ acceptance rate is ~15% and many rejected applicants are very proficient researchers or engineers, including some with AI safety research experience, who can’t find better options (e.g., independent research is worse for them). MATS scholars with prior AI safety research experience generally believe the program was significantly better than their counterfactual options, or was critical for finding collaborators or co-founders (alumni impact analysis forthcoming). So, the appropriate counterfactual for MATS and similar programs seems to be, “Junior researchers apply for funding and move to a research hub, hoping that a mentor responds to their emails, while orgs still struggle to scale even with extra cash.”
The “push vs. pull” model seems to neglect that e.g. many MATS scholars had highly paid roles in industry (or de facto offers given their qualifications) and chose to accept stipends at $30-50/h because working on AI safety is intrinsically a “pull” for a subset of talent and there were no better options. Additionally, MATS stipends are basically equivalent to LTFF funding; scholars are effectively self-employed as independent researchers, albeit with mentorship, operations, research management, and community support. Also, 63% of past MATS scholars have applied for funding immediately post-program as independent researchers for 4+ months as part of our extension program (many others go back to finish their PhDs or are hired) and 85% of those have been funded. I would guess that the median MATS scholar is slightly above the level of the median LTFF grantee from 2022 in terms of research impact, particularly given the boost they give to a mentor’s research.
Comparing the cost of funding marginal good independent researchers ($80k/year) to the cost of producing a good new researcher ($40k) seems like a false equivalence if you can’t have one without the other. I believe the most taut constraint on producing more AIS researchers is generally training/mentorship, not money. Even wizard software engineers generally need an on-ramp for a field as pre-paradigmatic and illegible as AI safety. If all MATS’ money instead went to the LTFF to support further independent researchers, I believe that substantially less impact would be generated. Many LTFF-funded researchers have enrolled in MATS! Caveat: you could probably hire e.g. Terry Tao for some amount of money, but this would likely be very large. Side note: independent researchers are likely cheaper than scholars in managed research programs or employees at AIS orgs because the latter two have overhead costs that benefit researcher output.
Some of the researchers who passed through AISC later did MATS. Similarly, several researchers who did MLAB or REMIX later did MATS. It’s often hard to appropriately attribute Shapley value to elements of the pipeline, so I recommend assessing orgs addressing different components of the pipeline by how well they achieve their role, and distributing funds between elements of the pipeline based on how much each is constraining the flow of new talent to later sections (anchored by elasticity to funding). For example, I believe that MATS and AISC should be assessed by their effectiveness (including cost, speedup, and mentor time) at converting “informed talent” (i.e., understands the scope of the problem) into “empowered talent” (i.e., can iterate on solutions and attract funding/get hired). This said, MATS aims to improve our advertising towards established academics and software engineers, which might bypass the pipeline in the diagram above. Side note: I believe that converting “unknown talent” into “informed talent” is generally much cheaper than converting “informed talent” into “empowered talent.”
Several MATS mentors (e.g., Neel Nanda) credit the program for helping them develop as research leads. Similarly, several MATS alumni have credited AISC (and SPAR) for helping them develop as research leads, similar to the way some Postdocs or PhDs take on supervisory roles on the way to Professorship. I believe the “carrying capacity” of the AI safety research field is largely bottlenecked on good research leads (i.e., who can scope and lead useful AIS research projects), especially given how many competent software engineers are flooding into AIS. It seems a mistake not to account for this source of impact in this review.