Thanks for this comment. To me this highlights how AISC is very much not like MATS. We’re very different programs doing very different things. MATS and AISC are both AI safety upskilling programs, but we are using different resources to help different people with different aspects of their journey.
I can’t say where AISC falls in the talent pipeline model, because that’s not how the world actually work.
AISC participants have obviously heard about AI safety, since they would not have found us otherwise. But other than that, people are all over the place in where they are on their journey, and that’s ok. This is actually more a help than a hindrance for AISC projects. Some people have participate in more than one AISC. One of last years research leads are a participants in one of this years projects. This don’t mean they are moving backwards in their journey, this is them lending their expertise to a project that could use it.
So, the appropriate counterfactual for MATS and similar programs seems to be, “Junior researchers apply for funding and move to a research hub, hoping that a mentor responds to their emails, while orgs still struggle to scale even with extra cash.”
This seems correct to me for MATS, and even if I disagreed you should trust Ryan over me. However this is very much not a correct counterfactual for AISC.
If all MATS’ money instead went to the LTFF to support further independent researchers, I believe that substantially less impact would be generated.
This seems correct. I don’t know exactly the cost of MATS, but assuming the majority of the cost is stipends, then giving this money to MATS scrollas with all the MATS support seems just straight up better, even with some overhead cost for the organisers.
I’m less sure about how MATS compare to funding researchers in lower cost locations than SF Bay and London.
I believe the most taut constraint on producing more AIS researchers is generally training/mentorship, not money.
I’m not so sure about this, but if true then this is an argument for funnelling more money to both MATS and AISC and other upskilling programs.
Some of the researchers who passed through AISC later did MATS. Similarly, several researchers who did MLAB or REMIX later did MATS. It’s often hard to appropriately attribute Shapley value to elements of the pipeline, so I recommend assessing orgs addressing different components of the pipeline by how well they achieve their role, and distributing funds between elements of the pipeline based on how much each is constraining the flow of new talent to later sections (anchored by elasticity to funding). For example, I believe that MATS and AISC should be assessed by their effectiveness (including cost, speedup, and mentor time) at converting “informed talent” (i.e., understands the scope of the problem) into “empowered talent” (i.e., can iterate on solutions and attract funding/get hired).
I agree that it’s hard to attribute value when someone done more than one program. They way we asked Arb to adress this is by just asking people. This will be in their second report. I also don’t know the result of this yet.
I don’t think programs should be evaluated based on how well they achieve their role in the pipeline, since I reject this framework.
This said, MATS aims to advertise better towards established academics and software engineers, which might bypass the pipeline in the diagram above. Side note: I believe that converting “unknown talent” into “informed talent” is generally much cheaper than converting “informed talent” into “empowered talent.”
We already have some established academics and software engineers joining AISC. Being a part-time online program is very helfull for being able to include people who have jobs, but would like to try out some AI safety research on the side. This is one of several ways AISC is complementary to MATS, and not a competitor.
Several MATS mentors (e.g., Neel Nanda) credit the program for helping them develop as research leads. Similarly, several MATS alumni have credited AISC (and SPAR) for helping them develop as research leads, similar to the way some Postdocs or PhDs take on supervisory roles on the way to Professorship. I believe the “carrying capacity” of the AI safety research field is largely bottlenecked on good research leads (i.e., who can scope and lead useful AIS research projects), especially given how many competent software engineers are flooding into AIS. It seems a mistake not to account for this source of impact in this review.
Thanks. This is something I’m very proud of as an organiser. Although I was not an organiser the year Neal Nanda was a mentor, I’ve heard this type of feedback from several of the research leads from the last cohort.
This is another way AISC is not like MATS. AISC has a much lower bar for research leads than MATS has for their mentors, which has several down stream effects on how we organise our programs.
MATS has very few, well known, top talent mentors. This means that for them, the time of the mentors is a very limited resource, and everything else is organised around this constraint.
AISC has a lower bar for our research leads, which means we have many more of them, letting up run a much bigger program. This is how AISC is so scalable. On the other hand we have some research leads learning-by-doing, along with everyone else, which creates some potential problems. AISC is structured around addressing this, and it seem to be working.
Thanks for this comment. To me this highlights how AISC is very much not like MATS. We’re very different programs doing very different things. MATS and AISC are both AI safety upskilling programs, but we are using different resources to help different people with different aspects of their journey.
I can’t say where AISC falls in the talent pipeline model, because that’s not how the world actually work.
AISC participants have obviously heard about AI safety, since they would not have found us otherwise. But other than that, people are all over the place in where they are on their journey, and that’s ok. This is actually more a help than a hindrance for AISC projects. Some people have participate in more than one AISC. One of last years research leads are a participants in one of this years projects. This don’t mean they are moving backwards in their journey, this is them lending their expertise to a project that could use it.
This seems correct to me for MATS, and even if I disagreed you should trust Ryan over me. However this is very much not a correct counterfactual for AISC.
This seems correct. I don’t know exactly the cost of MATS, but assuming the majority of the cost is stipends, then giving this money to MATS scrollas with all the MATS support seems just straight up better, even with some overhead cost for the organisers.
I’m less sure about how MATS compare to funding researchers in lower cost locations than SF Bay and London.
I’m not so sure about this, but if true then this is an argument for funnelling more money to both MATS and AISC and other upskilling programs.
I agree that it’s hard to attribute value when someone done more than one program. They way we asked Arb to adress this is by just asking people. This will be in their second report. I also don’t know the result of this yet.
I don’t think programs should be evaluated based on how well they achieve their role in the pipeline, since I reject this framework.
We already have some established academics and software engineers joining AISC. Being a part-time online program is very helfull for being able to include people who have jobs, but would like to try out some AI safety research on the side. This is one of several ways AISC is complementary to MATS, and not a competitor.
Thanks. This is something I’m very proud of as an organiser. Although I was not an organiser the year Neal Nanda was a mentor, I’ve heard this type of feedback from several of the research leads from the last cohort.
This is another way AISC is not like MATS. AISC has a much lower bar for research leads than MATS has for their mentors, which has several down stream effects on how we organise our programs.
MATS has very few, well known, top talent mentors. This means that for them, the time of the mentors is a very limited resource, and everything else is organised around this constraint.
AISC has a lower bar for our research leads, which means we have many more of them, letting up run a much bigger program. This is how AISC is so scalable. On the other hand we have some research leads learning-by-doing, along with everyone else, which creates some potential problems. AISC is structured around addressing this, and it seem to be working.