[meta: this comment is written in more argumentative way than what is my actual position (where I have more uncertainty); it seems more useful to state the disagreement than describe the uncertainties]
If I understand the model for how the AI safety field should grow, implicitly advocated for by this text, correctly, it seems to me the model is possibly wrong/harmful.
As I understand it, the model proposed is something like
people should earn enough money to create themselves comfortable runway
they should study hard in isolation
?
do research
(?) upload good papers on arxive or at least write impressive posts on LW
(?) than, they get noticed and start talking to other people in the field
This seems strange
I. They way how this path into the field is “filtering people” is a set of filters ca: “ability to get enough money to create themselves runway” AND “really really strong motivation to work on safety” AND “ability to work for long period of time in isolation” AND “ability to turn insight into papers”.
This seems to be filtering on somewhat arbitrary set of criteria. Likely dropping talented people, who for example
would have to pay high opportunity costs by working on “creating financial runway”
get depressed working in isolation on x-risk problems
are at least initially internally more motivated by interesting research problems than existential risk worries
(and many more!)
II. Doing research in this field “without talking to people” is probably quite hard. It is improving, but many important ideas and considerations are implicit/not shared publicly.
III. It encourages people to do some sort of moral sacrifice (opposite of moral hazard: you take the risk, mostly others benefit). It would be at least fair to point it out. Also it seems relatively easy for the community to decrease the personal risks, but this way of thinking does not lead people to actually do it.
IV. The model seems to suggest people should learn and start doing research in a very different way to other intellectual enterprises like physics or machine learning. Why it should be the case?
Ah. So I’m not sure I can represent Critch here off-the-cuff, but my interpretation of this post is a bit different than what you’ve laid out here.
This is not a proposal for how the field overall should grow. There should be infrastructural efforts made to onboard people via mentorship, things like AI Safety Camp, things like MIRI Fellows, etc.
This post is an on-the-margin recommendation to some subset of people. I think there were a few intents here:
1. If you’re basic plan is to donate, consider trying to become useful for direct work instead. Getting useful on direct work probably requires at least some chunk of time for thinking and understanding the problem, and some chunk of time for learning new skills.
2. The “take time off to think” thing isn’t meant to be “do solo work” (like writing papers) It’s more specifically for learning about the AI Alignment problem and landscape. From there, maybe the thing you do is write papers (solo or at an org), or maybe it’s apply for a managerial or ops position at an org, or maybe it’s founding a new project.
3. I think (personal opinion, although I expect Critch to agree), that when it comes to learning skills there are probably better ways to go about it than “just study independently.” (Note the sub-sections on taking advantage of being in school). This will vary from person to person.
4. Not really covered in the post, but I personally think there’s a “mentorship bottleneck”. It’s obviously better to have mentors and companions, and the field should try to flesh that out. The filter for people who can work at least somewhat independently and figure things out for themselves is a filter of necessity, not an ideal situation.
3. I think Critch was specifically trying to fill some particular-gaps on the margin, which is “people who can be trusted to flesh out the middle-tier hierarchy”, who can be entrusted to launch and run new projects competently without needing to be constantly doublechecked. This is necessary to grow the field for people who do still need mentorship or guidance. (My read from recent 80k posts is that the field is still somewhat “management bottlenecked”)
[meta: this comment is written in more argumentative way than what is my actual position (where I have more uncertainty); it seems more useful to state the disagreement than describe the uncertainties]
If I understand the model for how the AI safety field should grow, implicitly advocated for by this text, correctly, it seems to me the model is possibly wrong/harmful.
As I understand it, the model proposed is something like
people should earn enough money to create themselves comfortable runway
they should study hard in isolation
?
do research
(?) upload good papers on arxive or at least write impressive posts on LW
(?) than, they get noticed and start talking to other people in the field
This seems strange
I. They way how this path into the field is “filtering people” is a set of filters ca: “ability to get enough money to create themselves runway” AND “really really strong motivation to work on safety” AND “ability to work for long period of time in isolation” AND “ability to turn insight into papers”.
This seems to be filtering on somewhat arbitrary set of criteria. Likely dropping talented people, who for example
would have to pay high opportunity costs by working on “creating financial runway”
get depressed working in isolation on x-risk problems
are at least initially internally more motivated by interesting research problems than existential risk worries
(and many more!)
II. Doing research in this field “without talking to people” is probably quite hard. It is improving, but many important ideas and considerations are implicit/not shared publicly.
III. It encourages people to do some sort of moral sacrifice (opposite of moral hazard: you take the risk, mostly others benefit). It would be at least fair to point it out. Also it seems relatively easy for the community to decrease the personal risks, but this way of thinking does not lead people to actually do it.
IV. The model seems to suggest people should learn and start doing research in a very different way to other intellectual enterprises like physics or machine learning. Why it should be the case?
Ah. So I’m not sure I can represent Critch here off-the-cuff, but my interpretation of this post is a bit different than what you’ve laid out here.
This is not a proposal for how the field overall should grow. There should be infrastructural efforts made to onboard people via mentorship, things like AI Safety Camp, things like MIRI Fellows, etc.
This post is an on-the-margin recommendation to some subset of people. I think there were a few intents here:
1. If you’re basic plan is to donate, consider trying to become useful for direct work instead. Getting useful on direct work probably requires at least some chunk of time for thinking and understanding the problem, and some chunk of time for learning new skills.
2. The “take time off to think” thing isn’t meant to be “do solo work” (like writing papers) It’s more specifically for learning about the AI Alignment problem and landscape. From there, maybe the thing you do is write papers (solo or at an org), or maybe it’s apply for a managerial or ops position at an org, or maybe it’s founding a new project.
3. I think (personal opinion, although I expect Critch to agree), that when it comes to learning skills there are probably better ways to go about it than “just study independently.” (Note the sub-sections on taking advantage of being in school). This will vary from person to person.
4. Not really covered in the post, but I personally think there’s a “mentorship bottleneck”. It’s obviously better to have mentors and companions, and the field should try to flesh that out. The filter for people who can work at least somewhat independently and figure things out for themselves is a filter of necessity, not an ideal situation.
3. I think Critch was specifically trying to fill some particular-gaps on the margin, which is “people who can be trusted to flesh out the middle-tier hierarchy”, who can be entrusted to launch and run new projects competently without needing to be constantly doublechecked. This is necessary to grow the field for people who do still need mentorship or guidance. (My read from recent 80k posts is that the field is still somewhat “management bottlenecked”)