The comment below is made in a personal capacity, and is speaking about a specific part of the post, without intending to take a view on the broader picture (though I might make a broader comment later if I have time).
Thanks for writing this. I particularly appreciated this example:
A friend of mine at a different university attended the EA intro fellowship and found it lacking. He tells me that in the first session, foundational arguments were laid out, and he was encouraged to offer criticism. So he did. According to him, the organisers were grateful for the criticism, but didn’t really give him any satisfying replies. They then proceeded to build on the claims about which he remained unconvinced, without ever returning to it or making an effort to find an answer themselves.
I’m pretty worried about this. I got the impression from the rest of your post that you suspect some of the big picture problem is community builders focusing too much on what will work to get people into AI safety, but I think this particular failure mode is also a huge issue for people with that aim. The sorts of people who will hear high-level/introductory arguments and immediately be able come up with sensible responses seem like exactly the sorts of people who have high potential to make progress on alignment. I can’t imagine many more negative signals for bright, curious people than someone who’s meant to be introducing an idea not being able to adequately respond* to an objection they just thought of.
Though, to be fair, ‘hang on a sec, let me just check what my script says about that objection’ might actually be worse...
*To be clear, ‘adequately responding’ doesn’t necessarily mean ‘is so much of an expert that will just come up with a perfect response on the spot’. It’s fine to not know stuff, and it’s vital to be able to admit when you don’t. Signposting to a previous place the question has been discussed, or knowing that it will be covered later (if e.g. this comes up in a fellowship) if that is the case, both seem useful. It seems important to know enough about common questions, objections, and alternative viewpoints to be able to do this the majority of the time. If it’s genuinely something that the person running the session has never heard, this is exactly the time to demonstrate good epistemics—being willing to seriously engage, ask follow-up questions, and trying to double crux.
The comment below is made in a personal capacity, and is speaking about a specific part of the post, without intending to take a view on the broader picture (though I might make a broader comment later if I have time).
Thanks for writing this. I particularly appreciated this example:
I’m pretty worried about this. I got the impression from the rest of your post that you suspect some of the big picture problem is community builders focusing too much on what will work to get people into AI safety, but I think this particular failure mode is also a huge issue for people with that aim. The sorts of people who will hear high-level/introductory arguments and immediately be able come up with sensible responses seem like exactly the sorts of people who have high potential to make progress on alignment. I can’t imagine many more negative signals for bright, curious people than someone who’s meant to be introducing an idea not being able to adequately respond* to an objection they just thought of.
Though, to be fair, ‘hang on a sec, let me just check what my script says about that objection’ might actually be worse...
*To be clear, ‘adequately responding’ doesn’t necessarily mean ‘is so much of an expert that will just come up with a perfect response on the spot’. It’s fine to not know stuff, and it’s vital to be able to admit when you don’t. Signposting to a previous place the question has been discussed, or knowing that it will be covered later (if e.g. this comes up in a fellowship) if that is the case, both seem useful. It seems important to know enough about common questions, objections, and alternative viewpoints to be able to do this the majority of the time. If it’s genuinely something that the person running the session has never heard, this is exactly the time to demonstrate good epistemics—being willing to seriously engage, ask follow-up questions, and trying to double crux.