I feel like everyone I have ever talked about AI safety with would agree on the importance of thinking critically and staying skeptical, and this includes my facilitator and cohort members from the AGISF programme.
I think the 1.5h discussion session between 5 people who have read 5 texts does not allow really going deep into any topics, since it is just ~3 minutes per participant per text on average. I think these kind of programs are great for meeting new people, clearing misconceptions and providing structure/accountability on actually reading the material, but they by nature are not that good for having in-depth debates. I think that’s ok, but just to clarify why I think it is normal I probably did not mention most of the things I described on this post during the discussion sessions.
But there is an additional reason that I think is more important to me, which is differentiating between performing skepticism and actually voicing true opinions. It is not possible for my facilitator to notice which one I am doing because they don’t know me, and performing skepticism (in order to conform to the perceived standard of “you have to think about all of this critically and by your own, and you will probably arrive to similar conclusions than others in this field”) looks the same as actually raising the confusions you have. This is why I thought I can convey this failure mode to others by comparing to inner misalignment :)
When I was a Math freshman my professor told us he always encourages people to ask questions during lectures. Often, it had happened that he’d explained a concept and nobody would ask anything. He’d check what the students understood, and it would turn out they did not grasp the concept. When asking why nobody asked anything, the students would say that they did not understand enough to ask a good question. To avoid this dynamic, he told us that “I did not understand anything” counts as a valid question on his lectures. It helped somewhat but at least I still often stayed silent instead of raising my hand and saying “I did not understand anything”.
I feel like the same dynamic can easily happen when discussing AI safety (or any difficult EA concept, really). If people are encouraged to raise questions and concerns they might only raise the “good” ones, and stay silent if they feel like they just did not understand the concepts well enough (like I did in my avoidance strategy 1).
OK, this is the terrible terrible failure mode which I think we are both agreeing on (emphasis mine)
the perceived standard of “you have to think about all of this critically and by your own, and you will probably arrive to similar conclusions than others in this field”
By ‘a sceptical approach’ I basically mean ‘the thing where we don’t do that’. Because there is not enough epistemic credit in the field, yet, to expect that all (tentative, not-consensus-yet) conclusions to be definitely right.
In traditional/undergraduate mathematics, it’s different—almost always when you don’t understand or agree with the professor, she is simply right and you are simply wrong or confused! This is a justifiable perspective based on the enormous epistemic weight of all the existing work on mathematics.
I’m very glad you call out the distinction between performing skepticism and actually doing it.
Yeah, I think we agree on this, I think I want to write out more later on what communication strategies might help people actually voice scepticsm/concerns even if they are afraid of meeting some standards on elaborateness.
My mathematics example actually tried to be about this: in my university, the teachers tried to make us forget the teachers are more likely to be right, so that we would have to think about things on our own and voice scepticism even if we were objectively likely to be wrong. I remember another lecturer telling us: “if you finish an excercise and notice you did not use all the assuptions in your proof, you either did something wrong or you came up with a very important discovery”. I liked how she stated that it was indeed possible that a person from our freshman group could make a novel discovery, however unlikely that was.
The point is that my lecturers tried to teach that there is not a certain level you have to acquire before your opinions start to matter: you might be right even if you are a total beginner and the person you disagree with has a lot of experience.
This is something I would like to emphasize when doing EA community building myself, but it is not very easy. I’ve seen this when I’ve taught programming to kids. If a kid asks me if their program is “done” or “good”, I’d say “you are the programmer, do you think your program does what it is supposed to do”, but usually the kids think it is a trick question and I’m just withholding the correct answer for fun. Adults, too, do not always trust that I actually value their opinion.
I feel like everyone I have ever talked about AI safety with would agree on the importance of thinking critically and staying skeptical, and this includes my facilitator and cohort members from the AGISF programme.
I think the 1.5h discussion session between 5 people who have read 5 texts does not allow really going deep into any topics, since it is just ~3 minutes per participant per text on average. I think these kind of programs are great for meeting new people, clearing misconceptions and providing structure/accountability on actually reading the material, but they by nature are not that good for having in-depth debates. I think that’s ok, but just to clarify why I think it is normal I probably did not mention most of the things I described on this post during the discussion sessions.
But there is an additional reason that I think is more important to me, which is differentiating between performing skepticism and actually voicing true opinions. It is not possible for my facilitator to notice which one I am doing because they don’t know me, and performing skepticism (in order to conform to the perceived standard of “you have to think about all of this critically and by your own, and you will probably arrive to similar conclusions than others in this field”) looks the same as actually raising the confusions you have. This is why I thought I can convey this failure mode to others by comparing to inner misalignment :)
When I was a Math freshman my professor told us he always encourages people to ask questions during lectures. Often, it had happened that he’d explained a concept and nobody would ask anything. He’d check what the students understood, and it would turn out they did not grasp the concept. When asking why nobody asked anything, the students would say that they did not understand enough to ask a good question. To avoid this dynamic, he told us that “I did not understand anything” counts as a valid question on his lectures. It helped somewhat but at least I still often stayed silent instead of raising my hand and saying “I did not understand anything”.
I feel like the same dynamic can easily happen when discussing AI safety (or any difficult EA concept, really). If people are encouraged to raise questions and concerns they might only raise the “good” ones, and stay silent if they feel like they just did not understand the concepts well enough (like I did in my avoidance strategy 1).
OK, this is the terrible terrible failure mode which I think we are both agreeing on (emphasis mine)
By ‘a sceptical approach’ I basically mean ‘the thing where we don’t do that’. Because there is not enough epistemic credit in the field, yet, to expect that all (tentative, not-consensus-yet) conclusions to be definitely right.
In traditional/undergraduate mathematics, it’s different—almost always when you don’t understand or agree with the professor, she is simply right and you are simply wrong or confused! This is a justifiable perspective based on the enormous epistemic weight of all the existing work on mathematics.
I’m very glad you call out the distinction between performing skepticism and actually doing it.
Yeah, I think we agree on this, I think I want to write out more later on what communication strategies might help people actually voice scepticsm/concerns even if they are afraid of meeting some standards on elaborateness.
My mathematics example actually tried to be about this: in my university, the teachers tried to make us forget the teachers are more likely to be right, so that we would have to think about things on our own and voice scepticism even if we were objectively likely to be wrong. I remember another lecturer telling us: “if you finish an excercise and notice you did not use all the assuptions in your proof, you either did something wrong or you came up with a very important discovery”. I liked how she stated that it was indeed possible that a person from our freshman group could make a novel discovery, however unlikely that was.
The point is that my lecturers tried to teach that there is not a certain level you have to acquire before your opinions start to matter: you might be right even if you are a total beginner and the person you disagree with has a lot of experience.
This is something I would like to emphasize when doing EA community building myself, but it is not very easy. I’ve seen this when I’ve taught programming to kids. If a kid asks me if their program is “done” or “good”, I’d say “you are the programmer, do you think your program does what it is supposed to do”, but usually the kids think it is a trick question and I’m just withholding the correct answer for fun. Adults, too, do not always trust that I actually value their opinion.