I was one of the facilitators in the most recent run of EA Cambridge’s AGI Safety Fundamentals course, and I also have professional DS/ML experience.
In my case I very deliberately emphasised a sceptical approach to engaging with all the material, while providing clarifications and corrections where people’s misconceptions are the source of scepticism. I believe this was well-received by my cohort, all of whom appeared to engage thoughtfully and honestly with the material.
I think this is the best way to engage, when time permits, because (in brief)
many arguments invoke ill-defined terms, and we need to sharpen these
many arguments are (perhaps explicitly) speculative and empirically uncertain
even mathematically/empirically rigorous content has important modelling assumptions and experimental caveats
scepticism often produces better creative/generative engagement
collectively we will fail if our individual opinions are overly shaped by founder effects
I hope that this is a common perspective, but to the extent that it isn’t, I wonder if this (especially the last point) may be a source of some of your confusing experiences.
I’d also say: it seems appropriate to have ‘very messy views’ if by that you mean uncertainty about where things are going and how to make them better! I think folks who don’t are doing one of two things
mistakenly concentrating more hypothesis weight than their observations/thinking in fact justify (which is a bad idea)
engaging in a thinking manoeuvre something like ‘temporary MAP stance’ or ‘subjective probability matching’ (which may be a good idea, if done transparently)
‘Temporary MAP stance’ or ‘subjective probability matching’
MAP is Maximum A Posteriori i.e. your best guess after considering evidence. Probability matching is making actions/guesses proportional to your estimate of them being right (rather than picking the single MAP choice)
By this manoeuvre I’m gesturing at a kind of behaviour where you are quite unsure about what’s best (e.g. ‘should I work on interpretability or demystifying deception?’) and rather than allowing that to result in analysis paralysis, you temporarily collapse some uncertainty and make some concrete assumptions to get moving in one or other direction. Hopefully in so doing you a) make a contribution and b) grow your skills and collect new evidence to make better decisions/contributions next time.
I feel like everyone I have ever talked about AI safety with would agree on the importance of thinking critically and staying skeptical, and this includes my facilitator and cohort members from the AGISF programme.
I think the 1.5h discussion session between 5 people who have read 5 texts does not allow really going deep into any topics, since it is just ~3 minutes per participant per text on average. I think these kind of programs are great for meeting new people, clearing misconceptions and providing structure/accountability on actually reading the material, but they by nature are not that good for having in-depth debates. I think that’s ok, but just to clarify why I think it is normal I probably did not mention most of the things I described on this post during the discussion sessions.
But there is an additional reason that I think is more important to me, which is differentiating between performing skepticism and actually voicing true opinions. It is not possible for my facilitator to notice which one I am doing because they don’t know me, and performing skepticism (in order to conform to the perceived standard of “you have to think about all of this critically and by your own, and you will probably arrive to similar conclusions than others in this field”) looks the same as actually raising the confusions you have. This is why I thought I can convey this failure mode to others by comparing to inner misalignment :)
When I was a Math freshman my professor told us he always encourages people to ask questions during lectures. Often, it had happened that he’d explained a concept and nobody would ask anything. He’d check what the students understood, and it would turn out they did not grasp the concept. When asking why nobody asked anything, the students would say that they did not understand enough to ask a good question. To avoid this dynamic, he told us that “I did not understand anything” counts as a valid question on his lectures. It helped somewhat but at least I still often stayed silent instead of raising my hand and saying “I did not understand anything”.
I feel like the same dynamic can easily happen when discussing AI safety (or any difficult EA concept, really). If people are encouraged to raise questions and concerns they might only raise the “good” ones, and stay silent if they feel like they just did not understand the concepts well enough (like I did in my avoidance strategy 1).
OK, this is the terrible terrible failure mode which I think we are both agreeing on (emphasis mine)
the perceived standard of “you have to think about all of this critically and by your own, and you will probably arrive to similar conclusions than others in this field”
By ‘a sceptical approach’ I basically mean ‘the thing where we don’t do that’. Because there is not enough epistemic credit in the field, yet, to expect that all (tentative, not-consensus-yet) conclusions to be definitely right.
In traditional/undergraduate mathematics, it’s different—almost always when you don’t understand or agree with the professor, she is simply right and you are simply wrong or confused! This is a justifiable perspective based on the enormous epistemic weight of all the existing work on mathematics.
I’m very glad you call out the distinction between performing skepticism and actually doing it.
Yeah, I think we agree on this, I think I want to write out more later on what communication strategies might help people actually voice scepticsm/concerns even if they are afraid of meeting some standards on elaborateness.
My mathematics example actually tried to be about this: in my university, the teachers tried to make us forget the teachers are more likely to be right, so that we would have to think about things on our own and voice scepticism even if we were objectively likely to be wrong. I remember another lecturer telling us: “if you finish an excercise and notice you did not use all the assuptions in your proof, you either did something wrong or you came up with a very important discovery”. I liked how she stated that it was indeed possible that a person from our freshman group could make a novel discovery, however unlikely that was.
The point is that my lecturers tried to teach that there is not a certain level you have to acquire before your opinions start to matter: you might be right even if you are a total beginner and the person you disagree with has a lot of experience.
This is something I would like to emphasize when doing EA community building myself, but it is not very easy. I’ve seen this when I’ve taught programming to kids. If a kid asks me if their program is “done” or “good”, I’d say “you are the programmer, do you think your program does what it is supposed to do”, but usually the kids think it is a trick question and I’m just withholding the correct answer for fun. Adults, too, do not always trust that I actually value their opinion.
I was one of the facilitators in the most recent run of EA Cambridge’s AGI Safety Fundamentals course, and I also have professional DS/ML experience.
In my case I very deliberately emphasised a sceptical approach to engaging with all the material, while providing clarifications and corrections where people’s misconceptions are the source of scepticism. I believe this was well-received by my cohort, all of whom appeared to engage thoughtfully and honestly with the material.
I think this is the best way to engage, when time permits, because (in brief)
many arguments invoke ill-defined terms, and we need to sharpen these
many arguments are (perhaps explicitly) speculative and empirically uncertain
even mathematically/empirically rigorous content has important modelling assumptions and experimental caveats
scepticism often produces better creative/generative engagement
collectively we will fail if our individual opinions are overly shaped by founder effects
I hope that this is a common perspective, but to the extent that it isn’t, I wonder if this (especially the last point) may be a source of some of your confusing experiences.
I’d also say: it seems appropriate to have ‘very messy views’ if by that you mean uncertainty about where things are going and how to make them better! I think folks who don’t are doing one of two things
mistakenly concentrating more hypothesis weight than their observations/thinking in fact justify (which is a bad idea)
engaging in a thinking manoeuvre something like ‘temporary MAP stance’ or ‘subjective probability matching’ (which may be a good idea, if done transparently)
‘Temporary MAP stance’ or ‘subjective probability matching’
MAP is Maximum A Posteriori i.e. your best guess after considering evidence. Probability matching is making actions/guesses proportional to your estimate of them being right (rather than picking the single MAP choice)
By this manoeuvre I’m gesturing at a kind of behaviour where you are quite unsure about what’s best (e.g. ‘should I work on interpretability or demystifying deception?’) and rather than allowing that to result in analysis paralysis, you temporarily collapse some uncertainty and make some concrete assumptions to get moving in one or other direction. Hopefully in so doing you a) make a contribution and b) grow your skills and collect new evidence to make better decisions/contributions next time.
I feel like everyone I have ever talked about AI safety with would agree on the importance of thinking critically and staying skeptical, and this includes my facilitator and cohort members from the AGISF programme.
I think the 1.5h discussion session between 5 people who have read 5 texts does not allow really going deep into any topics, since it is just ~3 minutes per participant per text on average. I think these kind of programs are great for meeting new people, clearing misconceptions and providing structure/accountability on actually reading the material, but they by nature are not that good for having in-depth debates. I think that’s ok, but just to clarify why I think it is normal I probably did not mention most of the things I described on this post during the discussion sessions.
But there is an additional reason that I think is more important to me, which is differentiating between performing skepticism and actually voicing true opinions. It is not possible for my facilitator to notice which one I am doing because they don’t know me, and performing skepticism (in order to conform to the perceived standard of “you have to think about all of this critically and by your own, and you will probably arrive to similar conclusions than others in this field”) looks the same as actually raising the confusions you have. This is why I thought I can convey this failure mode to others by comparing to inner misalignment :)
When I was a Math freshman my professor told us he always encourages people to ask questions during lectures. Often, it had happened that he’d explained a concept and nobody would ask anything. He’d check what the students understood, and it would turn out they did not grasp the concept. When asking why nobody asked anything, the students would say that they did not understand enough to ask a good question. To avoid this dynamic, he told us that “I did not understand anything” counts as a valid question on his lectures. It helped somewhat but at least I still often stayed silent instead of raising my hand and saying “I did not understand anything”.
I feel like the same dynamic can easily happen when discussing AI safety (or any difficult EA concept, really). If people are encouraged to raise questions and concerns they might only raise the “good” ones, and stay silent if they feel like they just did not understand the concepts well enough (like I did in my avoidance strategy 1).
OK, this is the terrible terrible failure mode which I think we are both agreeing on (emphasis mine)
By ‘a sceptical approach’ I basically mean ‘the thing where we don’t do that’. Because there is not enough epistemic credit in the field, yet, to expect that all (tentative, not-consensus-yet) conclusions to be definitely right.
In traditional/undergraduate mathematics, it’s different—almost always when you don’t understand or agree with the professor, she is simply right and you are simply wrong or confused! This is a justifiable perspective based on the enormous epistemic weight of all the existing work on mathematics.
I’m very glad you call out the distinction between performing skepticism and actually doing it.
Yeah, I think we agree on this, I think I want to write out more later on what communication strategies might help people actually voice scepticsm/concerns even if they are afraid of meeting some standards on elaborateness.
My mathematics example actually tried to be about this: in my university, the teachers tried to make us forget the teachers are more likely to be right, so that we would have to think about things on our own and voice scepticism even if we were objectively likely to be wrong. I remember another lecturer telling us: “if you finish an excercise and notice you did not use all the assuptions in your proof, you either did something wrong or you came up with a very important discovery”. I liked how she stated that it was indeed possible that a person from our freshman group could make a novel discovery, however unlikely that was.
The point is that my lecturers tried to teach that there is not a certain level you have to acquire before your opinions start to matter: you might be right even if you are a total beginner and the person you disagree with has a lot of experience.
This is something I would like to emphasize when doing EA community building myself, but it is not very easy. I’ve seen this when I’ve taught programming to kids. If a kid asks me if their program is “done” or “good”, I’d say “you are the programmer, do you think your program does what it is supposed to do”, but usually the kids think it is a trick question and I’m just withholding the correct answer for fun. Adults, too, do not always trust that I actually value their opinion.