Thanks so much for writing this up. I carefully read through the “how to teach effectively in five steps” and got a lot of value from it. I’ve yet to read through other aspects of the blog post.
The thing I love the most about this section is that it builds on robust findings from meta-analyses. Why do I love this? Because i) nothingin community-building is based on robust evidence and ii) it counteracts a tendency for EA to rely to much on what’s EA—reinventing a new paradigm rather than recalling that the majority of what’s useful in the world in the world isn’t “EA” (we’re substantially standing on the shoulders of giants).
Additionally, here’s a couple of reactions and questions to five step section specifically: 1. I appreciate the focus on defining a learning objective as a skill and then working backward from that. I think that “creating personal theory of changes and impactful career plans that one can wholeheartedly (or agentically) pursue” is a good default to have in mind. One downside to this approach is that it might lead to Goodharting and leading the teacher to go in “exploitation” mode. E.g., I worry that I might become too attached to a specific outcome on behalf of the students and tacitly start to persuade (similar to some concerns expressed by Theo Hawkins) and/or neglect other important opportunities that might emerge during the program. How do you think of that risk?
2. Can you say anything about what forms summative assessments are particularly useful? For Future Academy, we’re contemplating pitching project ideas or presentation and discussion of career plans (although we likely wouldn’t label it as an assessment but rather as an exciting culmination of their work).
3. I think there’s a typo under 3a. (“Formative assessments” —> formative activities)?
4. While I appreciate the rigor of the evidence upon which this post is built, I worry about this being true for on average for average university students and might not generalize to the subpopulation that some portion of community-building efforts is targetted towards (e.g., people who are in the 90th percentile on various domains, including openness to experience, conscientiousness, need-for-cognition, etc.). How worried are you about this?
5. Strongly agree with the importance of role models. Humans are deeply social and our social incentives (including role models) might be the most important things to change. In fact, being generally good people (or virtuous) in addition to the unique virtues you mentioned appears important as we have some research showing that this might be off-putting. Finally, same-race role-models appear to be particularly important.
Thanks for taking the time to add these really useful observation, Seb.
One downside to this approach is that it might lead to Goodharting and leading the teacher to go in “exploitation” mode. E.g., I worry that I might become too attached to a specific outcome on behalf of the students and tacitly start to persuade (similar to some concerns expressed by Theo Hawkins) and/or neglect other important opportunities that might emerge during the program. How do you think of that risk?
It’s been a while since I read Theo’s post so I might be missing the mark here. I agree both explore and exploit are important, especially for young people. I haven’t thought deeply about this but my intuition says “if it’s also important to x, be explicit that you have multiple goals.” For example, to use ‘create personal theory of change’ via Future Academy is the goal, you might also want people to ‘create tentative career plans for 5 distinct careers’, or ‘develop connections so you have 3 people you could call to ask for career advice’. Sure the latter isn’t a ‘learning objective’ and it might be better un-said. Still, I think a generally good way of goodharting might be using multiple goals or criteria for success.
2. Can you say anything about what forms summative assessments are particularly useful? For Future Academy, we’re contemplating pitching project ideas or presentation and discussion of career plans
The word that comes to mind is to make it ‘authentic.’ Basically, make it as close as possible to the real world skill you want people to do. This is rare. Universities expect critical thinking, creativity, and communication, but use recall-based multiple-choice questions. I’ve seen essays and reflections to assess interpersonal skills, instead of videos or presentations. Pitching project ideas and presenting career plans sounds well above average. If I had to nit-pick, I’ve never ‘presented my career plans’, so to make it slightly closer to something people might do anyway would be ‘write a grant application.’
I think there’s a typo under 3a. (“Formative assessments” —> formative activities)?
Both are things. I should have clarified it. Formative assessments are formative activities that count toward a grade or completion status. As mentioned by another commenter, low-stakes quizzes are helpful for providing feedback and accountability to learners, but better fit university courses than fellowships etc.
I worry about this being true for on average for average university students and might not generalize to the subpopulation that some portion of community-building efforts is targetted towards (e.g., people who are in the 90th percentile on various domains, including openness to experience, conscientiousness, need-for-cognition, etc.). How worried are you about this?
This is an important question. I don’t think I know yet how big a problem this is (as I said, people should reach out if they want to work on it). One of the benefits of having worked in sport and performance psychology is that it mostly focuses on people in the top 1–5% of their field. As far as I can tell, the core principles underlying most of the above (psychological needs; deliberate practice; cognitive load limits) still apply to those people. You do need to calibrate the challenge to the person. People in the top 5% are going to be bored if you spend 10 hours explaining a t-test. So, I’m sure some things don’t generalise perfectly, but I think that’s more likely to be the specific techniques (e.g., ‘use quizzes’) than the mechanisms (e.g., ‘provide feedback’).
… being generally good people (or virtuous) in addition to the unique virtues you mentioned appears important as we have some research showing that this might be off-putting. Finally, same-race role-models appear to be particularly important.
Yeah I didn’t go into this much so it’s a good pickup. Both are useful to remember.
Thanks so much for writing this up.
I carefully read through the “how to teach effectively in five steps” and got a lot of value from it. I’ve yet to read through other aspects of the blog post.
The thing I love the most about this section is that it builds on robust findings from meta-analyses. Why do I love this? Because i) nothing in community-building is based on robust evidence and ii) it counteracts a tendency for EA to rely to much on what’s EA—reinventing a new paradigm rather than recalling that the majority of what’s useful in the world in the world isn’t “EA” (we’re substantially standing on the shoulders of giants).
Additionally, here’s a couple of reactions and questions to five step section specifically:
1. I appreciate the focus on defining a learning objective as a skill and then working backward from that. I think that “creating personal theory of changes and impactful career plans that one can wholeheartedly (or agentically) pursue” is a good default to have in mind. One downside to this approach is that it might lead to Goodharting and leading the teacher to go in “exploitation” mode. E.g., I worry that I might become too attached to a specific outcome on behalf of the students and tacitly start to persuade (similar to some concerns expressed by Theo Hawkins) and/or neglect other important opportunities that might emerge during the program. How do you think of that risk?
2. Can you say anything about what forms summative assessments are particularly useful? For Future Academy, we’re contemplating pitching project ideas or presentation and discussion of career plans (although we likely wouldn’t label it as an assessment but rather as an exciting culmination of their work).
3. I think there’s a typo under 3a. (“Formative assessments” —> formative activities)?
4. While I appreciate the rigor of the evidence upon which this post is built, I worry about this being true for on average for average university students and might not generalize to the subpopulation that some portion of community-building efforts is targetted towards (e.g., people who are in the 90th percentile on various domains, including openness to experience, conscientiousness, need-for-cognition, etc.). How worried are you about this?
5. Strongly agree with the importance of role models. Humans are deeply social and our social incentives (including role models) might be the most important things to change. In fact, being generally good people (or virtuous) in addition to the unique virtues you mentioned appears important as we have some research showing that this might be off-putting. Finally, same-race role-models appear to be particularly important.
Thanks for taking the time to add these really useful observation, Seb.
It’s been a while since I read Theo’s post so I might be missing the mark here. I agree both explore and exploit are important, especially for young people. I haven’t thought deeply about this but my intuition says “if it’s also important to x, be explicit that you have multiple goals.” For example, to use ‘create personal theory of change’ via Future Academy is the goal, you might also want people to ‘create tentative career plans for 5 distinct careers’, or ‘develop connections so you have 3 people you could call to ask for career advice’. Sure the latter isn’t a ‘learning objective’ and it might be better un-said. Still, I think a generally good way of goodharting might be using multiple goals or criteria for success.
The word that comes to mind is to make it ‘authentic.’ Basically, make it as close as possible to the real world skill you want people to do. This is rare. Universities expect critical thinking, creativity, and communication, but use recall-based multiple-choice questions. I’ve seen essays and reflections to assess interpersonal skills, instead of videos or presentations. Pitching project ideas and presenting career plans sounds well above average. If I had to nit-pick, I’ve never ‘presented my career plans’, so to make it slightly closer to something people might do anyway would be ‘write a grant application.’
Both are things. I should have clarified it. Formative assessments are formative activities that count toward a grade or completion status. As mentioned by another commenter, low-stakes quizzes are helpful for providing feedback and accountability to learners, but better fit university courses than fellowships etc.
This is an important question. I don’t think I know yet how big a problem this is (as I said, people should reach out if they want to work on it). One of the benefits of having worked in sport and performance psychology is that it mostly focuses on people in the top 1–5% of their field. As far as I can tell, the core principles underlying most of the above (psychological needs; deliberate practice; cognitive load limits) still apply to those people. You do need to calibrate the challenge to the person. People in the top 5% are going to be bored if you spend 10 hours explaining a t-test. So, I’m sure some things don’t generalise perfectly, but I think that’s more likely to be the specific techniques (e.g., ‘use quizzes’) than the mechanisms (e.g., ‘provide feedback’).
Yeah I didn’t go into this much so it’s a good pickup. Both are useful to remember.