Thanks for taking the time to add these really useful observation, Seb.
One downside to this approach is that it might lead to Goodharting and leading the teacher to go in āexploitationā mode. E.g., I worry that I might become too attached to a specific outcome on behalf of the students and tacitly start to persuade (similar to some concerns expressed by Theo Hawkins) and/āor neglect other important opportunities that might emerge during the program. How do you think of that risk?
Itās been a while since I read Theoās post so I might be missing the mark here. I agree both explore and exploit are important, especially for young people. I havenāt thought deeply about this but my intuition says āif itās also important to x, be explicit that you have multiple goals.ā For example, to use ācreate personal theory of changeā via Future Academy is the goal, you might also want people to ācreate tentative career plans for 5 distinct careersā, or ādevelop connections so you have 3 people you could call to ask for career adviceā. Sure the latter isnāt a ālearning objectiveā and it might be better un-said. Still, I think a generally good way of goodharting might be using multiple goals or criteria for success.
2. Can you say anything about what forms summative assessments are particularly useful? For Future Academy, weāre contemplating pitching project ideas or presentation and discussion of career plans
The word that comes to mind is to make it āauthentic.ā Basically, make it as close as possible to the real world skill you want people to do. This is rare. Universities expect critical thinking, creativity, and communication, but use recall-based multiple-choice questions. Iāve seen essays and reflections to assess interpersonal skills, instead of videos or presentations. Pitching project ideas and presenting career plans sounds well above average. If I had to nit-pick, Iāve never āpresented my career plansā, so to make it slightly closer to something people might do anyway would be āwrite a grant application.ā
I think thereās a typo under 3a. (āFormative assessmentsā ā> formative activities)?
Both are things. I should have clarified it. Formative assessments are formative activities that count toward a grade or completion status. As mentioned by another commenter, low-stakes quizzes are helpful for providing feedback and accountability to learners, but better fit university courses than fellowships etc.
I worry about this being true for on average for average university students and might not generalize to the subpopulation that some portion of community-building efforts is targetted towards (e.g., people who are in the 90th percentile on various domains, including openness to experience, conscientiousness, need-for-cognition, etc.). How worried are you about this?
This is an important question. I donāt think I know yet how big a problem this is (as I said, people should reach out if they want to work on it). One of the benefits of having worked in sport and performance psychology is that it mostly focuses on people in the top 1ā5% of their field. As far as I can tell, the core principles underlying most of the above (psychological needs; deliberate practice; cognitive load limits) still apply to those people. You do need to calibrate the challenge to the person. People in the top 5% are going to be bored if you spend 10 hours explaining a t-test. So, Iām sure some things donāt generalise perfectly, but I think thatās more likely to be the specific techniques (e.g., āuse quizzesā) than the mechanisms (e.g., āprovide feedbackā).
ā¦ being generally good people (or virtuous) in addition to the unique virtues you mentioned appears important as we have some research showing that this might be off-putting. Finally, same-race role-models appear to be particularly important.
Yeah I didnāt go into this much so itās a good pickup. Both are useful to remember.
Thanks for taking the time to add these really useful observation, Seb.
Itās been a while since I read Theoās post so I might be missing the mark here. I agree both explore and exploit are important, especially for young people. I havenāt thought deeply about this but my intuition says āif itās also important to x, be explicit that you have multiple goals.ā For example, to use ācreate personal theory of changeā via Future Academy is the goal, you might also want people to ācreate tentative career plans for 5 distinct careersā, or ādevelop connections so you have 3 people you could call to ask for career adviceā. Sure the latter isnāt a ālearning objectiveā and it might be better un-said. Still, I think a generally good way of goodharting might be using multiple goals or criteria for success.
The word that comes to mind is to make it āauthentic.ā Basically, make it as close as possible to the real world skill you want people to do. This is rare. Universities expect critical thinking, creativity, and communication, but use recall-based multiple-choice questions. Iāve seen essays and reflections to assess interpersonal skills, instead of videos or presentations. Pitching project ideas and presenting career plans sounds well above average. If I had to nit-pick, Iāve never āpresented my career plansā, so to make it slightly closer to something people might do anyway would be āwrite a grant application.ā
Both are things. I should have clarified it. Formative assessments are formative activities that count toward a grade or completion status. As mentioned by another commenter, low-stakes quizzes are helpful for providing feedback and accountability to learners, but better fit university courses than fellowships etc.
This is an important question. I donāt think I know yet how big a problem this is (as I said, people should reach out if they want to work on it). One of the benefits of having worked in sport and performance psychology is that it mostly focuses on people in the top 1ā5% of their field. As far as I can tell, the core principles underlying most of the above (psychological needs; deliberate practice; cognitive load limits) still apply to those people. You do need to calibrate the challenge to the person. People in the top 5% are going to be bored if you spend 10 hours explaining a t-test. So, Iām sure some things donāt generalise perfectly, but I think thatās more likely to be the specific techniques (e.g., āuse quizzesā) than the mechanisms (e.g., āprovide feedbackā).
Yeah I didnāt go into this much so itās a good pickup. Both are useful to remember.