Prediction is hard; it thus seems a bit cruel to grade students based on brier score, since it can be hard for some people to improve at that (also, over the course of one class you wouldn’t get much time to iterate and improve at forecasting, so you’d be grading people in part on their newbie guesses early on). This would be like teaching an intro-to-economics class and then grading students based on whether their stock picks outperformed the S&P 500 -- too difficult a task for an intro course, and too much randomness to seem fair. Maybe grading heavily on Brier score could be appropriate in an advanced class like a grad-level course for students in DC-area universities who are aiming to become intelligence-agency analysts. And of course it would be fine to have Brier score in an intro course be worth only a token portion of one’s grade, like 5%, just to make things fun.
But I like the general idea of a class based around learning about forecasting and prediction markets (or maybe as one unit in a larger class on economics, civics, or statistics). The idea of assigning students to create and participate in a metaculus market seems good; I just wouldn’t grade based on brier score. Instead of using students’ grades to motivate them, I’d split the class into teams, give each team a private group chat where they can discuss questions, and use people’s competitive instincts (individual + team leaderboards) to create motivation.
In general I think university teachers miss out on many chances to have their students get involved in doing useful work; eg having ecology students contribute to the wikipedia entry for a particular tree species rather than just write a report on that species.
Prediction is hard; it thus seems a bit cruel to grade students based on brier score, since it can be hard for some people to improve at that (also, over the course of one class you wouldn’t get much time to iterate and improve at forecasting, so you’d be grading people in part on their newbie guesses early on). This would be like teaching an intro-to-economics class and then grading students based on whether their stock picks outperformed the S&P 500 -- too difficult a task for an intro course, and too much randomness to seem fair. Maybe grading heavily on Brier score could be appropriate in an advanced class like a grad-level course for students in DC-area universities who are aiming to become intelligence-agency analysts. And of course it would be fine to have Brier score in an intro course be worth only a token portion of one’s grade, like 5%, just to make things fun.
But I like the general idea of a class based around learning about forecasting and prediction markets (or maybe as one unit in a larger class on economics, civics, or statistics). The idea of assigning students to create and participate in a metaculus market seems good; I just wouldn’t grade based on brier score. Instead of using students’ grades to motivate them, I’d split the class into teams, give each team a private group chat where they can discuss questions, and use people’s competitive instincts (individual + team leaderboards) to create motivation.
In general I think university teachers miss out on many chances to have their students get involved in doing useful work; eg having ecology students contribute to the wikipedia entry for a particular tree species rather than just write a report on that species.
That’s why I suggested the prediction market would be based on a curve relative to one’s classmates :) I may go back and emphasize that point.