I believe Angelina left a note in the post where she shared this graph, but right before that cycle we changed the quiz (only 4⁄10 of the questions are the same). We knew these new questions were quite a bit harder, although candidly we were not expecting score drops this dramatic. We’re doing some investigation of the questions, and my guess is we’ll iterate a bit more since I think some of the new questions were difficult in ways that are not actually about understanding of the content.
With all that said, I wouldn’t focus too much on changes in this score right now, since they are almost certainly tracking changes in evaluation criteria and not changes in student understanding.
Ok thats makes sense, it’s so extreme that a eval change makes sense.
The font/prominence about the note is very smol. For these situations, for non-internal tools, sometimes people choose to add a vertical line or shade the graph to indicate a program change.
Hi Defacto! I work on Virtual Programs at CEA.
I believe Angelina left a note in the post where she shared this graph, but right before that cycle we changed the quiz (only 4⁄10 of the questions are the same). We knew these new questions were quite a bit harder, although candidly we were not expecting score drops this dramatic. We’re doing some investigation of the questions, and my guess is we’ll iterate a bit more since I think some of the new questions were difficult in ways that are not actually about understanding of the content.
With all that said, I wouldn’t focus too much on changes in this score right now, since they are almost certainly tracking changes in evaluation criteria and not changes in student understanding.
Ok thats makes sense, it’s so extreme that a eval change makes sense.
The font/prominence about the note is very smol. For these situations, for non-internal tools, sometimes people choose to add a vertical line or shade the graph to indicate a program change.
Here’s a TikTok of a kitten
https://www.tiktok.com/@officialkittenlady/video/7279576743395167534