I believe Angelina left a note in the post where she shared this graph, but right before that cycle we changed the quiz (only 4⁄10 of the questions are the same). We knew these new questions were quite a bit harder, although candidly we were not expecting score drops this dramatic. We’re doing some investigation of the questions, and my guess is we’ll iterate a bit more since I think some of the new questions were difficult in ways that are not actually about understanding of the content.
With all that said, I wouldn’t focus too much on changes in this score right now, since they are almost certainly tracking changes in evaluation criteria and not changes in student understanding.
Ok thats makes sense, it’s so extreme that a eval change makes sense.
The font/prominence about the note is very smol. For these situations, for non-internal tools, sometimes people choose to add a vertical line or shade the graph to indicate a program change.
example reasons:
someone with a different vision has started and there’s some rough spots to smooth out
someone with a different vision has started and this is exactly the correct score produced by their correct programs
there was gaming or gardening of the scores and this gardening stopped
staff left or anticipates leaving after funding pulled, and have a “schools out” mentality, e.g. participants showing up to empty zoom meeting
reception to the surprise SBF appearance less positive than expectedHi Defacto! I work on Virtual Programs at CEA.
I believe Angelina left a note in the post where she shared this graph, but right before that cycle we changed the quiz (only 4⁄10 of the questions are the same). We knew these new questions were quite a bit harder, although candidly we were not expecting score drops this dramatic. We’re doing some investigation of the questions, and my guess is we’ll iterate a bit more since I think some of the new questions were difficult in ways that are not actually about understanding of the content.
With all that said, I wouldn’t focus too much on changes in this score right now, since they are almost certainly tracking changes in evaluation criteria and not changes in student understanding.
Ok thats makes sense, it’s so extreme that a eval change makes sense.
The font/prominence about the note is very smol. For these situations, for non-internal tools, sometimes people choose to add a vertical line or shade the graph to indicate a program change.
Here’s a TikTok of a kitten
https://www.tiktok.com/@officialkittenlady/video/7279576743395167534