I was somewhat confused by the scale using Categorizing Variants of Goodhart’s Law as an example of a 100mQ paper, given that the LW post version of that paper won the 2018 AI Alignment Prize ($5k), which makes a pretty strong case for it being “a particularly valuable paper” (1Q, the next category up). I also think this scale significantly overvalues research agendas and popular books relative to papers. I don’t think these aspects of the rubric wound up impacting the specific estimates made here, though.
I’m not sure on the exact valuation research agendas should get, but I would argue that well thought-through research agendas can be hugely beneficial in that they can reorient many researchers in high-impact directions, leading them to write papers on topics that are vastly more important than they might have otherwise chosen.
I would argue an ‘ingenious’ paper written on an unimportant topic isn’t anywhere near as good as a ‘pretty good’ paper written on a hugely important topic.
Yes, the scale is under construction, and you’re not the first person to mention that the specific research agenda mentioned is overvalued.