In addition to touching on the OP’s question, I think this is extremely important to point out because, of all the meta institutions that can be miscarried, evaluators or a “GiveWell for X” are structurally the most prone to misuse, and this is dangerous.
To explain:
Recall the narrative above, that evaluator skill is hard to see. As a result, the public face of an evaluator is inherently superficial (and often deliberately dressed for public consumption and socialization). Few actually see the underlying knowledge or ability.
Evaluators must also be independent—they can’t just be voted down by the entities they review. They must resist public pressure, sort of like a judge.
The danger is that these features can be cruelly captured by parties who talk the talk, and occupy the spaces for these evaluator institutions.
There is the further danger that, once they start polishing their public copy, it’s hard to draw the line. It’s a slippery place to be. It’s not difficult to glide into long or wacky worldviews, or introduce technical methods that are shallow, and defeat examination through rhetoric or obfuscation.
(While it can occur) the issues often aren’t driven by malice.
To see this, click on my name. You’ll see writing in many comments. Some of it touches on purported deep topics. It sounds impressive, but is it true?
What if most of my comments and ideas are sophomoric and hardly informative at all? If I lack deep understanding, I’ll never be able to see that I’m a hack.
I can’t tell I’m merely a person with just enough knowledge to write a coherent story. For many observers, they will know even less—but regardless, me, I’ll continue on whether I’m right or wrong.
In the same way that someone can write endlessly about meta or talk about politics, it can be tempting to feel that evaluation is easy.
The underlying patterns are similar, unfortunately the inertia and power of the institutions created can be large. Once started, it’s unclear what, if any, mechanism is available to stop it.
In addition to touching on the OP’s question, I think this is extremely important to point out because, of all the meta institutions that can be miscarried, evaluators or a “GiveWell for X” are structurally the most prone to misuse, and this is dangerous.
To explain:
Recall the narrative above, that evaluator skill is hard to see. As a result, the public face of an evaluator is inherently superficial (and often deliberately dressed for public consumption and socialization). Few actually see the underlying knowledge or ability.
Evaluators must also be independent—they can’t just be voted down by the entities they review. They must resist public pressure, sort of like a judge.
The danger is that these features can be cruelly captured by parties who talk the talk, and occupy the spaces for these evaluator institutions.
There is the further danger that, once they start polishing their public copy, it’s hard to draw the line. It’s a slippery place to be. It’s not difficult to glide into long or wacky worldviews, or introduce technical methods that are shallow, and defeat examination through rhetoric or obfuscation.
(While it can occur) the issues often aren’t driven by malice.
To see this, click on my name. You’ll see writing in many comments. Some of it touches on purported deep topics. It sounds impressive, but is it true?
What if most of my comments and ideas are sophomoric and hardly informative at all? If I lack deep understanding, I’ll never be able to see that I’m a hack.
I can’t tell I’m merely a person with just enough knowledge to write a coherent story. For many observers, they will know even less—but regardless, me, I’ll continue on whether I’m right or wrong.
In the same way that someone can write endlessly about meta or talk about politics, it can be tempting to feel that evaluation is easy.
The underlying patterns are similar, unfortunately the inertia and power of the institutions created can be large. Once started, it’s unclear what, if any, mechanism is available to stop it.