Thanks! That does seem super relevant. Is my understanding below roughly correct?
”A post on a non-community topic that receives ~500 karma is roughly equivalent in impact to a high quality research paper in a peer reviewed journal. And such a post receiving about 100 karma or a bit below is about 1-10% of the impact of such a journal article.”
A post on a non-community topic that receives ~500 karma is roughly equivalent in impact to a high quality research paper in a peer reviewed journal
This makes sense assuming i) impact increases logarithmically with karma, ii) Nuño’s impact estimates of the EA Forum Prize winners are correct, and iii) the relationship between karma and impact among these posts generalises to other posts. However, I have so little confidence about these assumptions that I would not use karma as the only proxy for impact. At best, I would use the logarithm of karma as one component of a weighted factor model (WFM). I note in the post that:
The predictions above [by assuming i), ii) and iii)] are quite poor. For instance, they imply:
An unreasonably small difference of:
0.367 karma between the impact of “a good blog post, a particularly good comment” and “a thoughtful comment”.
3.67 karma between the impact of “an excellent blog post” and “a good blog post, a particularly good comment”.
“A particularly valuable paper” is worth 475 karma, whereas I think the right value is of the order of magnitude of 10 kkarma (with huge variation), i.e. 21.1 times as large.
In any case, one should certainly be mindful of Goodhart’s Law, and do not start optimising posts just for karma!
Thanks! That does seem super relevant. Is my understanding below roughly correct?
”A post on a non-community topic that receives ~500 karma is roughly equivalent in impact to a high quality research paper in a peer reviewed journal. And such a post receiving about 100 karma or a bit below is about 1-10% of the impact of such a journal article.”
You are welcome!
This makes sense assuming i) impact increases logarithmically with karma, ii) Nuño’s impact estimates of the EA Forum Prize winners are correct, and iii) the relationship between karma and impact among these posts generalises to other posts. However, I have so little confidence about these assumptions that I would not use karma as the only proxy for impact. At best, I would use the logarithm of karma as one component of a weighted factor model (WFM). I note in the post that: