I agree with other commenters who have pointed out that using “more posts by previous prize-winning authors” as a proxy for the stated goal of “the creation of more content of the sort we want to see on the Forum” seems like a strange way to evaluate the efficacy of the Forum Prize. In addition to the points already mentioned, I would add two more:
It doesn’t consider potential variation in quality among posts by the same author. If prize-winning authors feel that they have set a standard that they feel it’s important to continue meeting in the future and that means they post less frequent but more thoughtful articles, that’s generally a trade I’d be happy to accept as a reader.
It ignores the potential impact of the Forum Prize on other people’s writing. How many people have been inspired to write something either because of the existence of the prize itself or because of some piece of writing that they learned about because of the prize? I would bet it’s not zero.
Indeed, I would argue that the prize adjudication process itself offers a useful infrastructure for evaluating the Forum experience. Since you have a record of the scores that posts received each month as well as the qualitative opinions of longtime judges, you have the tools you need to assess in a semi-rigorous way whether the quality of the top posts has increased or decreased over time.
I also wanted to express that if CEA really is ceasing the Forum Prize as such, that seems like a fairly major decision that should get its own top-level post, as the prize announcements themselves do. As it is, it’s buried in an article whose title poses what I think most people would consider to be a pretty esoteric research question, so I expect that a lot of people will miss it.
It ignores the potential impact of the Forum Prize on other people’s writing. How many people have been inspired to write something either because of the existence of the prize itself or because of some piece of writing that they learned about because of the prize? I would bet it’s not zero.
I’d also bet that it’s not zero, but my experience interviewing dozens of users and surveying hundreds more suggests that the number is not very high. (I focus my interviews on people we’d especially like to see submit more content, so the Prize being relatively unimportant to them seems quite informative in that light.)
I also wanted to express that if CEA really is ceasing the Forum Prize as such, that seems like a fairly major decision that should get its own top-level post, as the prize announcements themselves do. As it is, it’s buried in an article whose title poses what I think most people would consider to be a pretty esoteric research question, so I expect that a lot of people will miss it.
There will be another announcement in the upcoming Forum Prize post, which I expect to be read by more people. This post was ready before the other post, but it seemed a bit odd not to mention the decision to stop the Prize here, given the subject matter. (I’m about to note this in the above post, to clear up potential confusion.)
Indeed, I would argue that the prize adjudication process itself offers a useful infrastructure for evaluating the Forum experience. Since you have a record of the scores that posts received each month as well as the qualitative opinions of longtime judges, you have the tools you need to assess in a semi-rigorous way whether the quality of the top posts has increased or decreased over time.
Given that the number of votes remains basically fixed over time, the “record of the scores” wouldn’t reveal any change in quality — we don’t use a scale, just approval voting.
The judges are definitely the sorts of people I’d want to hear from w/r/t the perceived quality of top posts over time, but they are also the sorts of people who read the Forum a lot anyway (that’s why I asked them to judge), so I expect I’ll be able to get their opinion on this kind of thing pretty easily with or without the prize.
***
Overall, I also want to reiterate that the Forum Prize closing doesn’t mean we won’t be trying to reward quality posts, have judges evaluate posts, etc. — this will just be happening in other forms (using the budget we’ve freed up by not running the Prize). So I’m hoping that many of the benefits we looked for will be present in greater quantity with our new projects!
I agree with other commenters who have pointed out that using “more posts by previous prize-winning authors” as a proxy for the stated goal of “the creation of more content of the sort we want to see on the Forum” seems like a strange way to evaluate the efficacy of the Forum Prize. In addition to the points already mentioned, I would add two more:
It doesn’t consider potential variation in quality among posts by the same author. If prize-winning authors feel that they have set a standard that they feel it’s important to continue meeting in the future and that means they post less frequent but more thoughtful articles, that’s generally a trade I’d be happy to accept as a reader.
It ignores the potential impact of the Forum Prize on other people’s writing. How many people have been inspired to write something either because of the existence of the prize itself or because of some piece of writing that they learned about because of the prize? I would bet it’s not zero.
Indeed, I would argue that the prize adjudication process itself offers a useful infrastructure for evaluating the Forum experience. Since you have a record of the scores that posts received each month as well as the qualitative opinions of longtime judges, you have the tools you need to assess in a semi-rigorous way whether the quality of the top posts has increased or decreased over time.
I also wanted to express that if CEA really is ceasing the Forum Prize as such, that seems like a fairly major decision that should get its own top-level post, as the prize announcements themselves do. As it is, it’s buried in an article whose title poses what I think most people would consider to be a pretty esoteric research question, so I expect that a lot of people will miss it.
I’d also bet that it’s not zero, but my experience interviewing dozens of users and surveying hundreds more suggests that the number is not very high. (I focus my interviews on people we’d especially like to see submit more content, so the Prize being relatively unimportant to them seems quite informative in that light.)
There will be another announcement in the upcoming Forum Prize post, which I expect to be read by more people. This post was ready before the other post, but it seemed a bit odd not to mention the decision to stop the Prize here, given the subject matter. (I’m about to note this in the above post, to clear up potential confusion.)
Given that the number of votes remains basically fixed over time, the “record of the scores” wouldn’t reveal any change in quality — we don’t use a scale, just approval voting.
The judges are definitely the sorts of people I’d want to hear from w/r/t the perceived quality of top posts over time, but they are also the sorts of people who read the Forum a lot anyway (that’s why I asked them to judge), so I expect I’ll be able to get their opinion on this kind of thing pretty easily with or without the prize.
***
Overall, I also want to reiterate that the Forum Prize closing doesn’t mean we won’t be trying to reward quality posts, have judges evaluate posts, etc. — this will just be happening in other forms (using the budget we’ve freed up by not running the Prize). So I’m hoping that many of the benefits we looked for will be present in greater quantity with our new projects!