Does the Forum Prize lead people to write more posts?

For the past several years, CEA has awarded a regular prize to authors of Forum posts (selected by a group of 4-6 judges, one of whom works for CEA). The frequency has changed somewhat over the years, but has generally been a monthly prize given to 3-5 authors. Our hope was that this prize would encourage the creation of more content of the sort we want to see on the Forum. For more information about the prize, you can see our most recent post.

Perhaps the most plausible way in which this could happen is that the authors of prize-winning posts are incentivized to post more frequently. We therefore examined whether prize-winning authors post more frequently in the six months following their prize than in the six months prior to it, relative to a control group.

We found that there was no increase in frequency. This result, combined with other qualitative information we’ve heard from Forum users, has caused us to reconsider the Forum Prize process. In September we launched a creative writing contest, separate from the usual prize, and our initial impression is that this has generated much more engagement.

More on that qualitative feedback: While people generally react quite positively to winning the prize, few people have explicitly told us it made them want to write more (even when we asked directly), and our surveys of the Forum’s userbase haven’t found many people saying that the chance of winning a prize leads them to invest more time in writing.


Going forward, we no longer plan to run the “standard” Forum Prize process. Instead, we expect to experiment with different ways of incentivizing writing. While nothing is set in stone, examples of things we’ve considered include prizes specifically for new authors and contests for the best posts on particular topics.

(We’ll also make this announcement in the upcoming final Forum Prize post, so that it will be more visible.)

Below is more information on our methodology. We expect this to only be of interest to a small group of people, and have therefore left the post unpolished. Please feel free to comment with any questions or clarifications.

Method

Test set

A test set with one record for each Forum Prize awarded

  • Get the publication dates of all posts for every user that appears in this set

  • Compare each user’s mean post interval in the 6 months before the prize’s announcement date with that user’s mean post interval in the 6 months after the announcement

  • Each prize appears as a separate data point (i.e. repeat winners have multiple data points)

Control set

A control set with one row for all posts with >50 karma, excluding authors who won any Forum Prize at any time

  • Get the publication dates of all posts for every user that appears in this set

  • Pretend that each post won an imaginary prize and allocate a corresponding “announcement date” depending on when the post was published (e.g. a post written in April 2021 has an announcement date of 29th July 2021)

  • Compare each user’s mean post interval in the 6 months before the imaginary prize announcement date with the user’s mean post interval in the 6 months after the imaginary announcement

  • Each >50 karma post appears as a separate datapoint (i.e. “repeat winners” have multiple data points)

Results

The ratio [mean posting interval after] /​ [mean posting interval before] is shown. If there was an effect, we should expect the top two box and whisker plots to be to the left of the bottom two plots.

https://​​chart-studio.plotly.com/​​~cea_a_kara/​​1.embed

Here is the same data, but looking at the average change in terms of number of days (i.e. subtracting the after versus before frequency, as opposed to dividing them). Again, an effect would look like the top plots being to the left of the bottom ones.