Thanks for suggesting a tweak to the forum and doing so in such a clear way. I agree that voting is relatively limited, but it’d be good to first know whether this is an inevitable result of a relatively limited audience who are interested in voting. I’d be curious to hear Ryan Carey’s view as to that. If that’s the primary explanation, then I don’t think we can justify the time arranging for the experiment that you suggest. (It sounds relatively simple technically, but tweaks always take longer than you expect, and we’ve found the forum code can be surprisingly slow to work with.)
I did some quick, far from scientific analytics work on this to get a feeling for the additional room for participation.
First, I wanted to identify a series of posts to examine. I wanted a consecutive series so that there would not be any introduced bias. I wanted the series to be recent (to reflect recent activity) but not within a couple weeks, as I wanted it to reflect the lifecycle of a post. I also wanted the series to have what seemed like a representative distribution of upvotes.
I selected the 5 post series from Kuhn’s post about GiveWell’s 2015 recommendations to Ben Todd’s post on WALYs.
I ran some analytics. I do not know if there is a norm that EA Forum traffic numbers are not released, so instead I’ll just report a single statistic that seems most relevant.
In these 5 posts, on average 4.17% of unique visitors to the posts voted (I was sure to include both up and down votes). The range was from 1% to 6% of unique visitors to a post voting.
Of course, this only tells us a bit. If we consider the goal to be to determine what percent of visitors with meaningful opinions voted, we could say this number is inflated, since it only considers those that visited a post, and some people may know with confidence that they want to upvote or downvote based on the title on the front-page alone. This number could be an underestimate, because unique visitors is not a 100% reliable statistic, failing, for example, to adjust for the same visitor reading a post on two different devices.
Besides considering whether it is an over or underestimate, it still lacks context, and I do not believe context is going to be truly available. I did find a reddit post that has a slightly interesting discussion of possible voting behavior on that site: http://bit.ly/1OJq25b
Personally, I am of the opinion that voting behavior, percentage-wise is currently OK compared to other sites, but that this largely accounts for an engaged readership, with the other sites I’m referencing not being highly comparable. I think it is highly likely that our voting should well exceed that of other sites. And I think it is optimal to capture as much feedback as possible, so I would have a leaning toward implementation almost regardless of the current participation data.
Putting up and down arrows on the front page would make sense if the content included lots of hugs and memes that readers are quickly opening and closing. For longer-form pieces, its better for readers to be encouraged to vote on the page in which they read it, from a UX point of view. Especially so since we place a premium on quality. We can see that the quality of the most upvoted posts is currently good. Given that we only have a few levers that influence writing quality, let’s not lose this one.
I agree that more votes would be useful, but this should be readily achieved if the overall audience increases. I think that the real challenge is to roughly double readership and participation, and that people’s work ought to be put to that goal. If that can be achieved, then the future of the forum would beore stable and the experience of using it will be more fulfilling.
Thanks for suggesting a tweak to the forum and doing so in such a clear way. I agree that voting is relatively limited, but it’d be good to first know whether this is an inevitable result of a relatively limited audience who are interested in voting. I’d be curious to hear Ryan Carey’s view as to that. If that’s the primary explanation, then I don’t think we can justify the time arranging for the experiment that you suggest. (It sounds relatively simple technically, but tweaks always take longer than you expect, and we’ve found the forum code can be surprisingly slow to work with.)
I did some quick, far from scientific analytics work on this to get a feeling for the additional room for participation.
First, I wanted to identify a series of posts to examine. I wanted a consecutive series so that there would not be any introduced bias. I wanted the series to be recent (to reflect recent activity) but not within a couple weeks, as I wanted it to reflect the lifecycle of a post. I also wanted the series to have what seemed like a representative distribution of upvotes.
I selected the 5 post series from Kuhn’s post about GiveWell’s 2015 recommendations to Ben Todd’s post on WALYs.
I ran some analytics. I do not know if there is a norm that EA Forum traffic numbers are not released, so instead I’ll just report a single statistic that seems most relevant.
In these 5 posts, on average 4.17% of unique visitors to the posts voted (I was sure to include both up and down votes). The range was from 1% to 6% of unique visitors to a post voting.
Of course, this only tells us a bit. If we consider the goal to be to determine what percent of visitors with meaningful opinions voted, we could say this number is inflated, since it only considers those that visited a post, and some people may know with confidence that they want to upvote or downvote based on the title on the front-page alone. This number could be an underestimate, because unique visitors is not a 100% reliable statistic, failing, for example, to adjust for the same visitor reading a post on two different devices.
Besides considering whether it is an over or underestimate, it still lacks context, and I do not believe context is going to be truly available. I did find a reddit post that has a slightly interesting discussion of possible voting behavior on that site: http://bit.ly/1OJq25b
Personally, I am of the opinion that voting behavior, percentage-wise is currently OK compared to other sites, but that this largely accounts for an engaged readership, with the other sites I’m referencing not being highly comparable. I think it is highly likely that our voting should well exceed that of other sites. And I think it is optimal to capture as much feedback as possible, so I would have a leaning toward implementation almost regardless of the current participation data.
Putting up and down arrows on the front page would make sense if the content included lots of hugs and memes that readers are quickly opening and closing. For longer-form pieces, its better for readers to be encouraged to vote on the page in which they read it, from a UX point of view. Especially so since we place a premium on quality. We can see that the quality of the most upvoted posts is currently good. Given that we only have a few levers that influence writing quality, let’s not lose this one.
I agree that more votes would be useful, but this should be readily achieved if the overall audience increases. I think that the real challenge is to roughly double readership and participation, and that people’s work ought to be put to that goal. If that can be achieved, then the future of the forum would beore stable and the experience of using it will be more fulfilling.