One thing Iâve been wondering about is how much impact will result (indirectly) from the wiki/âtagging system, and, relatedly, from various types of individual contributions to it.
Iâm wondering this both to work out whether Iâm spending more time contributing than I really should and to work out whether I should do more/âless encouraging of other people to contribute. (This also seems relevant for âtargeted solicitation toward Forum users we expect will be strong editors for specific articlesââhaving a clearer impact assessment would help both in deciding when itâs worth people spending their time this way and in convincing them of that.)
Currently Iâas you can probably tellâfeel pretty excited about the wiki and tagging system, often send people links to entries, often suggest they edit or tag things, etc. But I donât actually have a clear sense of how good a use of peopleâs time this is.
---
I think itâd help to create an explicit theory of change for the wiki and tagging system, or maybe one for each of those two components. Maybe ideally in diagram form. Then you/âwe could try to use this to identify assumptions, identify what proxies we should pay attention to and date we should gather, identify what forecasts we should make, etc. (See also.)
---
Here are some other proxy/âimpact assessment ideas, though theyâre kind-of putting the cart before the horse without having worked out a more explicit theory of change first. (This also overlaps with discussion weâve previously had non-publicly, where Pablo mentioned some other ideas I wonât mention here.)
Look at total or average page views and work out how good or bad those results are
Obviously youâve already got the page view data, but at least personally I donât know whether thatâs good or bad or what I wanted to see (except âmore is betterâ)
Run something like Rethink Prioritiesâ impact survey or my personal adaptation of that (see also), where either relatively âkeyâ people or just a whole bunch of Forum users are asked how much theyâve engaged with the wiki/âtags, how high-quality it seemed to be, and whether and how it affected their beliefs or behaviours
(I work at Rethink Priorities, but amâas usualâwriting this comment in a personal capacity, and was a fan and referencer of their impact survey before joining)
What Iâd have in mind as a quick and dirty option is basically just replacing âanything Iâve writtenâ or âanything by Rethink Priorities on [topic]â with âany wiki entriesâ
Itâs less obvious how to use this to capture the benefits of the tag system, rather than the benefits of the wiki system. Maybe:
Similar questions, but for âhave you found content to read via looking at tag pages or filtering by tagsâ, âdo you think you wouldâve found that content anywayâ, âdo you think that that changed your beliefs (relative to what you mightâve done with that time otherwise)â, etc.
Also asking whether people have shared tag pages with people as collections of resources, whether they think that changed the recipients beliefs/âbehaviours, etc.
See if the wiki/âtags show up in a future EA Survey as a way people learned about or became more engaged with EA?
But I guess I feel like the wiki is relatively unlikely to serve that role?
A caveat I should add: The wiki could be seen more as a product or website than a research output/âproject. And Iâm more used to thinking about how to assess the impact of the latter (though Iâm not an expert on that anyway).So maybe my suggestions are a bit off, and maybe itâd be useful to think more about things like:
How 80k assess the impact of their services
How the Forum and LessWrong assess their impact or performance or whatever
How software companies assess how their products are doing (aside from just revenue)
Thanks for this post :)
One thing Iâve been wondering about is how much impact will result (indirectly) from the wiki/âtagging system, and, relatedly, from various types of individual contributions to it.
Iâm wondering this both to work out whether Iâm spending more time contributing than I really should and to work out whether I should do more/âless encouraging of other people to contribute. (This also seems relevant for âtargeted solicitation toward Forum users we expect will be strong editors for specific articlesââhaving a clearer impact assessment would help both in deciding when itâs worth people spending their time this way and in convincing them of that.)
Currently Iâas you can probably tellâfeel pretty excited about the wiki and tagging system, often send people links to entries, often suggest they edit or tag things, etc. But I donât actually have a clear sense of how good a use of peopleâs time this is.
---
I think itâd help to create an explicit theory of change for the wiki and tagging system, or maybe one for each of those two components. Maybe ideally in diagram form. Then you/âwe could try to use this to identify assumptions, identify what proxies we should pay attention to and date we should gather, identify what forecasts we should make, etc. (See also.)
---
Here are some other proxy/âimpact assessment ideas, though theyâre kind-of putting the cart before the horse without having worked out a more explicit theory of change first. (This also overlaps with discussion weâve previously had non-publicly, where Pablo mentioned some other ideas I wonât mention here.)
Look at total or average page views and work out how good or bad those results are
Obviously youâve already got the page view data, but at least personally I donât know whether thatâs good or bad or what I wanted to see (except âmore is betterâ)
Run something like Rethink Prioritiesâ impact survey or my personal adaptation of that (see also), where either relatively âkeyâ people or just a whole bunch of Forum users are asked how much theyâve engaged with the wiki/âtags, how high-quality it seemed to be, and whether and how it affected their beliefs or behaviours
(I work at Rethink Priorities, but amâas usualâwriting this comment in a personal capacity, and was a fan and referencer of their impact survey before joining)
What Iâd have in mind as a quick and dirty option is basically just replacing âanything Iâve writtenâ or âanything by Rethink Priorities on [topic]â with âany wiki entriesâ
Itâs less obvious how to use this to capture the benefits of the tag system, rather than the benefits of the wiki system. Maybe:
Similar questions, but for âhave you found content to read via looking at tag pages or filtering by tagsâ, âdo you think you wouldâve found that content anywayâ, âdo you think that that changed your beliefs (relative to what you mightâve done with that time otherwise)â, etc.
Also asking whether people have shared tag pages with people as collections of resources, whether they think that changed the recipients beliefs/âbehaviours, etc.
See if the wiki/âtags show up in a future EA Survey as a way people learned about or became more engaged with EA?
But I guess I feel like the wiki is relatively unlikely to serve that role?
It seems better suited for getting people from participants to contributors or contributors to âcoreâ, or helping them do so faster and better, than getting people to enter the audience or become followers
A caveat I should add: The wiki could be seen more as a product or website than a research output/âproject. And Iâm more used to thinking about how to assess the impact of the latter (though Iâm not an expert on that anyway).So maybe my suggestions are a bit off, and maybe itâd be useful to think more about things like:
How 80k assess the impact of their services
How the Forum and LessWrong assess their impact or performance or whatever
How software companies assess how their products are doing (aside from just revenue)