One thing I’ve been wondering about is how much impact will result (indirectly) from the wiki/tagging system, and, relatedly, from various types of individual contributions to it.
I’m wondering this both to work out whether I’m spending more time contributing than I really should and to work out whether I should do more/less encouraging of other people to contribute. (This also seems relevant for “targeted solicitation toward Forum users we expect will be strong editors for specific articles”—having a clearer impact assessment would help both in deciding when it’s worth people spending their time this way and in convincing them of that.)
Currently I—as you can probably tell—feel pretty excited about the wiki and tagging system, often send people links to entries, often suggest they edit or tag things, etc. But I don’t actually have a clear sense of how good a use of people’s time this is.
---
I think it’d help to create an explicit theory of change for the wiki and tagging system, or maybe one for each of those two components. Maybe ideally in diagram form. Then you/we could try to use this to identify assumptions, identify what proxies we should pay attention to and date we should gather, identify what forecasts we should make, etc. (See also.)
---
Here are some other proxy/impact assessment ideas, though they’re kind-of putting the cart before the horse without having worked out a more explicit theory of change first. (This also overlaps with discussion we’ve previously had non-publicly, where Pablo mentioned some other ideas I won’t mention here.)
Look at total or average page views and work out how good or bad those results are
Obviously you’ve already got the page view data, but at least personally I don’t know whether that’s good or bad or what I wanted to see (except “more is better”)
Run something like Rethink Priorities’ impact survey or my personal adaptation of that (see also), where either relatively “key” people or just a whole bunch of Forum users are asked how much they’ve engaged with the wiki/tags, how high-quality it seemed to be, and whether and how it affected their beliefs or behaviours
(I work at Rethink Priorities, but am—as usual—writing this comment in a personal capacity, and was a fan and referencer of their impact survey before joining)
What I’d have in mind as a quick and dirty option is basically just replacing “anything I’ve written” or “anything by Rethink Priorities on [topic]” with “any wiki entries”
It’s less obvious how to use this to capture the benefits of the tag system, rather than the benefits of the wiki system. Maybe:
Similar questions, but for “have you found content to read via looking at tag pages or filtering by tags”, “do you think you would’ve found that content anyway”, “do you think that that changed your beliefs (relative to what you might’ve done with that time otherwise)”, etc.
Also asking whether people have shared tag pages with people as collections of resources, whether they think that changed the recipients beliefs/behaviours, etc.
See if the wiki/tags show up in a future EA Survey as a way people learned about or became more engaged with EA?
But I guess I feel like the wiki is relatively unlikely to serve that role?
A caveat I should add: The wiki could be seen more as a product or website than a research output/project. And I’m more used to thinking about how to assess the impact of the latter (though I’m not an expert on that anyway).So maybe my suggestions are a bit off, and maybe it’d be useful to think more about things like:
How 80k assess the impact of their services
How the Forum and LessWrong assess their impact or performance or whatever
How software companies assess how their products are doing (aside from just revenue)
Thanks for this post :)
One thing I’ve been wondering about is how much impact will result (indirectly) from the wiki/tagging system, and, relatedly, from various types of individual contributions to it.
I’m wondering this both to work out whether I’m spending more time contributing than I really should and to work out whether I should do more/less encouraging of other people to contribute. (This also seems relevant for “targeted solicitation toward Forum users we expect will be strong editors for specific articles”—having a clearer impact assessment would help both in deciding when it’s worth people spending their time this way and in convincing them of that.)
Currently I—as you can probably tell—feel pretty excited about the wiki and tagging system, often send people links to entries, often suggest they edit or tag things, etc. But I don’t actually have a clear sense of how good a use of people’s time this is.
---
I think it’d help to create an explicit theory of change for the wiki and tagging system, or maybe one for each of those two components. Maybe ideally in diagram form. Then you/we could try to use this to identify assumptions, identify what proxies we should pay attention to and date we should gather, identify what forecasts we should make, etc. (See also.)
---
Here are some other proxy/impact assessment ideas, though they’re kind-of putting the cart before the horse without having worked out a more explicit theory of change first. (This also overlaps with discussion we’ve previously had non-publicly, where Pablo mentioned some other ideas I won’t mention here.)
Look at total or average page views and work out how good or bad those results are
Obviously you’ve already got the page view data, but at least personally I don’t know whether that’s good or bad or what I wanted to see (except “more is better”)
Run something like Rethink Priorities’ impact survey or my personal adaptation of that (see also), where either relatively “key” people or just a whole bunch of Forum users are asked how much they’ve engaged with the wiki/tags, how high-quality it seemed to be, and whether and how it affected their beliefs or behaviours
(I work at Rethink Priorities, but am—as usual—writing this comment in a personal capacity, and was a fan and referencer of their impact survey before joining)
What I’d have in mind as a quick and dirty option is basically just replacing “anything I’ve written” or “anything by Rethink Priorities on [topic]” with “any wiki entries”
It’s less obvious how to use this to capture the benefits of the tag system, rather than the benefits of the wiki system. Maybe:
Similar questions, but for “have you found content to read via looking at tag pages or filtering by tags”, “do you think you would’ve found that content anyway”, “do you think that that changed your beliefs (relative to what you might’ve done with that time otherwise)”, etc.
Also asking whether people have shared tag pages with people as collections of resources, whether they think that changed the recipients beliefs/behaviours, etc.
See if the wiki/tags show up in a future EA Survey as a way people learned about or became more engaged with EA?
But I guess I feel like the wiki is relatively unlikely to serve that role?
It seems better suited for getting people from participants to contributors or contributors to “core”, or helping them do so faster and better, than getting people to enter the audience or become followers
A caveat I should add: The wiki could be seen more as a product or website than a research output/project. And I’m more used to thinking about how to assess the impact of the latter (though I’m not an expert on that anyway).So maybe my suggestions are a bit off, and maybe it’d be useful to think more about things like:
How 80k assess the impact of their services
How the Forum and LessWrong assess their impact or performance or whatever
How software companies assess how their products are doing (aside from just revenue)