Here I’ll display summaries of the first 21 responses (I may update this later), and reflect on what I learned from this.[1]
I had also made predictions about what the survey results would be, to give myself some sort of ramshackle baseline to compare results against. I was going to share these predictions, then felt no one would be interested; but let me know if you’d like me to add them in a comment.
(Note that many of the things I’ve written were related to my work with Convergence Analysis, but my comments here reflect only my own opinions.)
The data
Q1:
Q2:
Q3:
Q4:
Q5: “If you think anything I’ve written has affected your beliefs, please say what that thing was (either titles or roughly what the topic was), and/or say how it affected your beliefs.”
(I didn’t ask for permission to share people’s comments, so, for this and the other comment questions, I’ll just highlight some recurring themes or seemingly noteworthy specifics.)
Most responses seemed to indicate the shift in beliefs caused by my work was fairly small.
Q6:
Q7: “If you think anything I’ve written has affected your decisions or plans, please say what that thing was (either titles or roughly what the topic was), and/or say how it affected your decisions or plans.”
5⁄21 respondents answered this
One respondent mentioned a way in which something I wrote contributed meaningfully to an output of theirs which I think is quite valuable
Q8, text box: “If you answered “Yes” to either of the above, could you say a bit about why?”
15⁄21 respondents filled in this text box
Some respondents indicated things “on their end” (e.g., busyness, attention span), or that they’d have said yes to one or both of those questions for most authors rather than just for me in particular
Some respondents mentioned topics just not seeming relevant to their interests
Some respondents mentioned my posts being long, being rambly, or failing to have a summary
Some respondents mentioned they were already well-versed in the areas I was writing about and didn’t feel my posts were necessary for them
Q9: “Do you have any other feedback on specific things I’ve written, my general writing style, my topic choices, or anything else?”
10⁄21 respondents answered this
Several non-specific positive comments/encouragements
Several positive or neutral comments on me having a lot of output
Several comments suggesting I should be more concise, use summaries more consistently, and/or be clearer about what the point of what I’m writing is
Some comments indicating appreciation of my summaries, collections, and efforts to make ideas accessible
Some comments on my writing style and clarity being good
Some comments that my original research wasn’t very impressive
One comment that I seem to hung up on defining things precisely/prescriptively
(I don’t actually endorse linguistic prescriptivism, and remember occasionally trying to make that explicit. But I’ll take this as useful data that I’ve sometimes accidentally given that impression, and try to adjust accordingly.)
Q10: “If you would like to share your name, please do so below. But this is 100% voluntary—you’re not at all obliged to do so :)”
6⁄21 respondents gave their name/username
2 gave their email for if I wanted to follow-up
Some takeaways from all this
Responses were notably more positive than expected for some questions, and notably less positive for others
I don’t think this should notably change my bottom-line view of the overall quality and impact of my work to date
But it does make me a little less uncertain about that all-things-considered view, as I now have slightly more data that roughly supports it
In turn, this updates me towards being a little more confident that it makes sense for me to focus on pursuing an EA research career for now (rather than, e.g., switching to operations or civil service roles)
This is because I’m now slightly less worried that I’m being strongly influenced by overconfidence or motivated reasoning. (I already wanted to do research or writing before learning about EA.)
I should definitely more consistently include summaries, and/or in other ways signal early and clearly what the point of a post is
I was already aiming to move in this direction, and had predicted responses would often mention this, but this has still given me an extra push
I should look out for ways in which I might appear linguistically prescriptive or overly focused on definitions/precision
I should more seriously consider moving more towards concision, even at the cost of precision, clarity, or comprehensiveness
Though I’m still not totally sold on that
I’m also aware that this shortform comment is not a great first step!
I should consider moving more towards concision, even at the cost of quantity/speed of output
With extra time on a given post, I could perhaps find ways to be more concise without sacrificing other valuable things
I should feel less like I “have to” produce writings rapidly
This point is harder to explain briefly, so I’ll just scratch the surface here
I don’t actually expect this to substantially change my behaviours, as that feeling wasn’t the main reason for my large amount of output
But if my output slows for some other reason, I think I’ll now not feel (as) bad about that
People have found my summaries and collections very useful, and some people have found my original research not so useful/impressive
The “direction” of this effect is in line with my expectations, but the strength was surprising
I’ve updated towards more confidence that my summaries and (especially) my collections were valuable and worth making, and this may slightly increase the already-high chance that I’ll continue creating that sort of thing
But this is also slightly confusing, as my original research/ideas and/or aptitude for future original research seems to have put me in good stead for various job and grant selection processes
And I don’t have indications that my summaries or collections helped there, though they may have
Much of my work to date may be less useful for more experienced/engaged EAs than less experienced/engaged EAs
This is in line with my sense that I was often trying to make ideas more accessible, make getting up to speed easier, etc.
There seemed to be a weak correlation between how recently something was posted and how often it was positively mentioned
This broadly aligns with trends from other data sources (e.g., researchers reaching out to me, upvotes)
This could suggest that:
my work is getting better
people are paying more attention to things written by me, regardless of their quality
people just remember the recent stuff more
I’d guess all three of those factors play some role
(I also have additional thoughts that are fuzzier or even less likely to be of interest to anyone other than me.)
[1] There are of course myriad reasons to not read into this data too much, including that:
it’s from a sample of only 21 people
the sample was non-representative, and indeed self-selecting (so it may, for example, disproportionately represent people who like my work)
the responses may be biased towards not hurting my feelings
That said, I think I can still learn something from this data, especially given flaws in other data sources I have. (E.g., comments from people who choose to randomly and non-anonymously reach out to me may be even more positively biased.)
“People have found my summaries and collections very useful, and some people have found my original research not so useful/impressive”
I haven’t read enough of your original research to know whether it applies in your case but just flagging that most original research has a much narrower target audience than the summaries/collections, so I’d expect fewer people to find it useful (and for a relatively broad summary to be biased against them).
That said, as you know, I think your summaries/collections are useful and underprovided.
Though I guess I suspect that, if the reason a person finds my original research not so useful is just because they aren’t the target audience, they’d be more likely to either not explicitly comment on it or to say something about it not seeming relevant to them. (Rather than making a generic comment about it not seeming useful.)
But I guess this seems less likely in cases where:
the person doesn’t realise that the key reason it wasn’t useful is that they weren’t the target audience, or
the person feels that what they’re focused on is substantially more important than anything else (because then they’ll perceive “useful to them” as meaning a very similar thing to “useful”)
In any case, I’m definitely just taking this survey as providing weak (though useful) evidence, and combining it with various other sources of evidence.
Reflections on data from a survey about things I’ve written
I recently requested people take a survey on the quality/impact of things I’ve written. So far, 22 people have generously taken the survey. (Please add yourself to that tally!)
Here I’ll display summaries of the first 21 responses (I may update this later), and reflect on what I learned from this.[1]
I had also made predictions about what the survey results would be, to give myself some sort of ramshackle baseline to compare results against. I was going to share these predictions, then felt no one would be interested; but let me know if you’d like me to add them in a comment.
For my thoughts on how worthwhile this was and whether other researchers/organisations should run similar surveys, see Should surveys about the quality/impact of research outputs be more common?
(Note that many of the things I’ve written were related to my work with Convergence Analysis, but my comments here reflect only my own opinions.)
The data
Q1:
Q2:
Q3:
Q4:
Q5: “If you think anything I’ve written has affected your beliefs, please say what that thing was (either titles or roughly what the topic was), and/or say how it affected your beliefs.”
(I didn’t ask for permission to share people’s comments, so, for this and the other comment questions, I’ll just highlight some recurring themes or seemingly noteworthy specifics.)
9⁄21 respondents answered this
The writings people mentioned specifically were my collections and summaries of existing ideas/work (e.g., A central directory for open research questions), Database of existential risk estimates, Improving the future by influencing actors’ benevolence, intelligence, and power, and my comments on the Google doc of another person who wanted feedback.
Most responses seemed to indicate the shift in beliefs caused by my work was fairly small.
Q6:
Q7: “If you think anything I’ve written has affected your decisions or plans, please say what that thing was (either titles or roughly what the topic was), and/or say how it affected your decisions or plans.”
5⁄21 respondents answered this
One respondent mentioned a way in which something I wrote contributed meaningfully to an output of theirs which I think is quite valuable
One respondent indicated Some history topics it might be very valuable to investigate influenced them somewhat
Another indicated Improving the future by influencing actors’ benevolence, intelligence, and power might inform an important decision
There was one other small influence
Q8:
Q8, text box: “If you answered “Yes” to either of the above, could you say a bit about why?”
15⁄21 respondents filled in this text box
Some respondents indicated things “on their end” (e.g., busyness, attention span), or that they’d have said yes to one or both of those questions for most authors rather than just for me in particular
Some respondents mentioned topics just not seeming relevant to their interests
Some respondents mentioned my posts being long, being rambly, or failing to have a summary
Some respondents mentioned they were already well-versed in the areas I was writing about and didn’t feel my posts were necessary for them
Q9: “Do you have any other feedback on specific things I’ve written, my general writing style, my topic choices, or anything else?”
10⁄21 respondents answered this
Several non-specific positive comments/encouragements
Several positive or neutral comments on me having a lot of output
Several comments suggesting I should be more concise, use summaries more consistently, and/or be clearer about what the point of what I’m writing is
Some comments indicating appreciation of my summaries, collections, and efforts to make ideas accessible
Some comments on my writing style and clarity being good
Some comments that my original research wasn’t very impressive
One comment that I seem to hung up on defining things precisely/prescriptively
(I don’t actually endorse linguistic prescriptivism, and remember occasionally trying to make that explicit. But I’ll take this as useful data that I’ve sometimes accidentally given that impression, and try to adjust accordingly.)
Q10: “If you would like to share your name, please do so below. But this is 100% voluntary—you’re not at all obliged to do so :)”
6⁄21 respondents gave their name/username
2 gave their email for if I wanted to follow-up
Some takeaways from all this
Responses were notably more positive than expected for some questions, and notably less positive for others
I don’t think this should notably change my bottom-line view of the overall quality and impact of my work to date
But it does make me a little less uncertain about that all-things-considered view, as I now have slightly more data that roughly supports it
In turn, this updates me towards being a little more confident that it makes sense for me to focus on pursuing an EA research career for now (rather than, e.g., switching to operations or civil service roles)
This is because I’m now slightly less worried that I’m being strongly influenced by overconfidence or motivated reasoning. (I already wanted to do research or writing before learning about EA.)
I should definitely more consistently include summaries, and/or in other ways signal early and clearly what the point of a post is
I was already aiming to move in this direction, and had predicted responses would often mention this, but this has still given me an extra push
I should look out for ways in which I might appear linguistically prescriptive or overly focused on definitions/precision
I should more seriously consider moving more towards concision, even at the cost of precision, clarity, or comprehensiveness
Though I’m still not totally sold on that
I’m also aware that this shortform comment is not a great first step!
I should consider moving more towards concision, even at the cost of quantity/speed of output
With extra time on a given post, I could perhaps find ways to be more concise without sacrificing other valuable things
I should feel less like I “have to” produce writings rapidly
This point is harder to explain briefly, so I’ll just scratch the surface here
I don’t actually expect this to substantially change my behaviours, as that feeling wasn’t the main reason for my large amount of output
But if my output slows for some other reason, I think I’ll now not feel (as) bad about that
People have found my summaries and collections very useful, and some people have found my original research not so useful/impressive
The “direction” of this effect is in line with my expectations, but the strength was surprising
I’ve updated towards more confidence that my summaries and (especially) my collections were valuable and worth making, and this may slightly increase the already-high chance that I’ll continue creating that sort of thing
But this is also slightly confusing, as my original research/ideas and/or aptitude for future original research seems to have put me in good stead for various job and grant selection processes
And I don’t have indications that my summaries or collections helped there, though they may have
Much of my work to date may be less useful for more experienced/engaged EAs than less experienced/engaged EAs
This is in line with my sense that I was often trying to make ideas more accessible, make getting up to speed easier, etc.
There seemed to be a weak correlation between how recently something was posted and how often it was positively mentioned
This broadly aligns with trends from other data sources (e.g., researchers reaching out to me, upvotes)
This could suggest that:
my work is getting better
people are paying more attention to things written by me, regardless of their quality
people just remember the recent stuff more
I’d guess all three of those factors play some role
(I also have additional thoughts that are fuzzier or even less likely to be of interest to anyone other than me.)
[1] There are of course myriad reasons to not read into this data too much, including that:
it’s from a sample of only 21 people
the sample was non-representative, and indeed self-selecting (so it may, for example, disproportionately represent people who like my work)
the responses may be biased towards not hurting my feelings
That said, I think I can still learn something from this data, especially given flaws in other data sources I have. (E.g., comments from people who choose to randomly and non-anonymously reach out to me may be even more positively biased.)
If you’ve made it this far, you may also be interested in the above-mentioned Should surveys about the quality/impact of research outputs be more common?
“People have found my summaries and collections very useful, and some people have found my original research not so useful/impressive”
I haven’t read enough of your original research to know whether it applies in your case but just flagging that most original research has a much narrower target audience than the summaries/collections, so I’d expect fewer people to find it useful (and for a relatively broad summary to be biased against them).
That said, as you know, I think your summaries/collections are useful and underprovided.
Good point.
Though I guess I suspect that, if the reason a person finds my original research not so useful is just because they aren’t the target audience, they’d be more likely to either not explicitly comment on it or to say something about it not seeming relevant to them. (Rather than making a generic comment about it not seeming useful.)
But I guess this seems less likely in cases where:
the person doesn’t realise that the key reason it wasn’t useful is that they weren’t the target audience, or
the person feels that what they’re focused on is substantially more important than anything else (because then they’ll perceive “useful to them” as meaning a very similar thing to “useful”)
In any case, I’m definitely just taking this survey as providing weak (though useful) evidence, and combining it with various other sources of evidence.
Seems reasonable