Here Iāll display summaries of the first 21 responses (I may update this later), and reflect on what I learned from this.[1]
I had also made predictions about what the survey results would be, to give myself some sort of ramshackle baseline to compare results against. I was going to share these predictions, then felt no one would be interested; but let me know if youād like me to add them in a comment.
(Note that many of the things Iāve written were related to my work with Convergence Analysis, but my comments here reflect only my own opinions.)
The data
Q1:
Q2:
Q3:
Q4:
Q5: āIf you think anything Iāve written has affected your beliefs, please say what that thing was (either titles or roughly what the topic was), and/āor say how it affected your beliefs.ā
(I didnāt ask for permission to share peopleās comments, so, for this and the other comment questions, Iāll just highlight some recurring themes or seemingly noteworthy specifics.)
Most responses seemed to indicate the shift in beliefs caused by my work was fairly small.
Q6:
Q7: āIf you think anything Iāve written has affected your decisions or plans, please say what that thing was (either titles or roughly what the topic was), and/āor say how it affected your decisions or plans.ā
5ā21 respondents answered this
One respondent mentioned a way in which something I wrote contributed meaningfully to an output of theirs which I think is quite valuable
Q8, text box: āIf you answered āYesā to either of the above, could you say a bit about why?ā
15ā21 respondents filled in this text box
Some respondents indicated things āon their endā (e.g., busyness, attention span), or that theyād have said yes to one or both of those questions for most authors rather than just for me in particular
Some respondents mentioned topics just not seeming relevant to their interests
Some respondents mentioned my posts being long, being rambly, or failing to have a summary
Some respondents mentioned they were already well-versed in the areas I was writing about and didnāt feel my posts were necessary for them
Q9: āDo you have any other feedback on specific things Iāve written, my general writing style, my topic choices, or anything else?ā
10ā21 respondents answered this
Several non-specific positive comments/āencouragements
Several positive or neutral comments on me having a lot of output
Several comments suggesting I should be more concise, use summaries more consistently, and/āor be clearer about what the point of what Iām writing is
Some comments indicating appreciation of my summaries, collections, and efforts to make ideas accessible
Some comments on my writing style and clarity being good
Some comments that my original research wasnāt very impressive
One comment that I seem to hung up on defining things precisely/āprescriptively
(I donāt actually endorse linguistic prescriptivism, and remember occasionally trying to make that explicit. But Iāll take this as useful data that Iāve sometimes accidentally given that impression, and try to adjust accordingly.)
Q10: āIf you would like to share your name, please do so below. But this is 100% voluntaryāyouāre not at all obliged to do so :)ā
6ā21 respondents gave their name/āusername
2 gave their email for if I wanted to follow-up
Some takeaways from all this
Responses were notably more positive than expected for some questions, and notably less positive for others
I donāt think this should notably change my bottom-line view of the overall quality and impact of my work to date
But it does make me a little less uncertain about that all-things-considered view, as I now have slightly more data that roughly supports it
In turn, this updates me towards being a little more confident that it makes sense for me to focus on pursuing an EA research career for now (rather than, e.g., switching to operations or civil service roles)
This is because Iām now slightly less worried that Iām being strongly influenced by overconfidence or motivated reasoning. (I already wanted to do research or writing before learning about EA.)
I should definitely more consistently include summaries, and/āor in other ways signal early and clearly what the point of a post is
I was already aiming to move in this direction, and had predicted responses would often mention this, but this has still given me an extra push
I should look out for ways in which I might appear linguistically prescriptive or overly focused on definitions/āprecision
I should more seriously consider moving more towards concision, even at the cost of precision, clarity, or comprehensiveness
Though Iām still not totally sold on that
Iām also aware that this shortform comment is not a great first step!
I should consider moving more towards concision, even at the cost of quantity/āspeed of output
With extra time on a given post, I could perhaps find ways to be more concise without sacrificing other valuable things
I should feel less like I āhave toā produce writings rapidly
This point is harder to explain briefly, so Iāll just scratch the surface here
I donāt actually expect this to substantially change my behaviours, as that feeling wasnāt the main reason for my large amount of output
But if my output slows for some other reason, I think Iāll now not feel (as) bad about that
People have found my summaries and collections very useful, and some people have found my original research not so useful/āimpressive
The ādirectionā of this effect is in line with my expectations, but the strength was surprising
Iāve updated towards more confidence that my summaries and (especially) my collections were valuable and worth making, and this may slightly increase the already-high chance that Iāll continue creating that sort of thing
But this is also slightly confusing, as my original research/āideas and/āor aptitude for future original research seems to have put me in good stead for various job and grant selection processes
And I donāt have indications that my summaries or collections helped there, though they may have
Much of my work to date may be less useful for more experienced/āengaged EAs than less experienced/āengaged EAs
This is in line with my sense that I was often trying to make ideas more accessible, make getting up to speed easier, etc.
There seemed to be a weak correlation between how recently something was posted and how often it was positively mentioned
This broadly aligns with trends from other data sources (e.g., researchers reaching out to me, upvotes)
This could suggest that:
my work is getting better
people are paying more attention to things written by me, regardless of their quality
people just remember the recent stuff more
Iād guess all three of those factors play some role
(I also have additional thoughts that are fuzzier or even less likely to be of interest to anyone other than me.)
[1] There are of course myriad reasons to not read into this data too much, including that:
itās from a sample of only 21 people
the sample was non-representative, and indeed self-selecting (so it may, for example, disproportionately represent people who like my work)
the responses may be biased towards not hurting my feelings
That said, I think I can still learn something from this data, especially given flaws in other data sources I have. (E.g., comments from people who choose to randomly and non-anonymously reach out to me may be even more positively biased.)
āPeople have found my summaries and collections very useful, and some people have found my original research not so useful/āimpressiveā
I havenāt read enough of your original research to know whether it applies in your case but just flagging that most original research has a much narrower target audience than the summaries/ācollections, so Iād expect fewer people to find it useful (and for a relatively broad summary to be biased against them).
That said, as you know, I think your summaries/ācollections are useful and underprovided.
Though I guess I suspect that, if the reason a person finds my original research not so useful is just because they arenāt the target audience, theyād be more likely to either not explicitly comment on it or to say something about it not seeming relevant to them. (Rather than making a generic comment about it not seeming useful.)
But I guess this seems less likely in cases where:
the person doesnāt realise that the key reason it wasnāt useful is that they werenāt the target audience, or
the person feels that what theyāre focused on is substantially more important than anything else (because then theyāll perceive āuseful to themā as meaning a very similar thing to āusefulā)
In any case, Iām definitely just taking this survey as providing weak (though useful) evidence, and combining it with various other sources of evidence.
Reflections on data from a survey about things Iāve written
I recently requested people take a survey on the quality/āimpact of things Iāve written. So far, 22 people have generously taken the survey. (Please add yourself to that tally!)
Here Iāll display summaries of the first 21 responses (I may update this later), and reflect on what I learned from this.[1]
I had also made predictions about what the survey results would be, to give myself some sort of ramshackle baseline to compare results against. I was going to share these predictions, then felt no one would be interested; but let me know if youād like me to add them in a comment.
For my thoughts on how worthwhile this was and whether other researchers/āorganisations should run similar surveys, see Should surveys about the quality/āimpact of research outputs be more common?
(Note that many of the things Iāve written were related to my work with Convergence Analysis, but my comments here reflect only my own opinions.)
The data
Q1:
Q2:
Q3:
Q4:
Q5: āIf you think anything Iāve written has affected your beliefs, please say what that thing was (either titles or roughly what the topic was), and/āor say how it affected your beliefs.ā
(I didnāt ask for permission to share peopleās comments, so, for this and the other comment questions, Iāll just highlight some recurring themes or seemingly noteworthy specifics.)
9ā21 respondents answered this
The writings people mentioned specifically were my collections and summaries of existing ideas/āwork (e.g., A central directory for open research questions), Database of existential risk estimates, Improving the future by influencing actorsā benevolence, intelligence, and power, and my comments on the Google doc of another person who wanted feedback.
Most responses seemed to indicate the shift in beliefs caused by my work was fairly small.
Q6:
Q7: āIf you think anything Iāve written has affected your decisions or plans, please say what that thing was (either titles or roughly what the topic was), and/āor say how it affected your decisions or plans.ā
5ā21 respondents answered this
One respondent mentioned a way in which something I wrote contributed meaningfully to an output of theirs which I think is quite valuable
One respondent indicated Some history topics it might be very valuable to investigate influenced them somewhat
Another indicated Improving the future by influencing actorsā benevolence, intelligence, and power might inform an important decision
There was one other small influence
Q8:
Q8, text box: āIf you answered āYesā to either of the above, could you say a bit about why?ā
15ā21 respondents filled in this text box
Some respondents indicated things āon their endā (e.g., busyness, attention span), or that theyād have said yes to one or both of those questions for most authors rather than just for me in particular
Some respondents mentioned topics just not seeming relevant to their interests
Some respondents mentioned my posts being long, being rambly, or failing to have a summary
Some respondents mentioned they were already well-versed in the areas I was writing about and didnāt feel my posts were necessary for them
Q9: āDo you have any other feedback on specific things Iāve written, my general writing style, my topic choices, or anything else?ā
10ā21 respondents answered this
Several non-specific positive comments/āencouragements
Several positive or neutral comments on me having a lot of output
Several comments suggesting I should be more concise, use summaries more consistently, and/āor be clearer about what the point of what Iām writing is
Some comments indicating appreciation of my summaries, collections, and efforts to make ideas accessible
Some comments on my writing style and clarity being good
Some comments that my original research wasnāt very impressive
One comment that I seem to hung up on defining things precisely/āprescriptively
(I donāt actually endorse linguistic prescriptivism, and remember occasionally trying to make that explicit. But Iāll take this as useful data that Iāve sometimes accidentally given that impression, and try to adjust accordingly.)
Q10: āIf you would like to share your name, please do so below. But this is 100% voluntaryāyouāre not at all obliged to do so :)ā
6ā21 respondents gave their name/āusername
2 gave their email for if I wanted to follow-up
Some takeaways from all this
Responses were notably more positive than expected for some questions, and notably less positive for others
I donāt think this should notably change my bottom-line view of the overall quality and impact of my work to date
But it does make me a little less uncertain about that all-things-considered view, as I now have slightly more data that roughly supports it
In turn, this updates me towards being a little more confident that it makes sense for me to focus on pursuing an EA research career for now (rather than, e.g., switching to operations or civil service roles)
This is because Iām now slightly less worried that Iām being strongly influenced by overconfidence or motivated reasoning. (I already wanted to do research or writing before learning about EA.)
I should definitely more consistently include summaries, and/āor in other ways signal early and clearly what the point of a post is
I was already aiming to move in this direction, and had predicted responses would often mention this, but this has still given me an extra push
I should look out for ways in which I might appear linguistically prescriptive or overly focused on definitions/āprecision
I should more seriously consider moving more towards concision, even at the cost of precision, clarity, or comprehensiveness
Though Iām still not totally sold on that
Iām also aware that this shortform comment is not a great first step!
I should consider moving more towards concision, even at the cost of quantity/āspeed of output
With extra time on a given post, I could perhaps find ways to be more concise without sacrificing other valuable things
I should feel less like I āhave toā produce writings rapidly
This point is harder to explain briefly, so Iāll just scratch the surface here
I donāt actually expect this to substantially change my behaviours, as that feeling wasnāt the main reason for my large amount of output
But if my output slows for some other reason, I think Iāll now not feel (as) bad about that
People have found my summaries and collections very useful, and some people have found my original research not so useful/āimpressive
The ādirectionā of this effect is in line with my expectations, but the strength was surprising
Iāve updated towards more confidence that my summaries and (especially) my collections were valuable and worth making, and this may slightly increase the already-high chance that Iāll continue creating that sort of thing
But this is also slightly confusing, as my original research/āideas and/āor aptitude for future original research seems to have put me in good stead for various job and grant selection processes
And I donāt have indications that my summaries or collections helped there, though they may have
Much of my work to date may be less useful for more experienced/āengaged EAs than less experienced/āengaged EAs
This is in line with my sense that I was often trying to make ideas more accessible, make getting up to speed easier, etc.
There seemed to be a weak correlation between how recently something was posted and how often it was positively mentioned
This broadly aligns with trends from other data sources (e.g., researchers reaching out to me, upvotes)
This could suggest that:
my work is getting better
people are paying more attention to things written by me, regardless of their quality
people just remember the recent stuff more
Iād guess all three of those factors play some role
(I also have additional thoughts that are fuzzier or even less likely to be of interest to anyone other than me.)
[1] There are of course myriad reasons to not read into this data too much, including that:
itās from a sample of only 21 people
the sample was non-representative, and indeed self-selecting (so it may, for example, disproportionately represent people who like my work)
the responses may be biased towards not hurting my feelings
That said, I think I can still learn something from this data, especially given flaws in other data sources I have. (E.g., comments from people who choose to randomly and non-anonymously reach out to me may be even more positively biased.)
If youāve made it this far, you may also be interested in the above-mentioned Should surveys about the quality/āimpact of research outputs be more common?
āPeople have found my summaries and collections very useful, and some people have found my original research not so useful/āimpressiveā
I havenāt read enough of your original research to know whether it applies in your case but just flagging that most original research has a much narrower target audience than the summaries/ācollections, so Iād expect fewer people to find it useful (and for a relatively broad summary to be biased against them).
That said, as you know, I think your summaries/ācollections are useful and underprovided.
Good point.
Though I guess I suspect that, if the reason a person finds my original research not so useful is just because they arenāt the target audience, theyād be more likely to either not explicitly comment on it or to say something about it not seeming relevant to them. (Rather than making a generic comment about it not seeming useful.)
But I guess this seems less likely in cases where:
the person doesnāt realise that the key reason it wasnāt useful is that they werenāt the target audience, or
the person feels that what theyāre focused on is substantially more important than anything else (because then theyāll perceive āuseful to themā as meaning a very similar thing to āusefulā)
In any case, Iām definitely just taking this survey as providing weak (though useful) evidence, and combining it with various other sources of evidence.
Seems reasonable