Hey David and everyone else involved, thank you for the analysis! This is useful data for us.
A quick request for next year: it would be great to keep working on the categories to get fewer ‘other’ responses and reduce overlap.
Approximately 19% of the open comment responses mentioned Podcasts (typically the Sam Harris or Joe Rogan podcasts), 15% mentioned Books, 10% mentioned Articles (online or in a newspaper), 9% mentioned a university or school Course, 7% a Blog, and 4% a Talk. 5% and 6% referred to an EA org that had already been included in the fixed response options (e.g. GWWC) and to a Personal Contact respectively.
The Sam Harris and Joe Rogan podcasts were done by Will as part of the promotion campaign for DGB while he was working at CEA/80k so could arguably be coded as DGB/CEA/80k. Presumably some of the books / articles / talks are also other materials produced by the organisations or press coverage they sought out—does that seem right?
Likewise, maybe ‘search’ and ‘facebook’ should be removed as categories, because they’re channels you use to find the other content listed. Presumably everyone who found out about EA through ‘facebook’ likely saw a post by a friend, so should be a personal referral, or saw a post by one of the orgs, so should be coded as an org.
We will definitely keep refining the categories year-by-year depending on which seem more or less significant.
“The Sam Harris and Joe Rogan podcasts were done by Will as part of the promotion campaign for DGB while he was working at CEA/80k so could arguably be coded as DGB/CEA/80k”
I entirely agree that Will’s podcasts can be counted as CEA/80K/DGB/Will in terms of assigning credit.
I’m not sure what that tells us about how we ought to design the survey though.
In terms of coding open comment responses: as I noted, how to classify people’s open comment answers is somewhat fuzzy, many comments ambiguously fit multiple different options and those numbers can’t be directly compared to the fixed category responses. Your podcast case is a perfect example since, as you note, if someone writes in “Joe Rogan Podcast” it could be classified as ‘DGB’, ‘CEA’, ‘80K’, ‘Podcast’ or ‘Will’ (or depending on our interests, we could code it as ‘mass media outreach’ or ‘online’ and so on). I could go back through the open comments and try to code things as CEA or 80K related (per your specification), but it will be a pretty vague and heterogeneous category with a lot of edge cases like these, that will make it hard to interpret. Explicit references to existing EA orgs or Will etc. were coded as such.
Also, crucially, a lot of the comments weren’t specific enough to code in that way. In the specific case of Sam Harris podcasts, for example, at least one person mentioned the Peter Singer episode (no-one explicitly mentioned Eliezer’s AI episode that I saw, but in principle some people might have first heard through that one), which means technically we can’t code every reference to Sam Harris’ podcast as Will’s- though I think it’s entirely reasonably of you to assume that many of the non-specific comments were referring to Will’s.
In terms of refining the categories: note that DGB and 80K were already available as options and respondents chose to select “Other” and write in a response anyway. It would be difficult to change the fixed categories, in such a way that people who would otherwise write “Joe Rogan podcast” would instead select “CEA”/“80K” etc. as a category. For one thing, people need to know (and remember) that, when they hear Will on a podcast, this should be counted as 80K or CEA or as part of the marketing for DGB or whichever category. They may also just feel that ‘Other: Podcast’ better captures their case than DGB or 80K. I think it would be difficult to specify all the things that could reasonably be counted as CEA’s work in a given fixed option (e.g. ‘80,000 Hours, inc. Doing Good Better, 80K Podcasts, Will MacAskill podcasts etc.’), but we’ll continue reviewing the results and trying to think of the best options.
I agree that in light of this year’s results, including Podcast as its own category next year may well make sense. We could also split out Book/Article etc. into multiple options. Though doing this might actually give us less information, in terms of attributing responses to CEA/80K/Will etc., as if people just select ‘Podcast’ as a fixed response, rather than writing in an ‘Other’ response, we won’t know which Podcast it was. We could include/require an open comment to specify further in addition to fixed responses, but requiring open comment would be significantly more onerous for respondents and might reduce response rate, so it’s a tricky balance. Alternatively we could include more fixed options (Podcast: Will; Podcast: 80K/Rob Wiblin; Podcast: Other, and so on), and ditto for other specific books, websites and so on (so far, Doing Good Better is the only one we’ve extended this treatment to), but of course that might make the question too unwieldy.
“Presumably some of the books / articles / talks are also other materials produced by the organisations or press coverage they sought out—does that seem right”
I definitely agree this is right in terms of assigning credit: no doubt EA orgs such as CEA should claim a lot of credit for vicariously bringing about lots of other outcomes (e.g. lots of Personal Contacts are presumably influenced by the prior work of EA orgs), but again, I don’t think there’s a way to capture that in people’s First Heard responses. And again, when open responses explicitly referred to an EA org, then it was coded accordingly, as well as coding it as ‘Book’ or ‘Article.’.
“Likewise, maybe ‘search’ and ‘facebook’ should be removed as categories, because they’re channels you use to find the other content listed. Presumably everyone who found out about EA through ‘facebook’ likely saw a post by a friend, so should be a personal referral, or saw a post by one of the orgs, so should be coded as an org.”
I think it’s pretty plausible that they should be removed next year (given the relatively small numbers attached to them). Although it doesn’t necessarily follow that the answers could neatly be split off into “personal contact” or “an EA org” because people may remember that they saw something on Facebook, but not which org was responsible (a lot of comments were pretty vague like this): so getting rid of “search” and “facebook” would probably mean a lot more “Other” responses.
We’d certainly be open to including this website if you are particularly interested in it (although then there’s a question of which other specific EA websites should or shouldn’t be included ). For what it’s worth we didn’t receive a single “Other” response referring to it, that I saw, but maybe people classified this as “Search” or something else.
Hey David and everyone else involved, thank you for the analysis! This is useful data for us.
A quick request for next year: it would be great to keep working on the categories to get fewer ‘other’ responses and reduce overlap.
The Sam Harris and Joe Rogan podcasts were done by Will as part of the promotion campaign for DGB while he was working at CEA/80k so could arguably be coded as DGB/CEA/80k. Presumably some of the books / articles / talks are also other materials produced by the organisations or press coverage they sought out—does that seem right?
Likewise, maybe ‘search’ and ‘facebook’ should be removed as categories, because they’re channels you use to find the other content listed. Presumably everyone who found out about EA through ‘facebook’ likely saw a post by a friend, so should be a personal referral, or saw a post by one of the orgs, so should be coded as an org.
I’m also surprised to see https://www.effectivealtruism.org/ isn’t listed—do you know what happened there?
Hi Ben,
Thanks for the comment!
We will definitely keep refining the categories year-by-year depending on which seem more or less significant.
I entirely agree that Will’s podcasts can be counted as CEA/80K/DGB/Will in terms of assigning credit.
I’m not sure what that tells us about how we ought to design the survey though. In terms of coding open comment responses: as I noted, how to classify people’s open comment answers is somewhat fuzzy, many comments ambiguously fit multiple different options and those numbers can’t be directly compared to the fixed category responses. Your podcast case is a perfect example since, as you note, if someone writes in “Joe Rogan Podcast” it could be classified as ‘DGB’, ‘CEA’, ‘80K’, ‘Podcast’ or ‘Will’ (or depending on our interests, we could code it as ‘mass media outreach’ or ‘online’ and so on). I could go back through the open comments and try to code things as CEA or 80K related (per your specification), but it will be a pretty vague and heterogeneous category with a lot of edge cases like these, that will make it hard to interpret. Explicit references to existing EA orgs or Will etc. were coded as such.
Also, crucially, a lot of the comments weren’t specific enough to code in that way. In the specific case of Sam Harris podcasts, for example, at least one person mentioned the Peter Singer episode (no-one explicitly mentioned Eliezer’s AI episode that I saw, but in principle some people might have first heard through that one), which means technically we can’t code every reference to Sam Harris’ podcast as Will’s- though I think it’s entirely reasonably of you to assume that many of the non-specific comments were referring to Will’s. In terms of refining the categories: note that DGB and 80K were already available as options and respondents chose to select “Other” and write in a response anyway. It would be difficult to change the fixed categories, in such a way that people who would otherwise write “Joe Rogan podcast” would instead select “CEA”/“80K” etc. as a category. For one thing, people need to know (and remember) that, when they hear Will on a podcast, this should be counted as 80K or CEA or as part of the marketing for DGB or whichever category. They may also just feel that ‘Other: Podcast’ better captures their case than DGB or 80K. I think it would be difficult to specify all the things that could reasonably be counted as CEA’s work in a given fixed option (e.g. ‘80,000 Hours, inc. Doing Good Better, 80K Podcasts, Will MacAskill podcasts etc.’), but we’ll continue reviewing the results and trying to think of the best options.
I agree that in light of this year’s results, including Podcast as its own category next year may well make sense. We could also split out Book/Article etc. into multiple options. Though doing this might actually give us less information, in terms of attributing responses to CEA/80K/Will etc., as if people just select ‘Podcast’ as a fixed response, rather than writing in an ‘Other’ response, we won’t know which Podcast it was. We could include/require an open comment to specify further in addition to fixed responses, but requiring open comment would be significantly more onerous for respondents and might reduce response rate, so it’s a tricky balance. Alternatively we could include more fixed options (Podcast: Will; Podcast: 80K/Rob Wiblin; Podcast: Other, and so on), and ditto for other specific books, websites and so on (so far, Doing Good Better is the only one we’ve extended this treatment to), but of course that might make the question too unwieldy.
I definitely agree this is right in terms of assigning credit: no doubt EA orgs such as CEA should claim a lot of credit for vicariously bringing about lots of other outcomes (e.g. lots of Personal Contacts are presumably influenced by the prior work of EA orgs), but again, I don’t think there’s a way to capture that in people’s First Heard responses. And again, when open responses explicitly referred to an EA org, then it was coded accordingly, as well as coding it as ‘Book’ or ‘Article.’.
I think it’s pretty plausible that they should be removed next year (given the relatively small numbers attached to them). Although it doesn’t necessarily follow that the answers could neatly be split off into “personal contact” or “an EA org” because people may remember that they saw something on Facebook, but not which org was responsible (a lot of comments were pretty vague like this): so getting rid of “search” and “facebook” would probably mean a lot more “Other” responses.
We’d certainly be open to including this website if you are particularly interested in it (although then there’s a question of which other specific EA websites should or shouldn’t be included ). For what it’s worth we didn’t receive a single “Other” response referring to it, that I saw, but maybe people classified this as “Search” or something else.
That makes sense. I agree none of this is simple.