In addition to my other comment, I would add that there are two different things to distinguish:
‘More engaged people are more likely to say that EAG helped them get involved, because more engaged people are more likely to have attended EAG.’ [As I understand it, this is not your claim.]
I think this is broadly true.
But the results are still not circular, since you can engage with a factor, but not think it important for getting involved. As I noted in my other comment, many more highly engaged people have engaged with an EA book than an EA Group, but EA Groups trounce books in terms of importance for getting highly engaged EAs involved.
‘More engaged people are more likely to say that EAG helped them get involved, because the wording of the engagement scale includes attending EAG among the examples of Considerable engagement...’
I think this is very unlikely to be true.
The crux seems to me to be: if we were to change the wording of the engagement scale so that EAG/EA Groups (or all the concrete examples) were removed from the items, would this change these results.
I would predict that this would neither significantly change the pattern of low/high engagement respondents who have attended EAG/an EA Group, nor change the results in terms of what factors are important for getting involved in EA.
As noted in my other comment, I think there are a number of reasons to think this, not least, that attending an EA Group is not even mentioned among the high engagement categories in the wording of the question.
I think the fact that the engagement scale makes reference to specific activities as examples is worth bearing in mind (we’ve discussed this before, going back to the 2019 post where we introduced the scale, and the 2020 post, and more recently in the comments on the Demographics post).
You’re right that there could be an influence from ‘the scale mentioning “attending an EA Global conference” as an example in one of the higher engagement categories’ → people who attend EAG being more likely to be select (and so be counted as) higher engagement → higher engagement people being more likely to say EAG was important for them getting involved (because they are more likely to have attended EAG).
However, I think there a few reasons why the results are nevertheless not likely to be largely circular.
As implied above, this is somewhat indirect. The engagement measure refers to attending EA Global, but not whether EA Global was important for getting involved. So, in theory, it’s possible for people to attend EAG but not select it as important for getting them involved. If people who attended EAG didn’t report it being important for them getting involved, increasing the association between engagement and attending EAG would not help the association between engagement and reporting EAG being important.
Although EAG is mentioned as an example in the scale, it’s possible to be highly engaged but not attend EAG (around just over 45% of highly engaged respondents had not attended EAG, or to attend EAG but not be highly engaged (around 31.6% of EAG attendees). This is using 2020 data, since we did not ask about EAG attendance this year.
You mention “being a member of a group”, but “regularly attending events at a local group” is given as an example of Moderate engagement, which would fall below High engagement in the binary categorisation we use in these analyses.
Similarly, engaging with “articles, videos, podcasts, discussions, or events on effective altruism” and “subscribing to the 80,000 Hours podcast” are mentioned as examples in the same level of the engagement scale, but we do not observe comparably large bumps for these factors. [Edit] However, we do observe large effects for other factors which aren’t mentioned in the definition of the scales (e.g. personal contacts).
I’d also add that many more EAs (both highly engaged and less engaged, though more among the highly engaged than the less engaged) have undertaken activities such as read an EA book (2019 data) than have attended EAG or attended a local group. So, one might expect that even greater numbers in both of these groups would report that a book was important for getting them involved. But we don’t observe this, EA groups are still cited as important considerably more frequently than EA books, despite fewer EAs having encountered them.
As another illustration, we can look at the association between engaging in different activities related to EA and reporting that EAG is important for getting involved in EA. This just asks whether people have completed each activity (0 = no, 1=yes), so there’s no possible circularity in the wording.
As we can see, people who have engaged in each of the individual the activities are much more likely to report that EAG was important for them getting involved.
In addition, I created a simple ‘engagement_score’ based on the number of activities individuals had completed (0=1SD lower below the mean, 1=1SD above the mean’). We can see that the results for the engagement_score are almost identical to the results for high_engagement on the self-reported engagement scale (in black).
For more discussion of the associations between and validation of the engagement measures see our 2019 and 2020 reports.
If the definition of being more engaged includes going to EAG and being a member of a group, aren’t some of these results a bit circular?
In addition to my other comment, I would add that there are two different things to distinguish:
‘More engaged people are more likely to say that EAG helped them get involved, because more engaged people are more likely to have attended EAG.’ [As I understand it, this is not your claim.]
I think this is broadly true.
But the results are still not circular, since you can engage with a factor, but not think it important for getting involved. As I noted in my other comment, many more highly engaged people have engaged with an EA book than an EA Group, but EA Groups trounce books in terms of importance for getting highly engaged EAs involved.
‘More engaged people are more likely to say that EAG helped them get involved, because the wording of the engagement scale includes attending EAG among the examples of Considerable engagement...’
I think this is very unlikely to be true.
The crux seems to me to be: if we were to change the wording of the engagement scale so that EAG/EA Groups (or all the concrete examples) were removed from the items, would this change these results.
I would predict that this would neither significantly change the pattern of low/high engagement respondents who have attended EAG/an EA Group, nor change the results in terms of what factors are important for getting involved in EA.
As noted in my other comment, I think there are a number of reasons to think this, not least, that attending an EA Group is not even mentioned among the high engagement categories in the wording of the question.
Thanks for the comment.
I think the fact that the engagement scale makes reference to specific activities as examples is worth bearing in mind (we’ve discussed this before, going back to the 2019 post where we introduced the scale, and the 2020 post, and more recently in the comments on the Demographics post).
You’re right that there could be an influence from ‘the scale mentioning “attending an EA Global conference” as an example in one of the higher engagement categories’ → people who attend EAG being more likely to be select (and so be counted as) higher engagement → higher engagement people being more likely to say EAG was important for them getting involved (because they are more likely to have attended EAG).
However, I think there a few reasons why the results are nevertheless not likely to be largely circular.
As implied above, this is somewhat indirect. The engagement measure refers to attending EA Global, but not whether EA Global was important for getting involved. So, in theory, it’s possible for people to attend EAG but not select it as important for getting them involved. If people who attended EAG didn’t report it being important for them getting involved, increasing the association between engagement and attending EAG would not help the association between engagement and reporting EAG being important.
Although EAG is mentioned as an example in the scale, it’s possible to be highly engaged but not attend EAG (around just over 45% of highly engaged respondents had not attended EAG, or to attend EAG but not be highly engaged (around 31.6% of EAG attendees). This is using 2020 data, since we did not ask about EAG attendance this year.
You mention “being a member of a group”, but “regularly attending events at a local group” is given as an example of Moderate engagement, which would fall below High engagement in the binary categorisation we use in these analyses.
Similarly, engaging with “articles, videos, podcasts, discussions, or events on effective altruism” and “subscribing to the 80,000 Hours podcast” are mentioned as examples in the same level of the engagement scale, but we do not observe comparably large bumps for these factors. [Edit] However, we do observe large effects for other factors which aren’t mentioned in the definition of the scales (e.g. personal contacts).
I’d also add that many more EAs (both highly engaged and less engaged, though more among the highly engaged than the less engaged) have undertaken activities such as read an EA book (2019 data) than have attended EAG or attended a local group. So, one might expect that even greater numbers in both of these groups would report that a book was important for getting them involved. But we don’t observe this, EA groups are still cited as important considerably more frequently than EA books, despite fewer EAs having encountered them.
As another illustration, we can look at the association between engaging in different activities related to EA and reporting that EAG is important for getting involved in EA. This just asks whether people have completed each activity (0 = no, 1=yes), so there’s no possible circularity in the wording.
As we can see, people who have engaged in each of the individual the activities are much more likely to report that EAG was important for them getting involved.
In addition, I created a simple ‘engagement_score’ based on the number of activities individuals had completed (0=1SD lower below the mean, 1=1SD above the mean’). We can see that the results for the engagement_score are almost identical to the results for high_engagement on the self-reported engagement scale (in black).
For more discussion of the associations between and validation of the engagement measures see our 2019 and 2020 reports.