are you sure this isn’t just a function of the definition of highly engaged?
No, I think it it probably is partly explained by that.
For context for other readers: the highest level of engagement on the engagement scale is defined as “I am heavily involved in the effective altruism community, perhaps helping to lead an EA group or working at an EA-aligned organization. I make heavy use of the principles of effective altruism when I make decisions about my career or charitable donations.” The next highest category of engagement (“I’ve engaged extensively with effective altruism content (e.g. attending an EA Global conference, applying for career coaching, or organizing an EA meetup). I often consider the principles of effective altruism when I make decisions about my career or charitable donations.”) is also included in the “high engagement” group, when we apply a binary division between high and low engagement).
The definition of the highest level of engagement may skew against people who are earning to give, even if they are highly engaged (in the ordinary sense of the term), and donating a lot, because it includes the specific example of “perhaps helping to lead an EA group or working at an EA-aligned organization” (even though a high-donating earn-to-giver could judge that they counted as highly engaged anyway, they might think that the category doesn’t best fit them).
That said, I would expect most engaged, high-donating earning-to-givers to fit at least the second highest category, which would still count them as “highly engaged” in our binary analyses. Note that the next category down (the highest category in the “low engagement” binary division) is quite a lot weaker “I’ve engaged with multiple articles, videos, podcasts, discussions, or events on effective altruism (e.g. subscribing to the 80,000 Hours podcast or attending regular events at a local group).I sometimes consider the principles of effective altruism when I make decisions about my career or charitable donations.”
Given that, I don’t expect these definitions to dramatically skew results related to earning-to-give, so long as we’re using the binary division, but the use of these specific examples is probably a part of the issue.
The engagement scale (which we didn’t design ourselves, it was a requested question from 2019 onwards) definitely has some undesirable features. It’s double or triple-barrelled (i.e. it asks people about multiple distinct dimensions in parallel (e.g. how far and/or how often you consider EA principles and about different actions you could take)). And the characterisations of the different levels of the different dimensions are not clearly hierarchical and seem to jump between different dimensions (e.g. levels 2-4 specify engagement with different content, level 5 refers to engagement in groups or EA orgs). The ideal way to design the measures would be to have each measure specifying a distinct dimension (e.g. self-identified engagement, degree or frequency of considering EA principles, and engagement in different activities) and have each of these dimensions unambigiously hierarchical, and then see how these different dimensions are associated.
That said, I think the scale mostly works passably well for most purposes. When we examine the association between the self-report scale and different concrete measures and between different concrete measures themselves (2019, 2020), we find a fairly high level of consilience between different measures. So people who are less/more engaged on the scale are also less/more likely to engage in a suite of different activities. Granted, this is still a certain kind of EA engagement (namely engagement with EA content and activities), and so won’t capture some cases of people who are very dedicated and do high value work (e.g. perhaps a high powered policy maker or donor); but most cases where people have “engaged extensively” with EA content, should still be captured as high engagement, regardless of their current activities.
No, I think it it probably is partly explained by that.
For context for other readers: the highest level of engagement on the engagement scale is defined as “I am heavily involved in the effective altruism community, perhaps helping to lead an EA group or working at an EA-aligned organization. I make heavy use of the principles of effective altruism when I make decisions about my career or charitable donations.” The next highest category of engagement (“I’ve engaged extensively with effective altruism content (e.g. attending an EA Global conference, applying for career coaching, or organizing an EA meetup). I often consider the principles of effective altruism when I make decisions about my career or charitable donations.”) is also included in the “high engagement” group, when we apply a binary division between high and low engagement).
The definition of the highest level of engagement may skew against people who are earning to give, even if they are highly engaged (in the ordinary sense of the term), and donating a lot, because it includes the specific example of “perhaps helping to lead an EA group or working at an EA-aligned organization” (even though a high-donating earn-to-giver could judge that they counted as highly engaged anyway, they might think that the category doesn’t best fit them).
That said, I would expect most engaged, high-donating earning-to-givers to fit at least the second highest category, which would still count them as “highly engaged” in our binary analyses. Note that the next category down (the highest category in the “low engagement” binary division) is quite a lot weaker “I’ve engaged with multiple articles, videos, podcasts, discussions, or events on effective altruism (e.g. subscribing to the 80,000 Hours podcast or attending regular events at a local group). I sometimes consider the principles of effective altruism when I make decisions about my career or charitable donations.”
Given that, I don’t expect these definitions to dramatically skew results related to earning-to-give, so long as we’re using the binary division, but the use of these specific examples is probably a part of the issue.
The engagement scale (which we didn’t design ourselves, it was a requested question from 2019 onwards) definitely has some undesirable features. It’s double or triple-barrelled (i.e. it asks people about multiple distinct dimensions in parallel (e.g. how far and/or how often you consider EA principles and about different actions you could take)). And the characterisations of the different levels of the different dimensions are not clearly hierarchical and seem to jump between different dimensions (e.g. levels 2-4 specify engagement with different content, level 5 refers to engagement in groups or EA orgs). The ideal way to design the measures would be to have each measure specifying a distinct dimension (e.g. self-identified engagement, degree or frequency of considering EA principles, and engagement in different activities) and have each of these dimensions unambigiously hierarchical, and then see how these different dimensions are associated.
That said, I think the scale mostly works passably well for most purposes. When we examine the association between the self-report scale and different concrete measures and between different concrete measures themselves (2019, 2020), we find a fairly high level of consilience between different measures. So people who are less/more engaged on the scale are also less/more likely to engage in a suite of different activities. Granted, this is still a certain kind of EA engagement (namely engagement with EA content and activities), and so won’t capture some cases of people who are very dedicated and do high value work (e.g. perhaps a high powered policy maker or donor); but most cases where people have “engaged extensively” with EA content, should still be captured as high engagement, regardless of their current activities.