“In an intermediate step, we removed items that were not predictive of relevant EA outcome measures (these outcome measures include overall agreement with the key principles of EA, interest in learning more about EA, willingness to change one’s career in order to have more impact).”
“Notably, we found that some items with an obligation framing (e.g., “it would be wrong …”) yielded high variance and were particularly predictive. Remember that our goal was not to create items that together define effective altruism conceptually or philosophically, but rather to identify items that are predictive of our outcome measures. Thus, our inclusion of these obligation items does not mean that we think that effective altruism should be defined in terms of an obligation to help effectively.”
I think prioritizing items with predictive power is questionable from a psychometric perspective. Predictive power should not be the goal of measurement (McIntosh, 2007), and latent indicators should not be preferentially chosen because they optimize prediction (Smits et al., 2018). A valid indicator of a construct does not necessarily have good predictive power and invalid indicators often do. Even though you are not conceptually defining Effective Altruism, the indicators should conceptually match your theoretical beliefs about Expansive Altruism and Effectiveness-Focus. If feelings of obligation are constitutive of those constructs, great, but if not then you might want to toss them even if they have predictive value. (Though having a single item with an obligation focus would not be a bad idea, as the variance attributable to the obligation framing would be partialed out anyway when you load the item on the factor.)
This procedure risks attributing more predictive power to Expansive Altruism and Effectiveness-Focus than would be found with optimal indicators for these constructs. Accordingly, measurement validity can decrease, and the latent variables may be endogenous with respect to the outcome variables that you optimized the factor to predict. One potential source of evidence for this would be implausibly large associations between the factors and the outcomes. Other sources of evidence would include the Effectiveness and Expansiveness factors correlating with the disturbances of the outcome variables, and the Effectiveness and Expansiveness factors’ disturbances correlating with the common variance of the outcome measures. Using Study 2 data, I found evidence consistent with each of these possibilities by regressing a latent variable composed of the charitable giving tasks on the Expansive and Effectiveness factors (model and modification indices), as well as regressing a latent variable composed of the EA-interest items on the Expansive and Effectiveness factors (model and modification indices). (I also wrote up a more expository walk-through of these analyses here.)
A stronger way of testing whether predictive power has been overstated would be to examine the predictive validity of measures of similar constructs. The principle of care scale, for instance, could stand in for the Expansive Altruism scale.
If I were Reviewer 2, I would probably also nitpick on these much more minor issues:
“α = 0.80, CFI = .97.”
I would recommend reporting McDonald’s omega instead of Cronbach’s alpha unless you provide evidence that the factor loadings are all equal or nearly so. Even more emphatically I would recommend reporting the chi-square test instead of or in addition to CFI. No approximate fit index can (or ever could) measure how much discrepancy there is between the fitted model and true model (Greiff & Heene, 2017); local fit indices and careful judgment are the only real guides, though fallible in their own ways. Note also that the models I linked to above find much worse fit by all metrics, probably because I embedded the factors in a larger model. Testing the fit of a factor in isolation of the other variables is a weak test because in practice those factors will be entered into models with other variables (Hayduk & Glaser, 2000; e.g., the Expansive Altruism and Effectiveness-Focus factors will often be entered into the same model in future studies).
“We believe that expansive altruism is a particular type of altruism. In contrast to other forms of altruism, expansive altruism is driven by a desire to not just help proximate and emotionally salient individuals, but also distant individuals that may be less emotionally salient. A similar type of distinction is made by psychologist Paul Bloom in his book Against Empathy (2017), in which he contrasts empathy-driven altruism with, what he refers to as, “rational compassion”. Our studies support the notion that such a form of rational compassion exists. However, our studies don’t suggest that this sort of compassion entails a lack of empathy. To the contrary, we find that expansive altruism strongly correlates with empathy (i.e., the trait “Empathic Concern”; see below).”
Another good source that I sense is sometimes overlooked is Hoffman. There is also closely related older work on the universal care ethic, such as in The Altruistic Personality. One thing I am currently unclear about with the Expansive Altruism scale is whether it measures the belief that one ought to care about distant others (which might or might not cause a desire to help) or an actual desire to help (which may or may not be caused by a belief that one ought to help).
“And notably effectiveness-focus and instrumental harm are also psychologically different: they correlated only weakly (r = .34; see below).”
“Only weakly” is probably too strong. This correlation is large for individual differences research (Gignac et al., 2016).
“In an intermediate step, we removed items that were not predictive of relevant EA outcome measures (these outcome measures include overall agreement with the key principles of EA, interest in learning more about EA, willingness to change one’s career in order to have more impact).”
“Notably, we found that some items with an obligation framing (e.g., “it would be wrong …”) yielded high variance and were particularly predictive. Remember that our goal was not to create items that together define effective altruism conceptually or philosophically, but rather to identify items that are predictive of our outcome measures. Thus, our inclusion of these obligation items does not mean that we think that effective altruism should be defined in terms of an obligation to help effectively.”
I think prioritizing items with predictive power is questionable from a psychometric perspective. Predictive power should not be the goal of measurement (McIntosh, 2007), and latent indicators should not be preferentially chosen because they optimize prediction (Smits et al., 2018). A valid indicator of a construct does not necessarily have good predictive power and invalid indicators often do. Even though you are not conceptually defining Effective Altruism, the indicators should conceptually match your theoretical beliefs about Expansive Altruism and Effectiveness-Focus. If feelings of obligation are constitutive of those constructs, great, but if not then you might want to toss them even if they have predictive value. (Though having a single item with an obligation focus would not be a bad idea, as the variance attributable to the obligation framing would be partialed out anyway when you load the item on the factor.)
This procedure risks attributing more predictive power to Expansive Altruism and Effectiveness-Focus than would be found with optimal indicators for these constructs. Accordingly, measurement validity can decrease, and the latent variables may be endogenous with respect to the outcome variables that you optimized the factor to predict. One potential source of evidence for this would be implausibly large associations between the factors and the outcomes. Other sources of evidence would include the Effectiveness and Expansiveness factors correlating with the disturbances of the outcome variables, and the Effectiveness and Expansiveness factors’ disturbances correlating with the common variance of the outcome measures. Using Study 2 data, I found evidence consistent with each of these possibilities by regressing a latent variable composed of the charitable giving tasks on the Expansive and Effectiveness factors (model and modification indices), as well as regressing a latent variable composed of the EA-interest items on the Expansive and Effectiveness factors (model and modification indices). (I also wrote up a more expository walk-through of these analyses here.)
A stronger way of testing whether predictive power has been overstated would be to examine the predictive validity of measures of similar constructs. The principle of care scale, for instance, could stand in for the Expansive Altruism scale.
If I were Reviewer 2, I would probably also nitpick on these much more minor issues:
“α = 0.80, CFI = .97.”
I would recommend reporting McDonald’s omega instead of Cronbach’s alpha unless you provide evidence that the factor loadings are all equal or nearly so. Even more emphatically I would recommend reporting the chi-square test instead of or in addition to CFI. No approximate fit index can (or ever could) measure how much discrepancy there is between the fitted model and true model (Greiff & Heene, 2017); local fit indices and careful judgment are the only real guides, though fallible in their own ways. Note also that the models I linked to above find much worse fit by all metrics, probably because I embedded the factors in a larger model. Testing the fit of a factor in isolation of the other variables is a weak test because in practice those factors will be entered into models with other variables (Hayduk & Glaser, 2000; e.g., the Expansive Altruism and Effectiveness-Focus factors will often be entered into the same model in future studies).
“We believe that expansive altruism is a particular type of altruism. In contrast to other forms of altruism, expansive altruism is driven by a desire to not just help proximate and emotionally salient individuals, but also distant individuals that may be less emotionally salient. A similar type of distinction is made by psychologist Paul Bloom in his book Against Empathy (2017), in which he contrasts empathy-driven altruism with, what he refers to as, “rational compassion”. Our studies support the notion that such a form of rational compassion exists. However, our studies don’t suggest that this sort of compassion entails a lack of empathy. To the contrary, we find that expansive altruism strongly correlates with empathy (i.e., the trait “Empathic Concern”; see below).”
Another good source that I sense is sometimes overlooked is Hoffman. There is also closely related older work on the universal care ethic, such as in The Altruistic Personality. One thing I am currently unclear about with the Expansive Altruism scale is whether it measures the belief that one ought to care about distant others (which might or might not cause a desire to help) or an actual desire to help (which may or may not be caused by a belief that one ought to help).
“And notably effectiveness-focus and instrumental harm are also psychologically different: they correlated only weakly (r = .34; see below).”
“Only weakly” is probably too strong. This correlation is large for individual differences research (Gignac et al., 2016).