This anonymous essay was submitted to Open Philanthropy’s Cause Exploration Prizes contest and posted to the Forum with the authors’ permission.
If you’re seeing this in summer 2022, we’ll be posting many submissions in a short period. If you want to stop seeing them so often, apply a filter for the appropriate tag!
TL;DR: Civic engagement – the extent to which a person feels involved with their community and participates in activities to benefit the wider society—appears to be a predictor of positive well-being effects that may be overlooked when solely considering health and/or income.
Overview
This worldview investigation is written in response to Open Philanthropy’s call for proposals of a new metric to measure the non-health, non-pecuniary benefits of their Global Health and Wellbeing grantmaking efforts.
In the report, I propose two distinct measures for assessing an initiative’s impact on civic engagement, i.e. the practices and attitudes of involvement that contribute to a healthy democratic society. ‘Estimated Engagement’ provides a metric to approximate an intervention’s effects on community participation in socio-political activities, whilst the ‘Net Involvement Score’, asks citizens to self-report their sense of engagement at a local level and desire to positively impact the wider society.
However, it should be noted that this summary is the product of a superficial investigation into the social science and theory surrounding civic engagement. More in-depth research would be required to ascertain how the measures could be best implemented in practice and to validate their use in informing Open Phil’s evaluations.
Civic Engagement as an index
What is civic engagement?
Civic engagement can be defined as ‘individual and collective actions designed to identify and address issues of public concern’[1] or, more passively, as ‘a set of practices and attitudes of involvement in social and political life that converge to increase the health of a democratic society’[2]. This can range from involvement with community organisations and institutions to participation in elections and volunteering—essentially encompassing any activity which promotes quality of life in a community through either political or non-political processes.
Four distinct constructs have been identified in the literature as being crucial for civic engagement:[3]
Civic Action—participation in activities such as volunteering or service-learning
Civic Commitment—the willingness to make positive contributions to society
Civic Skills—the ability to be involved in civil society, politics and democracy
Social Cohesion – a sense of reciprocity, trust and bonding with others
How does civic engagement contribute to global health and wellbeing?
A sense of engagement has been found to strongly correlate with citizens’ subjective well-being.[4] This is believed to be mediated by a greater sense of purpose and meaning in life, alongside increased feelings of self-efficacy.[5] Conversely, disengaged citizens frequently report mental health challenges, loneliness and isolation. The relationship between civic engagement and overall life satisfaction appears to be reciprocal, in this way engagement can be viewed as both a proxy for and a predictor of wellbeing.
Civic participation is also purported to facilitate the accumulation of social capital within communities, promoting equality whilst reducing feelings of apathy and alienation. The creation of such social and political bonds may also help to increase responsiveness to policy-making, giving countries’ governments a greater degree of legitimacy.[6]
Youth engagement, in particular, has been cited as a crucial contributor to the maintenance of strong and stable democracies. When young people are given the chance to meaningfully engage in the decision-making process it not only appears to strengthen social cohesion but also empowers them to challenge incidences of discrimination, abuse and injustice.
What are the wider impacts of disengaged citizens?
More broadly, increasing civic engagement could have downstream benefits on various other global health issues and even mitigate existential risks. It has been suggested that citizens’ sense of involvement has been historically overlooked when it comes to evaluating threats to global security.[7] Reports forecasting the potential impacts of geopolitical, technological, economic, and environmental trends rarely examine their effects on civic participation and community engagement. However, this appears to be an oversight given the possible risks posed by disengagement to a healthy and functioning society.
In 2021, for the first time, youth disillusionment was named a global risk by the World Economic Forum. They also observed that the prevalence of this phenomenon worsened during the recent pandemic.[8] This sentiment is not only being expressed by the 50% of youth living in dictatorships and countries with authoritarian regimes but also those residing in westernised, supposedly democratic, nations. For example, in France between 30-40% of young people are thought to have abstained from voting in the last election, bolstering claims that they have lost faith in their ability to influence the political system.
The pervasive discontentment about the state of the world that this generation is currently reporting could plausibly be reduced by increasing their sense of engagement. Helping young people feel more invested in societal and democratic health would appear especially important given that the leaders of the future are undoubted among them. Ultimately, these individuals will likely face decisions that could influence the long-term future of humanity, providing a strong argument for initiatives which promote the development of sound moral values.
In addition, having a high proportion of disengaged citizens could conceivably lead to some or all of the following problems:
Widespread social isolation and exclusion. This is already being observed to a worrying extent in some cultures e.g. the Japanese hikikomori
Students dropping out of school or further education earlier
Increased rates of crime and unemployment
Higher proportions of citizens experiencing burnout and mental health issues
Greater polarisation and civil unrest or, even worse, a total collapse of stable democracy
Increased susceptibility of vulnerable citizens to radicalisation and committing acts of atrocity e.g. terrorist attacks and school shootings
What might measures of civic engagement reveal that are not captured in current evaluations of pecuniary and health effects?
Since greater civic engagement is associated with higher incomes and improved health, it could be argued that this would negate the need to measure the state separately. However, research has shown that wealth and disability status do not necessarily correlate strongly with increased subjective wellbeing. Also, engagement can improve citizens’ lives in other ways not captured by these metrics, primarily through increasing individuals’ sense of connection and purpose.
When it comes to mental health, for example, the negative consequences of psychological disorders may be poorly captured by DALYs due to underreporting of these conditions, especially in LMICs.[9] This is in addition to the fact that the metric tends to weight years lost to life (YLL) more strongly than those lost to disabilities (YLD).[10] DALYs also fail to capture the downstream impact on the lives of direct beneficiaries’ family and friends e.g. the decreased stress placed on the parents of a child who is protected from malaria. These could plausibly be picked up in measurements of civic engagement which gives a better indication of life satisfaction and general wellbeing in the community.
I would also tentatively suggest that measuring engagement could also prove highly applicable to grant applications in some of Open Phil’s new focus areas, such as Effective Altruism Community Building, Global Health Policy and Criminal Justice/Land Use Reform.
How can engagement be measured?
Researchers in the social sciences posit that civic engagement is a difficult construct to measure since it results from the amalgamation of various contextual factors. Although some multidimensional assessment tools, such as the ‘Civic Engagement Scale’[11] and the United States ‘Civic Engagement Supplement to the Current Population Survey’,[12] do exist, these tests are poorly validated, require a significant investment of time to conduct and to analyse once they are complete.
In regards to grantmaking decisions, where calculations are rough and based on approximate metrics, it would appear valuable to develop a measure that can be calculated more rapidly to expedite the process. With the caveat that it could require a slightly reductionistic approach, it may therefore be worthwhile to consider how engagement is evaluated in time-pressed, less academically-rigorous settings:
Economic indicators: Voter turnout
The OECD better life index suggests that voter participation[13] is the most effective way to measure civic engagement for a variety of reasons, including high data quality and cross-country comparability. It cites the benefits of improving this figure as an increased likelihood that the political system represents the views of a majority of the population
However, voter turnout is by no means perfect the perfect metric, given that civic participation encompasses a wider range of issues than just political engagement. Differences in electoral systems e.g. whether or not it is compulsory to vote in a specific country, can also significantly influence the measure and it is of little to no use when it comes to measuring engagement amongst children and young people under the age of 18.
Business indicators: The Net Promotor Score
Looking to other industries, such as the measurement of engagement in corporate settings, it seems plausible that there may be transferable insights from this sector. In market research, for example, customer engagement is regularly assessed using a metric known as the Net Promoter Score, or NPS. This tool facilitates the rapid evaluation of an individual’s brand loyalty and consumer satisfaction. Similarly, HR departments use an adapted version of the test (the eNPS) to measure employees’ sense of engagement in the workplace and, more recently, it has even been adopted in healthcare settings as a way to assess and improve the patient experience.
The net promoter score is a subjective measure that asks individuals to rate the likelihood that they would recommend an organisation/service/experience to a family member or friend. Those who answer with a 9 or 10 are classed as ‘promoters’, whilst those scoring 6 or below are described as ‘detractors’. 7 and 8s are discarded as ‘passive’ consumers.
This can be summarised as follows:
NPS = % promoters - % detractors (expressed as a single figure from – 100 to 100)
Nevertheless, despite its popularity, the NPS score almost undoubtedly has its limitations. For example, the ease of assessment comes at the expense of detailed insights into the factors driving the trends in engagement. Additionally, the idea that a single figure could provide a comprehensive picture of customer behaviour has been approached with a high degree of scepticism by critics of the metric. These individuals have also called into question the accuracy of the values used to classify promoters, detractors and passive consumers, alongside its failure to draw a distinction between intention and observed behaviour.[14][15]
Estimated Engagement and Net Involvement Scores (NIS)
Estimated Engagement (EE)
Given the aforementioned recommendations from the OECD to base measurements of civic engagement in a community on concrete metrics and empirical evidence, a simple way to approximate the effect of an intervention on beneficiaries’ engagement could be to calculate the percentage change in one or more of these.
For example, the percentage of adults voting in an election could be combined with the percentage of youth who are in school, further education or permanent employment to approximate the proportion of engaged citizens in a given population. Whilst this would not be fully representative of the attitudes and behaviours which comprise full community engagement, it would give a rough guide of the extent to which people are participating in society and could perhaps be enhanced by factoring in unemployment or crime rates.
Net Involvement Score (NIS)
For a more detailed and reliable insight into an intervention’s effect on civic engagement, I propose the use of measurement based loosely on the NPS. Although this would, of course, be a subjective measurement, it would likely be sufficiently representative to help inform Open Phil’s grantmaking efforts whilst remaining feasible in terms of data collection. The simplicity of such a has the appeal of speed and accessibility, generating easily obtainable results which could be used to compare the engagement of people from a wide range of backgrounds and culturally diverse communities.
In practice, the scoring process could be as straightforward as asking individuals to self-report their sense of engagement by having them rate a) their feelings of connection to their local community and b) desire to participate in activities that positively impact the wider society, capturing data on both civic attitudes and behaviour.
Taking into account the concerns of the previously alluded to critics of the NPS, it may also be appropriate to make slight modifications to the current scoring system. A brief scan of various studies evaluating the usefulness of the NPS[16] confirmed my a priori beliefs about people’s proclivity to rate neutral experiences as a 7. Previous research suggests that the NPS both slightly underestimates and overestimates the numbers of promoters and detractors, respectively. These observations prompt my suggestion that it would be more accurate to lower the bar slightly when evaluating this metric, classing citizens reporting average involvement scores of 8, 9 or 10 as ‘engaged’, with those scoring > 5 defined as ‘disengaged’ citizens.
The net involvement score (NIS) could then be determined using the following equation:
NIS = % of engaged citizens - % of disengaged citizens
EE and NIS in action: Hypothetical case studies
Applying these metrics to two hypothetical grantmaking applications, I have attempted to demonstrate how they might be used to measure civic engagement in a real-world setting:
Grant application 1 – Grandad’s Chair
Cause Area: Global Health and Development
An initiative to target male suicide in agricultural communities which involves healthcare workers training elders to deliver mental health first aid and basic psychiatric care to younger members of their communities
Metrics:
Increased DALYS?: 2.2 per 100,000 individuals
Increased income?: N/A
Estimated Engagement: +25%
In this made-up scenario, measuring DALYS alone would provide some insight into the effectiveness of the intervention, primarily resulting from a decrease in direct deaths from suicide. However, as mentioned previously, the increased DALYs are likely to be an underrepresentation of the actual health and well-being effects as a consequence of underreporting of years lost due to disabilities, such as anxiety or depression.
The estimated engagement score here has been determined by evaluating the percentage increase in voter turnout, adjusted to factor in significantly decreased crime rates in the region. Had this figure not been included, this grant may have been rejected in favour of another which led to a greater apparent increase in DALYs but which had little to no effect on beneficiaries’ psychological wellbeing or community engagement.
Grant application 2 – MediMent
Cause Area:Effective Altruism Community Growth—Global Health and Wellbeing
A UK-based organisation which provides aspiring doctors from disadvantaged backgrounds with work experience opportunities and access to mentoring
Metrics:
Increased DALYS?: N/A
Increased income?: Approximately £20,000/year
Net Involvement Score: +70 points (from −10 to 60)
In this hypothetical example, the beneficiaries would appear to experience only a modest salary increase when considering the initiative’s pecuniary and health benefits. However, the NIS score provides a much more compelling case for supporting the grant application. It demonstrates the significant positive impact that participating in the scheme could have on beneficiaries’ sense of connection/civic engagement and once again illustrates how DALYs might fail to faithfully represent wider-ranging effects on wellbeing.
Key questions and uncertainties
Nevertheless, some key questions and uncertainties remain and should be taken into account when evaluating the usefulness of these metrics:
1. To what extent are civic engagement and well-being linked?
Whilst there is a clear association between increased engagement and subjective wellbeing, the strength and direction of this relationship are not as clear in the social science literature. At present, the prevailing narrative is that the relationship is somewhat bidirectional, i.e. people who feel more satisfied in life have a greater tendency toward societal involvement which also has benefits on citizens’ overall well-being. However, the observed improvements could plausibly be influenced by another contributing factor and more in-depth studies
2. How would we weigh the benefits of increasing income or extending lives against those of improving engagement?
When comparing two distinct initiatives where a change in engagement is the only significant distinguishing factor, it might also be useful to consider how we would compare its value to the benefits of increased health or income. When it comes to community involvement, what would be a suitable ‘bar’ for impact? Additionally, when evaluating causes targeting different populations, should a 10% increase in voter participation be viewed as having equivalent effects on subjective wellbeing as a 10% increase in young people attending school or engaged in employment?
3. Should a metric to measure civic engagement be considered not as a concrete figure but rather as a decision-making ‘compass’?
Studies comparing results obtained using EE and NIS scores to those obtained through more in-depth surveys would need to be performed to ascertain the validity of the measures. That being said, there is a nontrivial likelihood that the best approach could be to view these metrics not as definitive measures of engagement but rather indicators of our best judgement of an intervention’s effects. Whilst this would admittedly preclude a ‘cost per dollar’ evaluation, this may turn out not to be a feasible goal when assessing a psychological state with a high degree of subjectivity. In this way, the figures could help guide grantmakers without concern about whether they are sufficiently representative to be used in their BOTEC calculations.
Summary
In summary, it seems highly plausible that having reliable metrics to measure a psychological state like civic engagement would be valuable, adding both colour and depth to Open Phil’s grantmaking decisions. I believe that figures such as estimated engagement or net involvement scores could be a reasonable way of approximating this. Nevertheless, several questions remain about exactly how this would be achieved in practice and further investigation of this area appears to be warranted.
[6] Ginwright, S. 2011. “Hope, Healing, and Care Pushing the Boundaries of Civic Engagement for African American Youth.” Liberal Education 97, no. 2: 34- 39.
Ginwright, S. 2011. “Hope, Healing, and Care Pushing the Boundaries of Civic Engagement for African American Youth.” Liberal Education 97, no. 2: 34- 39.
[Cause Exploration Prizes] Measuring civic engagement—an important yet undervalued well-being metric?
This anonymous essay was submitted to Open Philanthropy’s Cause Exploration Prizes contest and posted to the Forum with the authors’ permission.
If you’re seeing this in summer 2022, we’ll be posting many submissions in a short period. If you want to stop seeing them so often, apply a filter for the appropriate tag!
TL;DR: Civic engagement – the extent to which a person feels involved with their community and participates in activities to benefit the wider society—appears to be a predictor of positive well-being effects that may be overlooked when solely considering health and/or income.
Overview
This worldview investigation is written in response to Open Philanthropy’s call for proposals of a new metric to measure the non-health, non-pecuniary benefits of their Global Health and Wellbeing grantmaking efforts.
In the report, I propose two distinct measures for assessing an initiative’s impact on civic engagement, i.e. the practices and attitudes of involvement that contribute to a healthy democratic society. ‘Estimated Engagement’ provides a metric to approximate an intervention’s effects on community participation in socio-political activities, whilst the ‘Net Involvement Score’, asks citizens to self-report their sense of engagement at a local level and desire to positively impact the wider society.
However, it should be noted that this summary is the product of a superficial investigation into the social science and theory surrounding civic engagement. More in-depth research would be required to ascertain how the measures could be best implemented in practice and to validate their use in informing Open Phil’s evaluations.
Civic Engagement as an index
What is civic engagement?
Civic engagement can be defined as ‘individual and collective actions designed to identify and address issues of public concern’[1] or, more passively, as ‘a set of practices and attitudes of involvement in social and political life that converge to increase the health of a democratic society’[2]. This can range from involvement with community organisations and institutions to participation in elections and volunteering—essentially encompassing any activity which promotes quality of life in a community through either political or non-political processes.
Four distinct constructs have been identified in the literature as being crucial for civic engagement:[3]
Civic Action—participation in activities such as volunteering or service-learning
Civic Commitment—the willingness to make positive contributions to society
Civic Skills—the ability to be involved in civil society, politics and democracy
Social Cohesion – a sense of reciprocity, trust and bonding with others
How does civic engagement contribute to global health and wellbeing?
A sense of engagement has been found to strongly correlate with citizens’ subjective well-being.[4] This is believed to be mediated by a greater sense of purpose and meaning in life, alongside increased feelings of self-efficacy.[5] Conversely, disengaged citizens frequently report mental health challenges, loneliness and isolation. The relationship between civic engagement and overall life satisfaction appears to be reciprocal, in this way engagement can be viewed as both a proxy for and a predictor of wellbeing.
Civic participation is also purported to facilitate the accumulation of social capital within communities, promoting equality whilst reducing feelings of apathy and alienation. The creation of such social and political bonds may also help to increase responsiveness to policy-making, giving countries’ governments a greater degree of legitimacy.[6]
Youth engagement, in particular, has been cited as a crucial contributor to the maintenance of strong and stable democracies. When young people are given the chance to meaningfully engage in the decision-making process it not only appears to strengthen social cohesion but also empowers them to challenge incidences of discrimination, abuse and injustice.
What are the wider impacts of disengaged citizens?
More broadly, increasing civic engagement could have downstream benefits on various other global health issues and even mitigate existential risks. It has been suggested that citizens’ sense of involvement has been historically overlooked when it comes to evaluating threats to global security.[7] Reports forecasting the potential impacts of geopolitical, technological, economic, and environmental trends rarely examine their effects on civic participation and community engagement. However, this appears to be an oversight given the possible risks posed by disengagement to a healthy and functioning society.
In 2021, for the first time, youth disillusionment was named a global risk by the World Economic Forum. They also observed that the prevalence of this phenomenon worsened during the recent pandemic.[8] This sentiment is not only being expressed by the 50% of youth living in dictatorships and countries with authoritarian regimes but also those residing in westernised, supposedly democratic, nations. For example, in France between 30-40% of young people are thought to have abstained from voting in the last election, bolstering claims that they have lost faith in their ability to influence the political system.
The pervasive discontentment about the state of the world that this generation is currently reporting could plausibly be reduced by increasing their sense of engagement. Helping young people feel more invested in societal and democratic health would appear especially important given that the leaders of the future are undoubted among them. Ultimately, these individuals will likely face decisions that could influence the long-term future of humanity, providing a strong argument for initiatives which promote the development of sound moral values.
In addition, having a high proportion of disengaged citizens could conceivably lead to some or all of the following problems:
Widespread social isolation and exclusion. This is already being observed to a worrying extent in some cultures e.g. the Japanese hikikomori
Students dropping out of school or further education earlier
Increased rates of crime and unemployment
Higher proportions of citizens experiencing burnout and mental health issues
Greater polarisation and civil unrest or, even worse, a total collapse of stable democracy
Increased susceptibility of vulnerable citizens to radicalisation and committing acts of atrocity e.g. terrorist attacks and school shootings
What might measures of civic engagement reveal that are not captured in current evaluations of pecuniary and health effects?
Since greater civic engagement is associated with higher incomes and improved health, it could be argued that this would negate the need to measure the state separately. However, research has shown that wealth and disability status do not necessarily correlate strongly with increased subjective wellbeing. Also, engagement can improve citizens’ lives in other ways not captured by these metrics, primarily through increasing individuals’ sense of connection and purpose.
When it comes to mental health, for example, the negative consequences of psychological disorders may be poorly captured by DALYs due to underreporting of these conditions, especially in LMICs.[9] This is in addition to the fact that the metric tends to weight years lost to life (YLL) more strongly than those lost to disabilities (YLD).[10] DALYs also fail to capture the downstream impact on the lives of direct beneficiaries’ family and friends e.g. the decreased stress placed on the parents of a child who is protected from malaria. These could plausibly be picked up in measurements of civic engagement which gives a better indication of life satisfaction and general wellbeing in the community.
I would also tentatively suggest that measuring engagement could also prove highly applicable to grant applications in some of Open Phil’s new focus areas, such as Effective Altruism Community Building, Global Health Policy and Criminal Justice/Land Use Reform.
How can engagement be measured?
Researchers in the social sciences posit that civic engagement is a difficult construct to measure since it results from the amalgamation of various contextual factors. Although some multidimensional assessment tools, such as the ‘Civic Engagement Scale’[11] and the United States ‘Civic Engagement Supplement to the Current Population Survey’,[12] do exist, these tests are poorly validated, require a significant investment of time to conduct and to analyse once they are complete.
In regards to grantmaking decisions, where calculations are rough and based on approximate metrics, it would appear valuable to develop a measure that can be calculated more rapidly to expedite the process. With the caveat that it could require a slightly reductionistic approach, it may therefore be worthwhile to consider how engagement is evaluated in time-pressed, less academically-rigorous settings:
Economic indicators: Voter turnout
The OECD better life index suggests that voter participation[13] is the most effective way to measure civic engagement for a variety of reasons, including high data quality and cross-country comparability. It cites the benefits of improving this figure as an increased likelihood that the political system represents the views of a majority of the population
However, voter turnout is by no means perfect the perfect metric, given that civic participation encompasses a wider range of issues than just political engagement. Differences in electoral systems e.g. whether or not it is compulsory to vote in a specific country, can also significantly influence the measure and it is of little to no use when it comes to measuring engagement amongst children and young people under the age of 18.
Business indicators: The Net Promotor Score
Looking to other industries, such as the measurement of engagement in corporate settings, it seems plausible that there may be transferable insights from this sector. In market research, for example, customer engagement is regularly assessed using a metric known as the Net Promoter Score, or NPS. This tool facilitates the rapid evaluation of an individual’s brand loyalty and consumer satisfaction. Similarly, HR departments use an adapted version of the test (the eNPS) to measure employees’ sense of engagement in the workplace and, more recently, it has even been adopted in healthcare settings as a way to assess and improve the patient experience.
The net promoter score is a subjective measure that asks individuals to rate the likelihood that they would recommend an organisation/service/experience to a family member or friend. Those who answer with a 9 or 10 are classed as ‘promoters’, whilst those scoring 6 or below are described as ‘detractors’. 7 and 8s are discarded as ‘passive’ consumers.
This can be summarised as follows:
NPS = % promoters - % detractors (expressed as a single figure from – 100 to 100)
Nevertheless, despite its popularity, the NPS score almost undoubtedly has its limitations. For example, the ease of assessment comes at the expense of detailed insights into the factors driving the trends in engagement. Additionally, the idea that a single figure could provide a comprehensive picture of customer behaviour has been approached with a high degree of scepticism by critics of the metric. These individuals have also called into question the accuracy of the values used to classify promoters, detractors and passive consumers, alongside its failure to draw a distinction between intention and observed behaviour.[14][15]
Estimated Engagement and Net Involvement Scores (NIS)
Estimated Engagement (EE)
Given the aforementioned recommendations from the OECD to base measurements of civic engagement in a community on concrete metrics and empirical evidence, a simple way to approximate the effect of an intervention on beneficiaries’ engagement could be to calculate the percentage change in one or more of these.
For example, the percentage of adults voting in an election could be combined with the percentage of youth who are in school, further education or permanent employment to approximate the proportion of engaged citizens in a given population. Whilst this would not be fully representative of the attitudes and behaviours which comprise full community engagement, it would give a rough guide of the extent to which people are participating in society and could perhaps be enhanced by factoring in unemployment or crime rates.
Net Involvement Score (NIS)
For a more detailed and reliable insight into an intervention’s effect on civic engagement, I propose the use of measurement based loosely on the NPS. Although this would, of course, be a subjective measurement, it would likely be sufficiently representative to help inform Open Phil’s grantmaking efforts whilst remaining feasible in terms of data collection. The simplicity of such a has the appeal of speed and accessibility, generating easily obtainable results which could be used to compare the engagement of people from a wide range of backgrounds and culturally diverse communities.
In practice, the scoring process could be as straightforward as asking individuals to self-report their sense of engagement by having them rate a) their feelings of connection to their local community and b) desire to participate in activities that positively impact the wider society, capturing data on both civic attitudes and behaviour.
Taking into account the concerns of the previously alluded to critics of the NPS, it may also be appropriate to make slight modifications to the current scoring system. A brief scan of various studies evaluating the usefulness of the NPS[16] confirmed my a priori beliefs about people’s proclivity to rate neutral experiences as a 7. Previous research suggests that the NPS both slightly underestimates and overestimates the numbers of promoters and detractors, respectively. These observations prompt my suggestion that it would be more accurate to lower the bar slightly when evaluating this metric, classing citizens reporting average involvement scores of 8, 9 or 10 as ‘engaged’, with those scoring > 5 defined as ‘disengaged’ citizens.
The net involvement score (NIS) could then be determined using the following equation:
NIS = % of engaged citizens - % of disengaged citizens
EE and NIS in action: Hypothetical case studies
Applying these metrics to two hypothetical grantmaking applications, I have attempted to demonstrate how they might be used to measure civic engagement in a real-world setting:
Grant application 1 – Grandad’s Chair
Cause Area: Global Health and Development
An initiative to target male suicide in agricultural communities which involves healthcare workers training elders to deliver mental health first aid and basic psychiatric care to younger members of their communities
Metrics:
Increased DALYS?: 2.2 per 100,000 individuals
Increased income?: N/A
Estimated Engagement: +25%
In this made-up scenario, measuring DALYS alone would provide some insight into the effectiveness of the intervention, primarily resulting from a decrease in direct deaths from suicide. However, as mentioned previously, the increased DALYs are likely to be an underrepresentation of the actual health and well-being effects as a consequence of underreporting of years lost due to disabilities, such as anxiety or depression.
The estimated engagement score here has been determined by evaluating the percentage increase in voter turnout, adjusted to factor in significantly decreased crime rates in the region. Had this figure not been included, this grant may have been rejected in favour of another which led to a greater apparent increase in DALYs but which had little to no effect on beneficiaries’ psychological wellbeing or community engagement.
Grant application 2 – MediMent
Cause Area: Effective Altruism Community Growth—Global Health and Wellbeing
A UK-based organisation which provides aspiring doctors from disadvantaged backgrounds with work experience opportunities and access to mentoring
Metrics:
Increased DALYS?: N/A
Increased income?: Approximately £20,000/year
Net Involvement Score: +70 points (from −10 to 60)
In this hypothetical example, the beneficiaries would appear to experience only a modest salary increase when considering the initiative’s pecuniary and health benefits. However, the NIS score provides a much more compelling case for supporting the grant application. It demonstrates the significant positive impact that participating in the scheme could have on beneficiaries’ sense of connection/civic engagement and once again illustrates how DALYs might fail to faithfully represent wider-ranging effects on wellbeing.
Key questions and uncertainties
Nevertheless, some key questions and uncertainties remain and should be taken into account when evaluating the usefulness of these metrics:
1. To what extent are civic engagement and well-being linked?
Whilst there is a clear association between increased engagement and subjective wellbeing, the strength and direction of this relationship are not as clear in the social science literature. At present, the prevailing narrative is that the relationship is somewhat bidirectional, i.e. people who feel more satisfied in life have a greater tendency toward societal involvement which also has benefits on citizens’ overall well-being. However, the observed improvements could plausibly be influenced by another contributing factor and more in-depth studies
2. How would we weigh the benefits of increasing income or extending lives against those of improving engagement?
When comparing two distinct initiatives where a change in engagement is the only significant distinguishing factor, it might also be useful to consider how we would compare its value to the benefits of increased health or income. When it comes to community involvement, what would be a suitable ‘bar’ for impact? Additionally, when evaluating causes targeting different populations, should a 10% increase in voter participation be viewed as having equivalent effects on subjective wellbeing as a 10% increase in young people attending school or engaged in employment?
3. Should a metric to measure civic engagement be considered not as a concrete figure but rather as a decision-making ‘compass’?
Studies comparing results obtained using EE and NIS scores to those obtained through more in-depth surveys would need to be performed to ascertain the validity of the measures. That being said, there is a nontrivial likelihood that the best approach could be to view these metrics not as definitive measures of engagement but rather indicators of our best judgement of an intervention’s effects. Whilst this would admittedly preclude a ‘cost per dollar’ evaluation, this may turn out not to be a feasible goal when assessing a psychological state with a high degree of subjectivity. In this way, the figures could help guide grantmakers without concern about whether they are sufficiently representative to be used in their BOTEC calculations.
Summary
In summary, it seems highly plausible that having reliable metrics to measure a psychological state like civic engagement would be valuable, adding both colour and depth to Open Phil’s grantmaking decisions. I believe that figures such as estimated engagement or net involvement scores could be a reasonable way of approximating this. Nevertheless, several questions remain about exactly how this would be achieved in practice and further investigation of this area appears to be warranted.
References
[1] https://www.apa.org/education-career/undergrad/civic-engagement
[2] https://www.britannica.com/topic/civic-engagement
[3] https://youth.gov/youth-topics/civic-engagement-and-volunteering
[4] https://ginsberg.umich.edu/article/well-being-and-civic-engagement
[5] https://journals.sagepub.com/doi/full/10.1177/08901171211001242
[6] Ginwright, S. 2011. “Hope, Healing, and Care Pushing the Boundaries of Civic Engagement for African American Youth.” Liberal Education 97, no. 2: 34- 39.
[7] https://www.newsecuritybeat.org/2022/06/disillusioned-youth-danger-democracy/
[8] https://www.weforum.org/agenda/2021/06/youth-disillusionment-global-risk-gen-z-resilience/
[9] https://ourworldindata.org/global-mental-health
[10] https://www.psychiatryadvisor.com/home/topics/general-psychiatry/2019-global-burden-of-diseases-found-dalys-increased-from-3-1-to-4-9-percent/
[11] https://journals.sagepub.com/doi/pdf/10.1177/2158244013495542
[12] https://nap.nationalacademies.org/catalog/18831/civic-engagement-and-social-cohesion-measuring-dimensions-of-social-capital
[13] Defined as ‘the percentage of the registered population that voted during an election’.
[14] https://www.researchgate.net/publication/230557989_A_Holistic_Examination_of_Net_Promoter
[15] https://www.forbes.com/sites/forbesagencycouncil/2021/05/17/why-nps-should-not-be-your-only-customer-experience-metric/
[16] https://hbr.org/2019/10/where-net-promoter-score-goes-wrong
https://www.apa.org/education-career/undergrad/civic-engagement
https://www.britannica.com/topic/civic-engagement
https://youth.gov/youth-topics/civic-engagement-and-volunteering
https://ginsberg.umich.edu/article/well-being-and-civic-engagement
https://journals.sagepub.com/doi/full/10.1177/08901171211001242
Ginwright, S. 2011. “Hope, Healing, and Care Pushing the Boundaries of Civic Engagement for African American Youth.” Liberal Education 97, no. 2: 34- 39.
https://www.newsecuritybeat.org/2022/06/disillusioned-youth-danger-democracy/
https://www.weforum.org/agenda/2021/06/youth-disillusionment-global-risk-gen-z-resilience/
https://ourworldindata.org/global-mental-health
https://www.psychiatryadvisor.com/home/topics/general-psychiatry/2019-global-burden-of-diseases-found-dalys-increased-from-3-1-to-4-9-percent/
https://journals.sagepub.com/doi/pdf/10.1177/2158244013495542
https://nap.nationalacademies.org/catalog/18831/civic-engagement-and-social-cohesion-measuring-dimensions-of-social-capital
Defined as ‘the percentage of the registered population that voted during an election’.
https://www.researchgate.net/publication/230557989_A_Holistic_Examination_of_Net_Promoter
https://www.forbes.com/sites/forbesagencycouncil/2021/05/17/why-nps-should-not-be-your-only-customer-experience-metric/
https://hbr.org/2019/10/where-net-promoter-score-goes-wrong