Thanks for publishing this very thorough review Max! I read most of it and the community building grants and group support sections were particularly important and useful for me to know, though the other parts were also useful to read.
I have a few questions which one of you at CEA may want to answer, and I’ll split these into different comments, so that people can reply separately to each question:
1. Regarding this line in your section on community building grants:
We judged 86 of the 145 group members to have taken significant action based on a good understanding of EA ideas, and we categorised these cases as strong, moderate, or weak based on our expectations about the counterfactual impact the group had on the individual.
I’d like to learn more about how CEA or the CB grants programme categorizes these cases into strong, moderate, or weak impact? I think there is a lot of value in community builders, especially CB grantees, having a better understanding of what CEA considers to be impactful (and how you measure it). This way, this prevents CB grantees from being very positive about what CEA thinks is just a weak case of impact, or grantees thinking that something is moderate impact, but CEA thinks it’s strong impact. This then allows community builders to focus on generating more moderate or strong cases of impact, although of course they should not Goodhart (i.e. optimize too hard in a way that hampers the group).
I also understand that examples of impact (and therefore evaluating these examples) can vary widely across different group types (national, city, or university) and those in different countries (i.e. EA Philippines vs. EA London), but I’d still like to hear more about it.
In my head, I think that CEA should be measuring two things when trying to measure the impact of a group on its members: a) How high is the expected value of the action or career plan change that the person has taken
b) How counterfactual the impact of the group is on the person
The two things above can then be combined so that a case can be classified as strong, moderate, or weak impact. I’d like to know if what I wrote above on high expected value + the degree of it being counterfactual is aligned with CEA and/or the CBG programme’s thinking on evaluating these cases of impact. If you think though this information is too sensitive to share on the forum, then you can just send it to me and/or other CB grantees privately (or let me know if Harri will release a writeup on this for community builders in 2021). Thanks!
Hi Brian, thanks for your question, and I’m glad the update was useful!
You’re correct about the overall approach we’re using (multiplying the expected value of the change by how much of that change is attributable to the group). I’ll flag this comment to Harri and he might follow up with some more details, publicly or privately.
Thanks for publishing this very thorough review Max! I read most of it and the community building grants and group support sections were particularly important and useful for me to know, though the other parts were also useful to read.
I have a few questions which one of you at CEA may want to answer, and I’ll split these into different comments, so that people can reply separately to each question:
1. Regarding this line in your section on community building grants:
I’d like to learn more about how CEA or the CB grants programme categorizes these cases into strong, moderate, or weak impact? I think there is a lot of value in community builders, especially CB grantees, having a better understanding of what CEA considers to be impactful (and how you measure it). This way, this prevents CB grantees from being very positive about what CEA thinks is just a weak case of impact, or grantees thinking that something is moderate impact, but CEA thinks it’s strong impact. This then allows community builders to focus on generating more moderate or strong cases of impact, although of course they should not Goodhart (i.e. optimize too hard in a way that hampers the group).
I also understand that examples of impact (and therefore evaluating these examples) can vary widely across different group types (national, city, or university) and those in different countries (i.e. EA Philippines vs. EA London), but I’d still like to hear more about it.
In my head, I think that CEA should be measuring two things when trying to measure the impact of a group on its members:
a) How high is the expected value of the action or career plan change that the person has taken
b) How counterfactual the impact of the group is on the person
The two things above can then be combined so that a case can be classified as strong, moderate, or weak impact. I’d like to know if what I wrote above on high expected value + the degree of it being counterfactual is aligned with CEA and/or the CBG programme’s thinking on evaluating these cases of impact. If you think though this information is too sensitive to share on the forum, then you can just send it to me and/or other CB grantees privately (or let me know if Harri will release a writeup on this for community builders in 2021). Thanks!
Hi Brian, thanks for your question, and I’m glad the update was useful!
You’re correct about the overall approach we’re using (multiplying the expected value of the change by how much of that change is attributable to the group). I’ll flag this comment to Harri and he might follow up with some more details, publicly or privately.
Got it, thanks Max!