I want to start off by saying how great it is that CEA is publishing this dashboard. Previously, I’ve been very critical about the fact that the community didn’t have access to such data, so I want to express my appreciation to Angelina and everyone else who made this possible. My interpretation of the data includes some critical observations, but I don’t want that to overshadow the overall point that this dashboard represents a huge improvement in CEA’s transparency.
My TLDR take on the data is that Events seem to be going well, the Forum metrics seem decent but not great, Groups metrics look somewhat worrisome (if you have an expectation that these programs should be growing), and the newsletter and effectivealtruism.org metrics look bad. Thoughts on metrics for specific programs, and some general observations, below.
Events
FWIW, I don’t find the “number of connections made” metric informative. Asking people at the end of a conference how many people they’d hypothetically feel comfortable asking for a favor seems akin to asking kids at the end of summer camp how many friends they made that they plan to stay in touch with; if you asked even a month later you’d probably get a much lower number. The connections metric probably provides a useful comparison across years or events, I just don’t think the unit or metric is particularly meaningful. Whereas if you waited a year and asked people how many favors they’ve asked from people they met at an event, that would provide some useful information.
That said, I like that CEA is not solely relying on the connections metric. The “willingness to recommend” metric seems a lot better, and the scores look pretty good. I found it interesting that the scores for EAG and EAGX look pretty similar.
Online (forum)
It doesn’t seem great that after a couple of years of steady growth, hours of engagement on the forum seems to have spiked from FTX (and to a lesser extent WWOTF), then fallen to roughly levels from April 2022. Views by forum users follows the same pattern, as does the number of posts with >2 upvotes.
Monthly users seem to have spiked a lot around WWOTF (September 2022 users are >50% higher than March 2022 users), and is now dropping, but hasn’t reverted as much as the other metrics. Not totally sure what to make of that. It would be interesting to see how new users acquired in mid-2022 have behaved subsequently.
It seems pretty bad that traffic to the homepage and intro pages grew only very modestly from early 2017 to early 2022 (CEA has acknowledged mistakenly failing to prioritize this site over that period). WWOTF, and then FTX, both seem to have led to enormous increases in traffic relative to that baseline, and homepage traffic remains significantly elevated (though it is falling rapidly).
IMO it is very bad that WWOTF doesn’t seem to have driven any traffic to the intro page and that intro page traffic is the lowest level since the data starts in April 2017, and has been falling steadily since FTX. Is CEA doing anything to address this?
Going forward, it would be great if the dashboard included some kind of engagement metric(s) such as average time on site in addition to showing the number of visitors.
Online (newsletter)
Subscriber growth grew dramatically from 2016-2018 (perhaps boosted by some ad campaigns during the period of fastest growth?), then there were essentially no net additions of subscribers in 2019-2020. We then saw very modest growth in 2021 and 2022, followed by a decline in subscribers year to date in 2023. So 2019, 2020, and 2023 all seem problematic, and from the end of 2018 to today subscriber growth has only grown about 15% (total, not annually) despite huge tailwinds (e.g. much more spent on community building and groups, big investments in promoting WWOTF, etc.) And the 2023 YTD decline seems particularly bad. Do we have any insight into what’s going on? There are obviously people unsubscribing (have they given reasons why?); are we also seeing a drop in people signing up?
Going forward, it would be great if the dashboard included some kind of engagement metric(s) in addition to showing the number of subscribers.
Groups (UGAP)
I was somewhat surprised there wasn’t any growth between spring 2022 and spring 2023, as I would have expected a new program to grow pretty rapidly (like we saw between fall 2021 and fall 2022). Does CEA expect this program to grow in the future? Are there any specific goals for number of groups/participants?
Groups (Virtual)
The data’s kind of noisy, but it looks like the number of participants has been flat or declining since the data set starts. Any idea why that’s the case? I would have expected pretty strong growth. The number of participants completing all or most also of the sessions also seems to be dropping, which seems like a bad trend.
The exit scores for the virtual programs have been very consistent for the last ~2 years. But the level of those scores (~80/100) doesn’t seem great. If I’m understanding the scale correctly, participants are grading the program at about a B-/C+ type level. Does CEA feel like it understands the reason for these mediocre scores and have a good sense of how to improve them?
General observations
When metrics are based on survey data, it would be very helpful to provide a sample size and any information that would help us think about response biases (e.g. are the virtual group exit scores from all participants? Just those who completed all the content? What percentage of EAG/x attendees completed a survey?)
It would be helpful to have a way to export the dashboard data into a csv file.
I’d be much more interested in reading CEA’s interpretation of the dashboard data vs. “impact stories.” I do think anecdata like impact stories is useful, but I view the more comprehensive data as a lot more important to discuss. I’d also like to see that discussion contextualize the dashboard data (e.g. placing the OP’s headlines numbers in historical context as they are almost impossible to interpret on a standalone basis).
My general expectation is that most growth metrics (e.g. number of forum users) should be improving over time due to large increases in community building funding and capacity. For quality metrics (e.g. willingness to recommend EAG) I think/hope there would be some upward bias over time, but much less so than the growth metrics. I’d be curious whether CEA agrees with these baseline expectations.
This is a good observation. As we mentioned in this retrospective from last Fall, we decided to restrict UGAP to only new university groups to keep the program focused. In the past, we had more leeway and admitted some university groups that had been around longer. I think we have hit a ~ plateau on the number of new groups we expect to pop up each semester (around 20-40) so I don’t expect this program to keep growing.
We piloted a new program for organizers from existing groups in the winter alongside the most recent round of UGAP. However, since this was a fairly scrappy pilot we didn’t include it on the dashboard. We are now running the full version of the program and have accepted >60 organizers to participate. This may scale even more in the future but we are more focused on improving quality than quantity at this time. It is plausible that we combine this program and UGAP into a single program with separate tracks but we are still exploring.
We are also experimenting with some higher-touch support for top groups which is less scalable (such as our organizer summit). This type of support also lends itself less well to dashboards but we are hoping to produce some shareable data in the future.
Thanks for providing that background Jessica, very helpful. It’d be great to see metrics for the OSP included in the dashboard at some point.
It might also make sense to have the dashboard provide links to additional information about the different programs (e.g. the blog posts you link to) so that users can contextualize the dashboard data.
I have been an “unofficial” golden rule guy all my life (do they have an org or certification?) I have been following EA for many years and have done my part in my profession over the years to think in terms of more than monetizing, aggregating, streamlining, making bucks. As a healthcare provide in the US until recent retirement, I provided thousands of hours of pro bono or sliding scale work to meet the real needs of the clients I assisted which, as you may know, in the US healthcare has been manipulated into aggressive money making schemes.
Intro over. I found that without a good domain name or service mark, my work on the web with providing up to date mental health care information did not get much attention. When I dove in using the domain name DearShrink.com, attention to and distribution of reliable information became much more effective.
I am wondering if CEA has thought about a service mark or motto that would be easier to understand, identify with, get attached to, and remember, such as BeKindAndProsper com (.org .net)
If so, please advise me as to how I could move these domain names on to the future they did not get a chance to have with me (I was a little optimistic...). I sure don’t want to sell those domain names to some manipulative merch to make themselves appear good.
Thanks for the suggestion! I think CEA is unlikely to change our domain, but if you have some you would like to go to a good use, you could consider putting them up on ea.domains.
Absolutely, thank you! I keep looking for a way to move several humanistic domain names on to new owners and users, since I did not get to use them as I wished. Now that I am getting close to 80 earth years, I figure I should get these potentially useful domain names into the right hands. I have not found the right place for that so far. So, your suggestion is awesome.
I wasn’t really suggesting a change in the EA branding or marks. I was hoping I could find just what you led me to. I will post there soon, after I introduce myself a bit more. I am trying to support the Stop Starlink Satellites movement and have a simple website that hopefully can be found by coincidence if any searches for “how do I stop my service?” Or, things like that. www.stopstarlink.com
Thanks for your comment and feedback. I work on the events team, so just responding to a few things:
Asking people at the end of a conference how many people they’d hypothetically feel comfortable asking for a favor seems akin to asking kids at the end of summer camp how many friends they made that they plan to stay in touch with; if you asked even a month later you’d probably get a much lower number
We do ask attendees about the number of new connections they made at an event a few months later and FYI it’s (surprisingly, for the reasons you describe) roughly the same. LTR dips slightly lower a few months after the event, but not by much.
I agree it’s an imperfect metric though, and we’re exploring whether different survey questions can give us better info. As you note, we also supplement this with the LTR score as well as qualitative analysis of written answers and user interviews. You can look at some more thorough analyses in my recent sequence, I’d love more feedback there.
I found it interesting that the scores for EAG and EAGX look pretty similar.
Just to note that I don’t think LTR closely tracks the overall value of the event either. We find that events that reach a lot of new people often get high LTRs (but not always) so a high LTR score at an EAGx event could be due to first-timers feeling excited/grateful to attend their first event which I think is a meaningfully good thing but perhaps worth mentioning.
What percentage of EAG/x attendees completed a survey?
It’s usually 20 − 50% of attendees in the surveys referenced by the data you see. Thanks for the suggestion of including no. of respondents.
Thanks for sharing the survey response rate, that’s helpful info. I’ve shared some other thoughts on specific metrics via a comment on your other post.
Thanks, I’m glad you like the dashboard and are finding it a productive way to engage with our work! That’s great feedback for us :) As a quick aside, we actually launched this in late December during our end of year updates, this version just includes some fresh data + a few new metrics.
I’m not sure if all our program owners will have the capacity to address all of your program-specific thoughts, but I wanted to provide some high level context on the metrics we included for now.
When selecting metrics to present, we had to make trade offs in terms of:
How decision guiding a particular metric is internally
How much signal we think a metric is tracking on our goals for a particular project
How legible this is to an external audience
How useful is this as an additional community-known data point to further outside research
E.g. Metrics like “number of EAG attendees” and “number of monthly active Forum users” seem potentially valuable as a benchmark for the size of various EA-oriented products, but aren’t necessarily the core KPIs we are trying to improve for a given project or time period – for one thing, the number of monthly active Forum users only tracks logged in users!
The quality of the underlying data
E.g. We’ve tried to include as much historical data as we could to make it easier to understand how CEA has changed over time, but our program data from many years ago is harder for current staff to verify and I expect is generally less reliable.
Whether something is tracking confidential information
Time constraints
Updating this dashboard was meant as a ~2 week sprint for me, and imposes a bunch of coordination costs on program owners to align on the data + presentation details.
So as a general note, while we generally try to present metrics that meet many of the above criteria, not all of these are numbers we are trying to blindly make go up over time. My colleague Ollie’s comment gives an example of how the metrics presented here (e.g. # total connections, LTR) are contextualized and supplemented with other evidence when we evaluate how our programs are performing.
Also thanks for your feature requests / really detailed feedback. This is a WIP project not (yet) intended to be comprehensive, so we’ll keep a note of these suggestions for if and when we make the next round of updates!
Thank you for this thoughtful response. It is helpful to see this list of tradeoffs you’re balancing in considering which metrics to present on the dashboard, and the metrics you’ve chosen seem reasonable. Might be worth adding your list to the “additional notes” section at the top of the dashboard (I found your comment more informative than the current “additional notes” FWIW).
While I understand some metrics might not be a good fit for the dashboard if they rely on confidential information or aren’t legible to external audiences, I would love to see CEA provide a description of what your most important metrics are for each of the major program areas even if you can’t share actual data for these metrics. I think that would provide valuable transparency into CEA’s thinking about what is valuable, and might also help other organizations think about which metrics they should use.
I want to start off by saying how great it is that CEA is publishing this dashboard. Previously, I’ve been very critical about the fact that the community didn’t have access to such data, so I want to express my appreciation to Angelina and everyone else who made this possible. My interpretation of the data includes some critical observations, but I don’t want that to overshadow the overall point that this dashboard represents a huge improvement in CEA’s transparency.
My TLDR take on the data is that Events seem to be going well, the Forum metrics seem decent but not great, Groups metrics look somewhat worrisome (if you have an expectation that these programs should be growing), and the newsletter and effectivealtruism.org metrics look bad. Thoughts on metrics for specific programs, and some general observations, below.
Events
FWIW, I don’t find the “number of connections made” metric informative. Asking people at the end of a conference how many people they’d hypothetically feel comfortable asking for a favor seems akin to asking kids at the end of summer camp how many friends they made that they plan to stay in touch with; if you asked even a month later you’d probably get a much lower number. The connections metric probably provides a useful comparison across years or events, I just don’t think the unit or metric is particularly meaningful. Whereas if you waited a year and asked people how many favors they’ve asked from people they met at an event, that would provide some useful information.
That said, I like that CEA is not solely relying on the connections metric. The “willingness to recommend” metric seems a lot better, and the scores look pretty good. I found it interesting that the scores for EAG and EAGX look pretty similar.
Online (forum)
It doesn’t seem great that after a couple of years of steady growth, hours of engagement on the forum seems to have spiked from FTX (and to a lesser extent WWOTF), then fallen to roughly levels from April 2022. Views by forum users follows the same pattern, as does the number of posts with >2 upvotes.
Monthly users seem to have spiked a lot around WWOTF (September 2022 users are >50% higher than March 2022 users), and is now dropping, but hasn’t reverted as much as the other metrics. Not totally sure what to make of that. It would be interesting to see how new users acquired in mid-2022 have behaved subsequently.
Online (effectivealtruism.org)
It seems pretty bad that traffic to the homepage and intro pages grew only very modestly from early 2017 to early 2022 (CEA has acknowledged mistakenly failing to prioritize this site over that period). WWOTF, and then FTX, both seem to have led to enormous increases in traffic relative to that baseline, and homepage traffic remains significantly elevated (though it is falling rapidly).
IMO it is very bad that WWOTF doesn’t seem to have driven any traffic to the intro page and that intro page traffic is the lowest level since the data starts in April 2017, and has been falling steadily since FTX. Is CEA doing anything to address this?
Going forward, it would be great if the dashboard included some kind of engagement metric(s) such as average time on site in addition to showing the number of visitors.
Online (newsletter)
Subscriber growth grew dramatically from 2016-2018 (perhaps boosted by some ad campaigns during the period of fastest growth?), then there were essentially no net additions of subscribers in 2019-2020. We then saw very modest growth in 2021 and 2022, followed by a decline in subscribers year to date in 2023. So 2019, 2020, and 2023 all seem problematic, and from the end of 2018 to today subscriber growth has only grown about 15% (total, not annually) despite huge tailwinds (e.g. much more spent on community building and groups, big investments in promoting WWOTF, etc.) And the 2023 YTD decline seems particularly bad. Do we have any insight into what’s going on? There are obviously people unsubscribing (have they given reasons why?); are we also seeing a drop in people signing up?
Going forward, it would be great if the dashboard included some kind of engagement metric(s) in addition to showing the number of subscribers.
Groups (UGAP)
I was somewhat surprised there wasn’t any growth between spring 2022 and spring 2023, as I would have expected a new program to grow pretty rapidly (like we saw between fall 2021 and fall 2022). Does CEA expect this program to grow in the future? Are there any specific goals for number of groups/participants?
Groups (Virtual)
The data’s kind of noisy, but it looks like the number of participants has been flat or declining since the data set starts. Any idea why that’s the case? I would have expected pretty strong growth. The number of participants completing all or most also of the sessions also seems to be dropping, which seems like a bad trend.
The exit scores for the virtual programs have been very consistent for the last ~2 years. But the level of those scores (~80/100) doesn’t seem great. If I’m understanding the scale correctly, participants are grading the program at about a B-/C+ type level. Does CEA feel like it understands the reason for these mediocre scores and have a good sense of how to improve them?
General observations
When metrics are based on survey data, it would be very helpful to provide a sample size and any information that would help us think about response biases (e.g. are the virtual group exit scores from all participants? Just those who completed all the content? What percentage of EAG/x attendees completed a survey?)
It would be helpful to have a way to export the dashboard data into a csv file.
I’d be much more interested in reading CEA’s interpretation of the dashboard data vs. “impact stories.” I do think anecdata like impact stories is useful, but I view the more comprehensive data as a lot more important to discuss. I’d also like to see that discussion contextualize the dashboard data (e.g. placing the OP’s headlines numbers in historical context as they are almost impossible to interpret on a standalone basis).
My general expectation is that most growth metrics (e.g. number of forum users) should be improving over time due to large increases in community building funding and capacity. For quality metrics (e.g. willingness to recommend EAG) I think/hope there would be some upward bias over time, but much less so than the growth metrics. I’d be curious whether CEA agrees with these baseline expectations.
Hi! Just responding on the groups team side :)
This is a good observation. As we mentioned in this retrospective from last Fall, we decided to restrict UGAP to only new university groups to keep the program focused. In the past, we had more leeway and admitted some university groups that had been around longer. I think we have hit a ~ plateau on the number of new groups we expect to pop up each semester (around 20-40) so I don’t expect this program to keep growing.
We piloted a new program for organizers from existing groups in the winter alongside the most recent round of UGAP. However, since this was a fairly scrappy pilot we didn’t include it on the dashboard. We are now running the full version of the program and have accepted >60 organizers to participate. This may scale even more in the future but we are more focused on improving quality than quantity at this time. It is plausible that we combine this program and UGAP into a single program with separate tracks but we are still exploring.
We are also experimenting with some higher-touch support for top groups which is less scalable (such as our organizer summit). This type of support also lends itself less well to dashboards but we are hoping to produce some shareable data in the future.
Thanks for providing that background Jessica, very helpful. It’d be great to see metrics for the OSP included in the dashboard at some point.
It might also make sense to have the dashboard provide links to additional information about the different programs (e.g. the blog posts you link to) so that users can contextualize the dashboard data.
I have been an “unofficial” golden rule guy all my life (do they have an org or certification?) I have been following EA for many years and have done my part in my profession over the years to think in terms of more than monetizing, aggregating, streamlining, making bucks. As a healthcare provide in the US until recent retirement, I provided thousands of hours of pro bono or sliding scale work to meet the real needs of the clients I assisted which, as you may know, in the US healthcare has been manipulated into aggressive money making schemes.
Intro over. I found that without a good domain name or service mark, my work on the web with providing up to date mental health care information did not get much attention. When I dove in using the domain name DearShrink.com, attention to and distribution of reliable information became much more effective.
I am wondering if CEA has thought about a service mark or motto that would be easier to understand, identify with, get attached to, and remember, such as BeKindAndProsper com (.org .net)
If so, please advise me as to how I could move these domain names on to the future they did not get a chance to have with me (I was a little optimistic...). I sure don’t want to sell those domain names to some manipulative merch to make themselves appear good.
Let me know
Ron Sterling MD (retired)
Author, writer, neuroscientist
Thanks for the suggestion! I think CEA is unlikely to change our domain, but if you have some you would like to go to a good use, you could consider putting them up on ea.domains.
Absolutely, thank you! I keep looking for a way to move several humanistic domain names on to new owners and users, since I did not get to use them as I wished. Now that I am getting close to 80 earth years, I figure I should get these potentially useful domain names into the right hands. I have not found the right place for that so far. So, your suggestion is awesome.
I wasn’t really suggesting a change in the EA branding or marks. I was hoping I could find just what you led me to. I will post there soon, after I introduce myself a bit more. I am trying to support the Stop Starlink Satellites movement and have a simple website that hopefully can be found by coincidence if any searches for “how do I stop my service?” Or, things like that. www.stopstarlink.com
Thanks for your comment and feedback. I work on the events team, so just responding to a few things:
We do ask attendees about the number of new connections they made at an event a few months later and FYI it’s (surprisingly, for the reasons you describe) roughly the same. LTR dips slightly lower a few months after the event, but not by much.
I agree it’s an imperfect metric though, and we’re exploring whether different survey questions can give us better info. As you note, we also supplement this with the LTR score as well as qualitative analysis of written answers and user interviews. You can look at some more thorough analyses in my recent sequence, I’d love more feedback there.
Just to note that I don’t think LTR closely tracks the overall value of the event either. We find that events that reach a lot of new people often get high LTRs (but not always) so a high LTR score at an EAGx event could be due to first-timers feeling excited/grateful to attend their first event which I think is a meaningfully good thing but perhaps worth mentioning.
It’s usually 20 − 50% of attendees in the surveys referenced by the data you see. Thanks for the suggestion of including no. of respondents.
Thanks for sharing the survey response rate, that’s helpful info. I’ve shared some other thoughts on specific metrics via a comment on your other post.
Thanks, I’m glad you like the dashboard and are finding it a productive way to engage with our work! That’s great feedback for us :) As a quick aside, we actually launched this in late December during our end of year updates, this version just includes some fresh data + a few new metrics.
I’m not sure if all our program owners will have the capacity to address all of your program-specific thoughts, but I wanted to provide some high level context on the metrics we included for now.
When selecting metrics to present, we had to make trade offs in terms of:
How decision guiding a particular metric is internally
How much signal we think a metric is tracking on our goals for a particular project
How legible this is to an external audience
How useful is this as an additional community-known data point to further outside research
E.g. Metrics like “number of EAG attendees” and “number of monthly active Forum users” seem potentially valuable as a benchmark for the size of various EA-oriented products, but aren’t necessarily the core KPIs we are trying to improve for a given project or time period – for one thing, the number of monthly active Forum users only tracks logged in users!
The quality of the underlying data
E.g. We’ve tried to include as much historical data as we could to make it easier to understand how CEA has changed over time, but our program data from many years ago is harder for current staff to verify and I expect is generally less reliable.
Whether something is tracking confidential information
Time constraints
Updating this dashboard was meant as a ~2 week sprint for me, and imposes a bunch of coordination costs on program owners to align on the data + presentation details.
So as a general note, while we generally try to present metrics that meet many of the above criteria, not all of these are numbers we are trying to blindly make go up over time. My colleague Ollie’s comment gives an example of how the metrics presented here (e.g. # total connections, LTR) are contextualized and supplemented with other evidence when we evaluate how our programs are performing.
Also thanks for your feature requests / really detailed feedback. This is a WIP project not (yet) intended to be comprehensive, so we’ll keep a note of these suggestions for if and when we make the next round of updates!
Thank you for this thoughtful response. It is helpful to see this list of tradeoffs you’re balancing in considering which metrics to present on the dashboard, and the metrics you’ve chosen seem reasonable. Might be worth adding your list to the “additional notes” section at the top of the dashboard (I found your comment more informative than the current “additional notes” FWIW).
While I understand some metrics might not be a good fit for the dashboard if they rely on confidential information or aren’t legible to external audiences, I would love to see CEA provide a description of what your most important metrics are for each of the major program areas even if you can’t share actual data for these metrics. I think that would provide valuable transparency into CEA’s thinking about what is valuable, and might also help other organizations think about which metrics they should use.