Thanks, I’m glad you like the dashboard and are finding it a productive way to engage with our work! That’s great feedback for us :) As a quick aside, we actually launched this in late December during our end of year updates, this version just includes some fresh data + a few new metrics.
I’m not sure if all our program owners will have the capacity to address all of your program-specific thoughts, but I wanted to provide some high level context on the metrics we included for now.
When selecting metrics to present, we had to make trade offs in terms of:
How decision guiding a particular metric is internally
How much signal we think a metric is tracking on our goals for a particular project
How legible this is to an external audience
How useful is this as an additional community-known data point to further outside research
E.g. Metrics like “number of EAG attendees” and “number of monthly active Forum users” seem potentially valuable as a benchmark for the size of various EA-oriented products, but aren’t necessarily the core KPIs we are trying to improve for a given project or time period – for one thing, the number of monthly active Forum users only tracks logged in users!
The quality of the underlying data
E.g. We’ve tried to include as much historical data as we could to make it easier to understand how CEA has changed over time, but our program data from many years ago is harder for current staff to verify and I expect is generally less reliable.
Whether something is tracking confidential information
Time constraints
Updating this dashboard was meant as a ~2 week sprint for me, and imposes a bunch of coordination costs on program owners to align on the data + presentation details.
So as a general note, while we generally try to present metrics that meet many of the above criteria, not all of these are numbers we are trying to blindly make go up over time. My colleague Ollie’s comment gives an example of how the metrics presented here (e.g. # total connections, LTR) are contextualized and supplemented with other evidence when we evaluate how our programs are performing.
Also thanks for your feature requests / really detailed feedback. This is a WIP project not (yet) intended to be comprehensive, so we’ll keep a note of these suggestions for if and when we make the next round of updates!
Thank you for this thoughtful response. It is helpful to see this list of tradeoffs you’re balancing in considering which metrics to present on the dashboard, and the metrics you’ve chosen seem reasonable. Might be worth adding your list to the “additional notes” section at the top of the dashboard (I found your comment more informative than the current “additional notes” FWIW).
While I understand some metrics might not be a good fit for the dashboard if they rely on confidential information or aren’t legible to external audiences, I would love to see CEA provide a description of what your most important metrics are for each of the major program areas even if you can’t share actual data for these metrics. I think that would provide valuable transparency into CEA’s thinking about what is valuable, and might also help other organizations think about which metrics they should use.
Thanks, I’m glad you like the dashboard and are finding it a productive way to engage with our work! That’s great feedback for us :) As a quick aside, we actually launched this in late December during our end of year updates, this version just includes some fresh data + a few new metrics.
I’m not sure if all our program owners will have the capacity to address all of your program-specific thoughts, but I wanted to provide some high level context on the metrics we included for now.
When selecting metrics to present, we had to make trade offs in terms of:
How decision guiding a particular metric is internally
How much signal we think a metric is tracking on our goals for a particular project
How legible this is to an external audience
How useful is this as an additional community-known data point to further outside research
E.g. Metrics like “number of EAG attendees” and “number of monthly active Forum users” seem potentially valuable as a benchmark for the size of various EA-oriented products, but aren’t necessarily the core KPIs we are trying to improve for a given project or time period – for one thing, the number of monthly active Forum users only tracks logged in users!
The quality of the underlying data
E.g. We’ve tried to include as much historical data as we could to make it easier to understand how CEA has changed over time, but our program data from many years ago is harder for current staff to verify and I expect is generally less reliable.
Whether something is tracking confidential information
Time constraints
Updating this dashboard was meant as a ~2 week sprint for me, and imposes a bunch of coordination costs on program owners to align on the data + presentation details.
So as a general note, while we generally try to present metrics that meet many of the above criteria, not all of these are numbers we are trying to blindly make go up over time. My colleague Ollie’s comment gives an example of how the metrics presented here (e.g. # total connections, LTR) are contextualized and supplemented with other evidence when we evaluate how our programs are performing.
Also thanks for your feature requests / really detailed feedback. This is a WIP project not (yet) intended to be comprehensive, so we’ll keep a note of these suggestions for if and when we make the next round of updates!
Thank you for this thoughtful response. It is helpful to see this list of tradeoffs you’re balancing in considering which metrics to present on the dashboard, and the metrics you’ve chosen seem reasonable. Might be worth adding your list to the “additional notes” section at the top of the dashboard (I found your comment more informative than the current “additional notes” FWIW).
While I understand some metrics might not be a good fit for the dashboard if they rely on confidential information or aren’t legible to external audiences, I would love to see CEA provide a description of what your most important metrics are for each of the major program areas even if you can’t share actual data for these metrics. I think that would provide valuable transparency into CEA’s thinking about what is valuable, and might also help other organizations think about which metrics they should use.