Thank you for this. I really appreciate this research because I think the EA community should do more to evaluate interventions (e.g., conferences, pledges, programs etc) considering the focus on cost-effectiveness etc. Especially repeat interventions. I also like the idea of having independent evaluations.
Having said that, good evaluations are very hard to do and don’t always offer a comparatively good ROI in expectation as compared to other uses of resources.
I think CEA are doing a very good job with conferences now, and feel pretty confident that the EA conference provides a lot of value. However, I am weakly in favor of it being evaluated more.
From my personal experience attending conferences, I’d like to suggest two considerations
First, 6 months is probably too short a timeframe to measure conference impact—many of my most valuable changes in behavior (e.g., starting new projects/collaborations, or providing (and getting) support/advice) occurred years after the conference where I met someone for the first time.
Second, I think that most of the conferences impacts come from network related behaviors and these don’t seem to be well captured. Many if not most of my higher impact actions only occurred because of people I met at EA Conferences.
First, 6 months is probably too short a timeframe to measure conference impact—many of my most valuable changes in behavior (e.g., starting new projects/collaborations, or providing (and getting) support/advice) occurred years after the conference where I met someone for the first time.
Possibly, though there is a trade-off here. We also hear in our 3–6 month follow-up surveys that people don’t really remember conversations from the event. Maybe that’s just a sign that nothing super valuable occurred but if you attend lots of events, questions about an event that occurred >6mo ago can be difficult to answer even if it was impactful. If we ask straight after the event, and 3–6 months later and a year later, I’d worry about survey fatigue.
Fair. Perhaps during the post event survey you could ask people who have attended previous events if they want to report any significant impacts from those past events? Then they can respond as relevant.
Thank you for this. I really appreciate this research because I think the EA community should do more to evaluate interventions (e.g., conferences, pledges, programs etc) considering the focus on cost-effectiveness etc. Especially repeat interventions. I also like the idea of having independent evaluations.
Having said that, good evaluations are very hard to do and don’t always offer a comparatively good ROI in expectation as compared to other uses of resources.
I think CEA are doing a very good job with conferences now, and feel pretty confident that the EA conference provides a lot of value. However, I am weakly in favor of it being evaluated more.
From my personal experience attending conferences, I’d like to suggest two considerations
First, 6 months is probably too short a timeframe to measure conference impact—many of my most valuable changes in behavior (e.g., starting new projects/collaborations, or providing (and getting) support/advice) occurred years after the conference where I met someone for the first time.
Second, I think that most of the conferences impacts come from network related behaviors and these don’t seem to be well captured. Many if not most of my higher impact actions only occurred because of people I met at EA Conferences.
Possibly, though there is a trade-off here. We also hear in our 3–6 month follow-up surveys that people don’t really remember conversations from the event. Maybe that’s just a sign that nothing super valuable occurred but if you attend lots of events, questions about an event that occurred >6mo ago can be difficult to answer even if it was impactful. If we ask straight after the event, and 3–6 months later and a year later, I’d worry about survey fatigue.
Fair. Perhaps during the post event survey you could ask people who have attended previous events if they want to report any significant impacts from those past events? Then they can respond as relevant.