I totally agree here if we are talking about giving people the best experience, which is a lot of what we want to do to facilitate friendships that will support people long-term in their motivation and making big decisions related to their career or life that could be quite impactful.
I also worry about feedback loops here, and how it’s easiest to optimize for people giving you good reviews at the end of your event, which means optimizing for people’s happiness over everything else. I’d be very excited about events and retreats that more consistently do follow-ups 1-12 months after the event so we can see what really impacted and supported people. I’m guessing a lot of is vibes, but it could be a lot less than I currently think (my position is currently similar to yours). There are big impactful wins to be had that optimizing for people’s well-being will liekly not get us to.
Strong +1! Don’t think I have much more to add here—I think doing more post-event follow up would be great.
Would also be interesting to think through complications attributing value from a event several months prior, especially when people who attend one event usually attend a bunch more, and disentangling the impact from each could be hard—curious if you’ve thought about ways to address that.
But I would guess that about the closest you can get is doing user interviews (or surveys but I don’t think you could get many people to fill them out) multiple months out, and just asking people how they think it affected them, and how counterfactual they think that impact was. I think people will have good enough insight here for this to get you most of the valuable information. My first EAG was the difference between me working in an EA org and being a software engineering. My most recent EAG did almost nothing for me, on reflection, even though I made new connections and rated it very highly.
I think just asking this directly probably gets us closer than trying to assign what portion of the impact each particular event might get, even though I agree in reality the picture is much more complicated than this.
And if anyone has ideas on how to do better impact on analysis than this on events, PLEASE tell me. But I think this is already a huge improvement on my sense of what the default impact analysis for EA events is, and anything more complicated won’t give us too much more information.
I totally agree here if we are talking about giving people the best experience, which is a lot of what we want to do to facilitate friendships that will support people long-term in their motivation and making big decisions related to their career or life that could be quite impactful.
I also worry about feedback loops here, and how it’s easiest to optimize for people giving you good reviews at the end of your event, which means optimizing for people’s happiness over everything else.
I’d be very excited about events and retreats that more consistently do follow-ups 1-12 months after the event so we can see what really impacted and supported people. I’m guessing a lot of is vibes, but it could be a lot less than I currently think (my position is currently similar to yours). There are big impactful wins to be had that optimizing for people’s well-being will liekly not get us to.
For more on this you can check out similar thoughts from this forum post on why CFAR didn’t go as well as planned, or Andy Matuschak’s thoughts on “enabling environments”.
Strong +1! Don’t think I have much more to add here—I think doing more post-event follow up would be great.
Would also be interesting to think through complications attributing value from a event several months prior, especially when people who attend one event usually attend a bunch more, and disentangling the impact from each could be hard—curious if you’ve thought about ways to address that.
I have not!
But I would guess that about the closest you can get is doing user interviews (or surveys but I don’t think you could get many people to fill them out) multiple months out, and just asking people how they think it affected them, and how counterfactual they think that impact was. I think people will have good enough insight here for this to get you most of the valuable information. My first EAG was the difference between me working in an EA org and being a software engineering. My most recent EAG did almost nothing for me, on reflection, even though I made new connections and rated it very highly.
I think just asking this directly probably gets us closer than trying to assign what portion of the impact each particular event might get, even though I agree in reality the picture is much more complicated than this.
And if anyone has ideas on how to do better impact on analysis than this on events, PLEASE tell me. But I think this is already a huge improvement on my sense of what the default impact analysis for EA events is, and anything more complicated won’t give us too much more information.