I agree that it provides a bit of evidence on whether EAGs have a very spiky effect, but I had a relatively low prior on that. My sense is the kind of people who attend EAGs already do a bunch of stuff that connects them to the EA community, and while I do expect a mild peak in the weeks afterwards, 6 months is really far beyond where I expect any measurable effect.
but this seems to provide significant evidence against one major supposed channel of impact.
I don’t understand this. Why would it be important for EAGs impact to have a spiky intervention profile? If EAGs and EAGx events are useful in a way where the benefit is spread out so that they form a relatively stable baseline for EA involvement (which honestly seems almost inevitable given the kind of behavior that you were measuring for EA involvement), then we would measure no effect with your methodology.
6 months is really far beyond where I expect any measurable effect
I agree there wouldn’t be new effects at that point, but we’re asking about total effects over the 6 months before/since the conference. If the connections etc. persist for 6 months then it should show up in the survey and if they have dissapeared within a few months then that indicates these effects of EAGx attendance are short-lived, which presumably makes them far less significant for a person’s EA engagement and impact overall.
Why would it be important for EAGs impact to have a spiky intervention profile?
If the EAG impacts are spiky enough that they start disspiating substantially within several months (but get re-upped but future attendance) then we should be able to detect a change with our methodology (higher engagement after). You’re right that if the effects persist for many years (and don’t stack much with repeat attendance) then we wouldn’t be able to measure any effect on repeat attendees but this would presume that it isn’t having much impact on repeat attendees anyway. On the other hand, if effects persist for many years then we should be able to detect a strong effect for first-time attendees (though you’d need a bigger sample).
I agree that it provides a bit of evidence on whether EAGs have a very spiky effect, but I had a relatively low prior on that. My sense is the kind of people who attend EAGs already do a bunch of stuff that connects them to the EA community, and while I do expect a mild peak in the weeks afterwards, 6 months is really far beyond where I expect any measurable effect.
I don’t understand this. Why would it be important for EAGs impact to have a spiky intervention profile? If EAGs and EAGx events are useful in a way where the benefit is spread out so that they form a relatively stable baseline for EA involvement (which honestly seems almost inevitable given the kind of behavior that you were measuring for EA involvement), then we would measure no effect with your methodology.
I agree there wouldn’t be new effects at that point, but we’re asking about total effects over the 6 months before/since the conference. If the connections etc. persist for 6 months then it should show up in the survey and if they have dissapeared within a few months then that indicates these effects of EAGx attendance are short-lived, which presumably makes them far less significant for a person’s EA engagement and impact overall.
If the EAG impacts are spiky enough that they start disspiating substantially within several months (but get re-upped but future attendance) then we should be able to detect a change with our methodology (higher engagement after). You’re right that if the effects persist for many years (and don’t stack much with repeat attendance) then we wouldn’t be able to measure any effect on repeat attendees but this would presume that it isn’t having much impact on repeat attendees anyway. On the other hand, if effects persist for many years then we should be able to detect a strong effect for first-time attendees (though you’d need a bigger sample).