Yeah, in the abstract, I’m skeptical of the way you are measuring this, because you are measuring quantity and not quality. You don’t just want “more connections”, you want more connections that lead somewhere, and it’s not clear to me that doubling the number of (junior) participants does this. You have a higher number of potential connections, but also a dillution effect.
So in a simple model where there are only “junior” (people looking for opportunities) and “senior” (people giving out opportunities) people, if you double the number of junior people who attend the conference, you will get more junior-junior connections, perhaps fewer senior-senior connections, and the effect on junior-senior connections depends on whether allowing more senior people to filter themselves produces better connections or mostly overwhelms them and increases the number of junior people they have to talk with until they find a good match (and thus reduces the number of total good matches). You can fix the reduced number of senior-senior connections by having separate senior-only events, but that then creates fewer senior-junior connections.
My impression is that that simplified model is in fact not that far off from what is actually happening. There are also longer term effects (e.g., maybe someone who goes to EA global and is offered no opportunities nonetheless is still inspired). But then I’d sort of want to see some attempt to estimate that instead, or at least point to it.
In the concrete, I’d say that the promisingness of the average participant was noticeably lower than in previous years, and that this effect made it harder for me specifically to find people that I got something out of talking with (but easier to find people who I could offer something to). This effect is cofounded by me in particular becoming more “senior”, or at least older, though. Nonetheless, I’ve attended a few EA Globals, and I wouldn’t say this one has been the best one.
Separately, I’m a bit skeptical of the value of “connections” at these types of events, and I tend to think that they are a bit fake. But you could test this, e.g., you could ask a random number of participants to actually follow up on their reported connections, and see whether they respond, or you could ask in a few months whether people actually followed up (maybe you are already doing this, though).
It’s also not clear to me whether this is an interim update (in which case I’m probably being unnecessarily harsh), or whether this is the extent of your evaluation (in which case I am somewhat worried, the same as I would be if 80,000 hours only measured “number of people we had calls with”, rather than “number of importance-adjusted career changes”).
Easy fix, if we can link survey responses to accounts:
modify the event survey in year n to ask for a list of named connections this year, then pull this same response in year n+1 and ask what number have so far proved to be valuable.
Tl;DR: we basically agree. We think the number of connections is (one of!) our decent, measurable proxies for Good Things Happening but we could do better and we’re working on that.
Yeah, in the abstract, I’m skeptical of the way you are measuring this, because you are measuring quantity and not quality. You don’t just want “more connections”, you want more connections that lead somewhere
Yes, we agree. We’re working on ideas that actually capture the “lead somewhere” part. This might be impact-adjusted connections or, more crudely, “critical connections” (connections that actually cause something good to happen).
You can fix the reduced number of senior-senior connections by having separate senior-only events, but that then creates fewer senior-junior connections.
Our (pretty scrappy) data suggests that the proportion of attendees who made “professional connections” was about the same in 2021 as previous conferences (~80%). The question for this was an option to select “I made an important professional connection (e.g. a potential career or hiring opportunity, or maybe a potential funder or collaborator)”. This sounds like it captures a lot of what you’d call “senior-junior connections”, but includes some senior-senior connections (funder/collaborator).
the promisingness of the average participant was noticeably lower than in previous years, and that this effect made it harder for me specifically to find people that I got something out of talking with
I’m sorry you found the conference a bit less valuable yourself. We’re hoping to build out our events portfolio such that more experienced EAs can continue to get a lot of value out of conferences, even as EAGs grow. Do let me know (here or via ollie@eaglobal.org) if you have ideas for this.
You could ask a random number of participants to actually follow up on their reported connections, and see whether they respond, or you could ask in a few months whether people actually followed up (maybe you are already doing this, though).
I share your scepticism. We already have plans to do this but this is a helpful nudge to make sure it happens!
It’s also not clear to me whether this is an interim update (in which case I’m probably being unnecessarily harsh), or whether this is the extent of your evaluation (in which case I am somewhat worried, the same as I would be if 80,000 hours only measured “number of people we had calls with”, rather than “number of importance-adjusted career changes”).
It’s an interim update, reporting immediate event outcomes, not the extent of our evaluation :)
Thanks for your comment Nuno, we do really appreciate the constructive feedback!
My read of your comment is that you have a well informed, personal view of many EAGs, and this is driving your skepticism.
I think your perspective and experience on EAG is useful, especially if this can lead to insights that could improve it, or create new narratives that allow us to perceive EA coordination better.
I wanted to write a few questions with this motivation.
I think I find the metric less relevant. (I think it might turn out to be difficult to measure match quality, especially impact, because Goodhart or something. Sometimes simple metrics as a proxy is ok.)
My impression is that that simplified model is in fact not that far off from what is actually happening….if you double the number of junior people who attend the conference…[this] increases the number of junior people [that senior people] have to talk with until they find a good match
I’d say that the promisingness of the average participant was noticeably lower than in previous years, and that this effect made it harder for me specifically to find people that I got something out of talking with
From what I understand, for the qualities of “juniorness” and “promisingness” that you mention, it seems they are two distinct issues or channels for how EAG is getting worse:
The noise or difficulty of matches due to quantity of “juniors” due to volume / communication / coordination costs imposed on seniors, and
The participant quality or “promisingness” decreasing (due to event size).
But I’m not sure 1) is important?
As long as the average junior person is equally promising, you can have equally good senior-junior matches, and the value of the event for a senior person seems unaffected as event size increases from more juniors. (Also, the total value of the event seems non-decreasing as you add more junior people).
If participant quality is the same, I don’t understand why this increases search costs for senior people or something. Or maybe it does for some, or maybe it involves management and is a skill to be learned? I guess I view these as second order effects and I don’t understand why the effect could be very large (I could be wrong).
So I’m speculating that actually 2) “promisingness” is the issue, and so I think the implication or subtext of your comment is that with more people, we’re walking down the quality curve of participants (and pretty quickly. If you are perceiving the change in total value of the EAG as negative from event size increases, this gives us a sense of the slope of participant quality).
Does this sound plausible or reflect your perspective?
I guess I what I am trying to do in this comment is punch through the discussion of the metric, and punch through several confounding things (your seniorness and “search costs”) to get at what I speculate is the essence of your experience, the change in promisingness—which actually is still sort of complicated and has confounding issues itself.
So, if you’re still with me here and this comment isn’t too rambling or confusing, it would be really interesting to hear you talk a lot more about your perspectives on promisingness and these confounding issues.
So if the pool of potential candidates is pretty similar from year to year and there isn’t a lot more going on, simply moving the cutoff criteria could account for your changed experiences, as I think you are saying. (My read is this is the subtext of your comment, that the size of EAG is affecting match quality).
But if the pool of candidates or some other quality of EAG is changing, maybe your experiences has a different explanation?
For example, a explanation might be some effect of a changed selection process, “advertising”, or the recent output of community builders. COVID might have affected the pool too.
Maybe movements get older and the energy changes naturally. This is neither good or bad.
The effect might be vary a lot by cause area, or some small but important community inside a cause area.
Do you have any insights about the above, or can you dismiss any of these speculations out of hand?
Another topic is the “design of EAG”, or something.
Again, my read is that the subtext of your comment is that the pool is being adversely affected by increasing the event size. Let’s fully accept this perspective.
So, regarding the “total value of the event for altruistic impact”. As touched on above, it’s not clear that decreasing match quality outweighs the value of quantity increases. If you’re straight up doubling the event size, that could allow for some decrease in average match quality.
Like, if EAG was 500 before, and 1,000 now, the “value for the 500 new people” might outweigh the “impact on the original 500 people”.
Obviously, a judgement about change in total value of the event is hard, but maybe you have a good guess or intuition about the sign of the change?
But there’s still more considerations:
EAG acceptance is seen as a badge of ingroupness, and this badge seems valuable for an decentralized movement. For example, being rejected hurts. Obviously, I understand if a dud like me gets culled. But I can think of many people, maybe who aren’t strong networkers or signallers, that EA should value, and many of them are not going to make it into a size 500 event and that seems bad.
You might think that EAG impact is driven by very very talented people, like the next Christiano or SBF and this affects your view of sizing and experience for these people. But there’s several special “programs”, that range from those explicitly run by formal EA orgs, as well as powerful de facto “programs” and personal recommendations, that find develop and connect talent at physical events, including EAGs. These exist among different cause areas. How these programs play out or could be improved in different conceptions of EAG seems pretty relevant.
I’m just some dude writing stuff onto an internet forum, but my guess is that a larger EAG is better given a view of a growing EA.
To the degree you think that previous executions of EAG are inadequate, I’m not sure that this is informative about the future.
It seems bad if a movement couldn’t scale a gathering beyond 500 or 1000 people. It seems like careful, high effort design, maybe new meeting formats or ways to improve match quality, could more than offset the downside of larger event sizes.
Hey, I think that these are all good comments, and I wouldn’t call you “a dud”. I agree with your thoughts around possible cofounders, though a decrease in average participant quality was the most salient explanation to me.
Sure! We’d be happy for some red-teaming or suggestions on how to improve our work.
Epistemic status: See profile.
tl;dr: Skeptical about measuring “conections”.
Yeah, in the abstract, I’m skeptical of the way you are measuring this, because you are measuring quantity and not quality. You don’t just want “more connections”, you want more connections that lead somewhere, and it’s not clear to me that doubling the number of (junior) participants does this. You have a higher number of potential connections, but also a dillution effect.
So in a simple model where there are only “junior” (people looking for opportunities) and “senior” (people giving out opportunities) people, if you double the number of junior people who attend the conference, you will get more junior-junior connections, perhaps fewer senior-senior connections, and the effect on junior-senior connections depends on whether allowing more senior people to filter themselves produces better connections or mostly overwhelms them and increases the number of junior people they have to talk with until they find a good match (and thus reduces the number of total good matches). You can fix the reduced number of senior-senior connections by having separate senior-only events, but that then creates fewer senior-junior connections.
My impression is that that simplified model is in fact not that far off from what is actually happening. There are also longer term effects (e.g., maybe someone who goes to EA global and is offered no opportunities nonetheless is still inspired). But then I’d sort of want to see some attempt to estimate that instead, or at least point to it.
In the concrete, I’d say that the promisingness of the average participant was noticeably lower than in previous years, and that this effect made it harder for me specifically to find people that I got something out of talking with (but easier to find people who I could offer something to). This effect is cofounded by me in particular becoming more “senior”, or at least older, though. Nonetheless, I’ve attended a few EA Globals, and I wouldn’t say this one has been the best one.
Separately, I’m a bit skeptical of the value of “connections” at these types of events, and I tend to think that they are a bit fake. But you could test this, e.g., you could ask a random number of participants to actually follow up on their reported connections, and see whether they respond, or you could ask in a few months whether people actually followed up (maybe you are already doing this, though).
It’s also not clear to me whether this is an interim update (in which case I’m probably being unnecessarily harsh), or whether this is the extent of your evaluation (in which case I am somewhat worried, the same as I would be if 80,000 hours only measured “number of people we had calls with”, rather than “number of importance-adjusted career changes”).
Easy fix, if we can link survey responses to accounts:
modify the event survey in year n to ask for a list of named connections this year, then pull this same response in year n+1 and ask what number have so far proved to be valuable.
Ollie here from CEA’s events team
Tl;DR: we basically agree. We think the number of connections is (one of!) our decent, measurable proxies for Good Things Happening but we could do better and we’re working on that.
Yes, we agree. We’re working on ideas that actually capture the “lead somewhere” part. This might be impact-adjusted connections or, more crudely, “critical connections” (connections that actually cause something good to happen).
Our (pretty scrappy) data suggests that the proportion of attendees who made “professional connections” was about the same in 2021 as previous conferences (~80%). The question for this was an option to select “I made an important professional connection (e.g. a potential career or hiring opportunity, or maybe a potential funder or collaborator)”. This sounds like it captures a lot of what you’d call “senior-junior connections”, but includes some senior-senior connections (funder/collaborator).
I’m sorry you found the conference a bit less valuable yourself. We’re hoping to build out our events portfolio such that more experienced EAs can continue to get a lot of value out of conferences, even as EAGs grow. Do let me know (here or via ollie@eaglobal.org) if you have ideas for this.
I share your scepticism. We already have plans to do this but this is a helpful nudge to make sure it happens!
It’s an interim update, reporting immediate event outcomes, not the extent of our evaluation :)
Thanks for your comment Nuno, we do really appreciate the constructive feedback!
Great to hear!
My read of your comment is that you have a well informed, personal view of many EAGs, and this is driving your skepticism.
I think your perspective and experience on EAG is useful, especially if this can lead to insights that could improve it, or create new narratives that allow us to perceive EA coordination better.
I wanted to write a few questions with this motivation.
I think I find the metric less relevant. (I think it might turn out to be difficult to measure match quality, especially impact, because Goodhart or something. Sometimes simple metrics as a proxy is ok.)
From what I understand, for the qualities of “juniorness” and “promisingness” that you mention, it seems they are two distinct issues or channels for how EAG is getting worse:
The noise or difficulty of matches due to quantity of “juniors” due to volume / communication / coordination costs imposed on seniors, and
The participant quality or “promisingness” decreasing (due to event size).
But I’m not sure 1) is important?
As long as the average junior person is equally promising, you can have equally good senior-junior matches, and the value of the event for a senior person seems unaffected as event size increases from more juniors. (Also, the total value of the event seems non-decreasing as you add more junior people).
If participant quality is the same, I don’t understand why this increases search costs for senior people or something. Or maybe it does for some, or maybe it involves management and is a skill to be learned? I guess I view these as second order effects and I don’t understand why the effect could be very large (I could be wrong).
So I’m speculating that actually 2) “promisingness” is the issue, and so I think the implication or subtext of your comment is that with more people, we’re walking down the quality curve of participants (and pretty quickly. If you are perceiving the change in total value of the EAG as negative from event size increases, this gives us a sense of the slope of participant quality).
Does this sound plausible or reflect your perspective?
I guess I what I am trying to do in this comment is punch through the discussion of the metric, and punch through several confounding things (your seniorness and “search costs”) to get at what I speculate is the essence of your experience, the change in promisingness—which actually is still sort of complicated and has confounding issues itself.
So, if you’re still with me here and this comment isn’t too rambling or confusing, it would be really interesting to hear you talk a lot more about your perspectives on promisingness and these confounding issues.
So if the pool of potential candidates is pretty similar from year to year and there isn’t a lot more going on, simply moving the cutoff criteria could account for your changed experiences, as I think you are saying. (My read is this is the subtext of your comment, that the size of EAG is affecting match quality).
But if the pool of candidates or some other quality of EAG is changing, maybe your experiences has a different explanation?
For example, a explanation might be some effect of a changed selection process, “advertising”, or the recent output of community builders. COVID might have affected the pool too.
Maybe movements get older and the energy changes naturally. This is neither good or bad.
The effect might be vary a lot by cause area, or some small but important community inside a cause area.
Do you have any insights about the above, or can you dismiss any of these speculations out of hand?
Or is this comment just really confusing?
Another topic is the “design of EAG”, or something.
Again, my read is that the subtext of your comment is that the pool is being adversely affected by increasing the event size. Let’s fully accept this perspective.
So, regarding the “total value of the event for altruistic impact”. As touched on above, it’s not clear that decreasing match quality outweighs the value of quantity increases. If you’re straight up doubling the event size, that could allow for some decrease in average match quality.
Like, if EAG was 500 before, and 1,000 now, the “value for the 500 new people” might outweigh the “impact on the original 500 people”.
Obviously, a judgement about change in total value of the event is hard, but maybe you have a good guess or intuition about the sign of the change?
But there’s still more considerations:
EAG acceptance is seen as a badge of ingroupness, and this badge seems valuable for an decentralized movement. For example, being rejected hurts. Obviously, I understand if a dud like me gets culled. But I can think of many people, maybe who aren’t strong networkers or signallers, that EA should value, and many of them are not going to make it into a size 500 event and that seems bad.
You might think that EAG impact is driven by very very talented people, like the next Christiano or SBF and this affects your view of sizing and experience for these people. But there’s several special “programs”, that range from those explicitly run by formal EA orgs, as well as powerful de facto “programs” and personal recommendations, that find develop and connect talent at physical events, including EAGs. These exist among different cause areas. How these programs play out or could be improved in different conceptions of EAG seems pretty relevant.
I’m just some dude writing stuff onto an internet forum, but my guess is that a larger EAG is better given a view of a growing EA.
To the degree you think that previous executions of EAG are inadequate, I’m not sure that this is informative about the future.
It seems bad if a movement couldn’t scale a gathering beyond 500 or 1000 people. It seems like careful, high effort design, maybe new meeting formats or ways to improve match quality, could more than offset the downside of larger event sizes.
Hey, I think that these are all good comments, and I wouldn’t call you “a dud”. I agree with your thoughts around possible cofounders, though a decrease in average participant quality was the most salient explanation to me.