While I agree with a lot in this post, I do want to push back on this reasoning:
About 500 more people who met the admissions bar would get to experience the conference. However much impact the default conference would produce — sparked collaborations, connections for years to come, inspirations for new projects, positive career changes — there would be about double that
I think an estimate of “double that” is pretty wrong. I think the first 500 people who would be admitted would of course be selected for getting value out of the conference, and I expect the value that different people gain to be heavy-tailed. It is hard to predict who exactly will get value out of a conference, but it wouldn’t surprise me if you get to a state where you capture 90% of the value by admitting the right 500 people.
On the other hand, I think a conference might produce value in the square of the number of the participants, since people can self sort, and meeting more people is more valuable than meeting less people.
I think in one line of reasoning you get something like “a conference twice the size would be maybe 10-20% more valuable” and in the other line of reasoning you get “a conference twice the size could be 4x as valuable”, but I don’t have any line of reasoning I endorse that outputs the 2x number.
Thanks for the pushback. I agree that a linear model will be importantly wrong, although if you approximate the impact from the conference using the number of connections people report and assume that stays roughly the same, it doesn’t seem wild as a first pass. (Please let me know if you disagree!)
[Half-formed thoughts below.]
On the other hand, I think 10-20% more valuable seems very off to me, especially in this case, given we were not “lowering the bar” for the second group of attendees. Setting this case aside, I can imagine a world in which someone is very confident in their ability to admit the people who will benefit the most from a conference (and the people who would be most useful for them to meet with), and in this world, you might be able to get 90% of the value with 50% of the size — but I don’t really think we’re in this world (especially in terms of identifying people who will benefit most from the event).
I’m not really sure how well people self-sort at conferences, which was a big uncertainty for me when I was thinking about these things more. I do think people will often identify (often with help) some of the people with whom it would be most useful to meet. If people are good at self-sorting (e.g. searching through swapcard and finding the most promising 10-15 meetings), and if those most-useful meetings over the whole conference aren’t somehow concentrated on meetings with a small number of nodes, then admitting double the people will likely lead to more than double the impact.[1] If people are not good at self-sorting, though, it seems more likely that we’d get closer to straightforward doubling, I think. (I’m fairly confident that people are better than random, though.)
It does seem possible that there are some “nodes” in the network — at a very bad first pass, you could imagine that everyone’s most valuable meetings are with the speakers. The speakers each meet with lots of people (say, they have lots of time and don’t get tired) and would be at the conference in any world (doubling or not). Then the addition of 500 extra people doesn’t significantly improve the set of possible meetings for the 500 first attendees, although 500 extra people get to meet with the speakers (which is nearly all that matters in this model).
I’m really unsure about the extent to which the “nodes” thing is true (and if it’s true I don’t really think that “speakers” are the right group), but there’s something here that seems like it could be right given what we hear. There’s also the added nuance that some nodes are probably in the second group of 500, and also that the size and capacity for meetings of the “nodes” group would matter.
Hmm, I do really think there is a very wrong intuition here. I think by-default in most situations the return to doubling a specific resource should be modeled as logarithmic (i.e. the first doubling is as valuable as the second doubling). I think in this model, it is very rare that doubling a thing along any specific dimension produces twice the value. I think the value of marginally more people in EA should likely also be modeled as having logarithmic returns (or I might argue worse than logarithmic returns, but I think logarithmic is the right prior).
I think you will get estimates wrong by many orders of magnitude if you do reasoning of the type “if I just double this resource that will double the whole value of the event”, unless you have a strong argument for network effects.
I wonder how much of your intuition comes from thinking that marginal (ex ante) impact of marginal EAG attendees is much lower than the existing average, vs normal logarithmic prior considerations vs how much of it comes from diseases of scale (e.g. higher population making things harder to coordinate, pressure towards conformity).
The first consideration is especially interesting to isolate, since:
I think the value of marginally more people in EA should likely also be modeled as having logarithmic returns (or I might argue worse than logarithmic returns, but I think logarithmic is the right prior
If you think doubling the quality-adjusted people in EA overall has logarithmic returns, you still get ~linear effects from doubling the output of one event or outreach project, since differential functions are locally linear.
I would say the square in the number of participants is too extreme, since the average attendee probably wouldn’t meet many more people than otherwise, except for those who wouldn’t have gotten to attend at all.
(EDIT, nvm this bit; I don’t know how to strike it out via mobile: Plus, because people are going to meet based on interests, if you were thinking about the number of possible meetings, I think it would be better to think about it like multiple cliques doubling in size than a single large clique doubling in size, or something more complicated.)
The first 500 (except those hiring?) probably wouldn’t get much more out of it, since it at most only slightly adds to who they might have met in terms of counterfactual value, and they might even get less, since they need to compete with the new 500 over meetings with the first 500. Then the next 500 at least get to meet each other, but also the first 500, and especially the (I assume) roughly fixed number of organizations that are hiring.
The first 500 is also plausibly made up of many people who are already largely in contact with one another because they work at EA-related orgs, either the same org, or orgs working in the same area who review each other’s work, strategize together or collaborate.
Around 2x seems plausible to me, but my best guess is less than 2x.
Sorry, I think you could arrive at 2x for bayesian reasons (like weighing multiple models), but I just wanted to push back on the model that an event with twice as many attendees should be straightforwardly modeled as twice as valuable.
I agree that it’s not straightforward that a linear model is approximately correct. I do think a linear model could still be approximately correct for straightforward linear reasons, like the value being roughly proportional to the number of one-on-ones, though, and not just because you weighed multiple models together and it happened to come out to about 2x.
While I agree with a lot in this post, I do want to push back on this reasoning:
I think an estimate of “double that” is pretty wrong. I think the first 500 people who would be admitted would of course be selected for getting value out of the conference, and I expect the value that different people gain to be heavy-tailed. It is hard to predict who exactly will get value out of a conference, but it wouldn’t surprise me if you get to a state where you capture 90% of the value by admitting the right 500 people.
On the other hand, I think a conference might produce value in the square of the number of the participants, since people can self sort, and meeting more people is more valuable than meeting less people.
I think in one line of reasoning you get something like “a conference twice the size would be maybe 10-20% more valuable” and in the other line of reasoning you get “a conference twice the size could be 4x as valuable”, but I don’t have any line of reasoning I endorse that outputs the 2x number.
Thanks for the pushback. I agree that a linear model will be importantly wrong, although if you approximate the impact from the conference using the number of connections people report and assume that stays roughly the same, it doesn’t seem wild as a first pass. (Please let me know if you disagree!)
[Half-formed thoughts below.]
On the other hand, I think 10-20% more valuable seems very off to me, especially in this case, given we were not “lowering the bar” for the second group of attendees. Setting this case aside, I can imagine a world in which someone is very confident in their ability to admit the people who will benefit the most from a conference (and the people who would be most useful for them to meet with), and in this world, you might be able to get 90% of the value with 50% of the size — but I don’t really think we’re in this world (especially in terms of identifying people who will benefit most from the event).
I’m not really sure how well people self-sort at conferences, which was a big uncertainty for me when I was thinking about these things more. I do think people will often identify (often with help) some of the people with whom it would be most useful to meet. If people are good at self-sorting (e.g. searching through swapcard and finding the most promising 10-15 meetings), and if those most-useful meetings over the whole conference aren’t somehow concentrated on meetings with a small number of nodes, then admitting double the people will likely lead to more than double the impact.[1] If people are not good at self-sorting, though, it seems more likely that we’d get closer to straightforward doubling, I think. (I’m fairly confident that people are better than random, though.)
It does seem possible that there are some “nodes” in the network — at a very bad first pass, you could imagine that everyone’s most valuable meetings are with the speakers. The speakers each meet with lots of people (say, they have lots of time and don’t get tired) and would be at the conference in any world (doubling or not). Then the addition of 500 extra people doesn’t significantly improve the set of possible meetings for the 500 first attendees, although 500 extra people get to meet with the speakers (which is nearly all that matters in this model).
I’m really unsure about the extent to which the “nodes” thing is true (and if it’s true I don’t really think that “speakers” are the right group), but there’s something here that seems like it could be right given what we hear. There’s also the added nuance that some nodes are probably in the second group of 500, and also that the size and capacity for meetings of the “nodes” group would matter.
Hmm, I do really think there is a very wrong intuition here. I think by-default in most situations the return to doubling a specific resource should be modeled as logarithmic (i.e. the first doubling is as valuable as the second doubling). I think in this model, it is very rare that doubling a thing along any specific dimension produces twice the value. I think the value of marginally more people in EA should likely also be modeled as having logarithmic returns (or I might argue worse than logarithmic returns, but I think logarithmic is the right prior).
I think you will get estimates wrong by many orders of magnitude if you do reasoning of the type “if I just double this resource that will double the whole value of the event”, unless you have a strong argument for network effects.
I wonder how much of your intuition comes from thinking that marginal (ex ante) impact of marginal EAG attendees is much lower than the existing average, vs normal logarithmic prior considerations vs how much of it comes from diseases of scale (e.g. higher population making things harder to coordinate, pressure towards conformity).
The first consideration is especially interesting to isolate, since:
If you think doubling the quality-adjusted people in EA overall has logarithmic returns, you still get ~linear effects from doubling the output of one event or outreach project, since differential functions are locally linear.
I would say the square in the number of participants is too extreme, since the average attendee probably wouldn’t meet many more people than otherwise, except for those who wouldn’t have gotten to attend at all.
(EDIT, nvm this bit; I don’t know how to strike it out via mobile: Plus, because people are going to meet based on interests, if you were thinking about the number of possible meetings, I think it would be better to think about it like multiple cliques doubling in size than a single large clique doubling in size, or something more complicated.)
The first 500 (except those hiring?) probably wouldn’t get much more out of it, since it at most only slightly adds to who they might have met in terms of counterfactual value, and they might even get less, since they need to compete with the new 500 over meetings with the first 500. Then the next 500 at least get to meet each other, but also the first 500, and especially the (I assume) roughly fixed number of organizations that are hiring.
The first 500 is also plausibly made up of many people who are already largely in contact with one another because they work at EA-related orgs, either the same org, or orgs working in the same area who review each other’s work, strategize together or collaborate.
Around 2x seems plausible to me, but my best guess is less than 2x.
Sorry, I think you could arrive at 2x for bayesian reasons (like weighing multiple models), but I just wanted to push back on the model that an event with twice as many attendees should be straightforwardly modeled as twice as valuable.
I agree that it’s not straightforward that a linear model is approximately correct. I do think a linear model could still be approximately correct for straightforward linear reasons, like the value being roughly proportional to the number of one-on-ones, though, and not just because you weighed multiple models together and it happened to come out to about 2x.