EA Global London 2021 took place last week (29th-31st October 2021). As with all EA Globals, it was in most respects an amazing conference, far above the standard I expect from other similar events. However, it also took place in the midst of a major COVID wave in the UK, with major concerns about vaccine escape and long COVID (including for vaccinated people). Many people I spoke to were concerned about the COVID risks posed by attending the conference, and I know of at least a few high-value members of the community who declined to attend for this reason.
The event had been cancelled or rescheduled multiple times in the past, and this iteration was originally intended to be relatively small, about 500 people. However, after many more people applied than there were places available, the decision was made to roughly double the size of the event. This change was made relatively last-minute, being announced about a month prior to the conference. I believe this event was the largest EA Global to take place for a number of years.
I was concerned about the decision to increase the size of EAG, and especially unhappy with the last-minute nature of the change, the low level of transparency around the decision process and reasoning involved, and the lack of apparent compensation mechanisms to reduce COVID risk in the face of so many more people. When I shared an earlier (and more aggressive) version of this post with CEA prior to the conference, I received a number of corrections, clarifications, and counter-arguments – but almost none of these were in the public domain. Even though the conference is now past, I think it’s important to have a public conversation about these issues, and perhaps help avoid similar issues with future events.
At present, I am highly uncertain whether or not the event should have been expanded last-minute as it was. At the very least, I think it’s not-blindingly-obvious that this was the correct decision, and if CEA had good evidence that it was I strongly believe they should have shared it publicly (and should still). In general, my discussions with CEA about this have updated me about the situation in some important ways, but I still believe that many of my arguments below are correct:
[Medium confidence] The decision to expand was based on unsurprising information that should have been taken into account in advance.
EA Global conferences have been routinely majorly oversubscribed for years now, especially since the decision was made some years ago to deliberately reduce their size. Given this, the fact that there was a large waiting list for this conference was not at all surprising; I think I would have given an ex ante probability of over 90%. I’d be very surprised if the events team wasn’t also expecting this.
The feedback I’ve received from CEA on this point is that, while the mere fact of a waitlist wasn’t necessarily surprising, the quality of the people in that waitlist was much higher than anticipated. This is consistent with the various metric updates from CEA’s Q3 updates; however, as such I think it should still not have been hugely surprising (though hindsight is of course 20⁄20).
(I previously had some discussion of COVID news in September here, but I’ve cut it since it wasn’t raised by CEA as a major factor in their responses to me.)
Based on communication with CEA, it is also my understanding that many of the most promising applicants only applied after the initial early-bird deadline – and thus after many places at the conference had already been allocated, making the need for expansion much more pressing. I don’t see any reason to doubt this – but it suggests to me that CEA’s systems for deciding how and when to allocate conference places was quite badly flawed.
[High confidence] The decision to expand the event was underjustified and untransparent.
As with most COVID-related decisions made by the CEA events team over the course of the pandemic, the change in policy was justified solely by an appeal to authority – namely, that they consulted with their COVID advisory board. Although my understanding is that more people were consulted, no further justification or discussion was made available.
In particular, no estimate was given of how much this change in policy would alter the risk of the event – the expansion email advised attendees to use microCOVID to help estimate their risk, but didn’t provide any modelling or estimates they had made themselves as part of making this decision.
When one prominent Forum user (Saulius) asked for estimates of the risk, this went unresponded to for over a week. And in my discussions with CEA over this issue, I have yet to be linked to any pre-existing in-house models or cost-benefit estimates CEA used to make this decision.
In general, it’s currently my impression that CEA did relatively little in the way of in-house, quantitative cost-benefit modelling before making this decision. This is very surprising to me – the least I would expect to see as part of a decision like this is an estimate of how it would affect the COVID exposure of attendees, and once such an estimate had been made I’m quite confident it should have been shared with the community.
I’m also fairly confident that more transparency around the decision would have enabled others in the community to make, argue about, and refine these models, giving the community as a whole a much better idea of the relative costs and benefits of attending the conference.
The above is a specific version of a general charge, which is that this decision, and other important COVID-related decisions made by the CEA events team, have been made in an untransparent and uncommunicative manner that (i) makes those decisions worse in expectation, and (ii) makes it difficult for the wider community to evaluate those decisions and make good decisions themselves about how to respond. Apart from those few people privileged to have close links with CEA or the Oxford office, most attendees didn’t have access to the reasoning used to make the decision to expand. The arguments and data used to make that consequential decision should have been made available to the community, so that we could evaluate whether we found them convincing, challenge them if not, and update our beliefs and plans accordingly. This was not done.
(This does not necessarily imply that the decision to expand or not expand should have been left up to to the community – this doesn’t sound great to me for various reasons. Rather, I think CEA should have made its decision, then shared the in-depth reasoning for that decision with the community. I would make similar claims for all COVID-related events decisions by CEA.)
[Low confidence] The decision to expand EA Global was inconsistent with CEA’s previous claims about conference size.
As mentioned above, EA Global conferences have been oversubscribed for many years now, in part because of a decision to intentionally reduce the size of the conferences: “The largest EA Global was about 1000 people in 2016, and we got feedback that it was too big and that it was easy to get lost in the shuffle. Our recent events have been between 500 − 650 people including speakers, volunteers, and staff.”
While I was never super confident in these arguments, they seem sufficient to make expanding the event ambiguously good even in the absence of COVID; it seems especially strange to reverse this policy at the time when the downside risk of doing so is drastically larger than it was in the past.
My understanding now is that there are various reasons why CEA no longer believes these arguments – for example, the community is now larger, the number of high-quality applicants is higher, and software for connecting attendees and facilitating a good conference experience has improved.
These seem like reasonable arguments to me – but I only learned about them by sharing a draft of this post with CEA! The decision to expand EAG London 2021 would have been a great opportunity to lay out those updates for the community, but this was not done.
[Medium confidence] CEA undercompensated for the increase in risk.
Based on my communications with CEA, I believe they did take some steps to increase the safety of the venue in response to the expansion, including communicating with the venue about increasing ventilation and setting up an additional marquee.
These were good changes! The increase in ventilation in particular seems great to me.
However, at the event itself I didn’t get the sense that the event organisers were trying especially hard to reduce COVID risks from the event.
The marquees, for example, were very well-sealed and robust, which is good for keeping atteendees warm but not for improving safety. One of the marquees had a large flap at the front which I and some others opened on Friday afternoon; however, this was closed on Friday night and never re-opened. Most windows at the event were also kept shut throughout the event, even when the weather was good. And as far as I know, no real effort was made to make sure attendees were getting tested – probably the most valuable intervention in terms of reducing risk without impairing the conference experience.
(I’m not including masks here, because I think requiring masks would have made the experience of the event for many attendees much worse, which doesn’t seem worth it to me.)
(In the event, the weather was better than forecast and I was able to have a lot of meetings outside, circumventing some of these concerns. But that was sheer luck.)
My loose impression from communicating with various people at or close to CEA is that they (or at least some of them) were operating under a model where the expected costs from COVID just weren’t really something to worry too much about, relative to the other costs of operating such an event, and as such not worth putting too much effort or resources into mitigating. This explains many of my observations above, and might even be true – however, I don’t think this was made at all clear to the community, and I think a number of attendees would have been surprised, and perhaps alarmed (not necessarily correctly!), to learn this was the case.
[Fairly high confidence] Expanding after the fact was substantially worse than deciding on a larger conference from the beginning.
Upon making the decision to expand the event, CEA offered to allow those who had signed up when it was due to have an attendance of 500 a free refund if they decided they didn’t want to attend, as well as compensation for travel and accommodation. While this is all good, and I admit I’m not sure what else they should have done at that point given their other decisions, it’s far from a great solution.
People are generally much less likely to cancel something they no longer think is a good idea, than to book it in the first place. I think these sorts of psychological lock-in effects will have led to many less people deciding not to come than might, on deep reflection, have done so. Even if they did do so, overcoming these biases is much more costly in terms of time and effort than deciding not to attend ex ante would have been; in the words of Algorithms to Live By, it was computationally unkind.
All these effects are exacerbated if they also have to cancel & seek compensation for travel & accommodation.
Given this, being transparent about the reasoning underlying the decision was especially important.
In addition, only deciding to expand the event a month in advance reduced CEA’s (and other actors’) ability to make changes to the event to compensate for the increased size. (Though as discussed above, I don’t think they took full advantage of the opportunities they did have, so possibly this wouldn’t have mattered too much.)
For these and other reasons, I think changing the size of the event like this was significantly worse than deciding in advance on the size of the event and sticking to it.
In almost all things non-COVID, I believe that the CEA events team are some of the best events organisers in the world. They consistently put on events that far exceed the quality of any other conference series I have experience of, and EA Global London 2021 continued that trend. I attended the conference, and am confident it was good for the world for me to do so. But I wish the decision to expand the conference had been made more carefully, more transparently, and sooner.
Should EA Global London 2021 have been expanded?
EA Global London 2021 took place last week (29th-31st October 2021). As with all EA Globals, it was in most respects an amazing conference, far above the standard I expect from other similar events. However, it also took place in the midst of a major COVID wave in the UK, with major concerns about vaccine escape and long COVID (including for vaccinated people). Many people I spoke to were concerned about the COVID risks posed by attending the conference, and I know of at least a few high-value members of the community who declined to attend for this reason.
The event had been cancelled or rescheduled multiple times in the past, and this iteration was originally intended to be relatively small, about 500 people. However, after many more people applied than there were places available, the decision was made to roughly double the size of the event. This change was made relatively last-minute, being announced about a month prior to the conference. I believe this event was the largest EA Global to take place for a number of years.
I was concerned about the decision to increase the size of EAG, and especially unhappy with the last-minute nature of the change, the low level of transparency around the decision process and reasoning involved, and the lack of apparent compensation mechanisms to reduce COVID risk in the face of so many more people. When I shared an earlier (and more aggressive) version of this post with CEA prior to the conference, I received a number of corrections, clarifications, and counter-arguments – but almost none of these were in the public domain. Even though the conference is now past, I think it’s important to have a public conversation about these issues, and perhaps help avoid similar issues with future events.
At present, I am highly uncertain whether or not the event should have been expanded last-minute as it was. At the very least, I think it’s not-blindingly-obvious that this was the correct decision, and if CEA had good evidence that it was I strongly believe they should have shared it publicly (and should still). In general, my discussions with CEA about this have updated me about the situation in some important ways, but I still believe that many of my arguments below are correct:
[Medium confidence] The decision to expand was based on unsurprising information that should have been taken into account in advance.
EA Global conferences have been routinely majorly oversubscribed for years now, especially since the decision was made some years ago to deliberately reduce their size. Given this, the fact that there was a large waiting list for this conference was not at all surprising; I think I would have given an ex ante probability of over 90%. I’d be very surprised if the events team wasn’t also expecting this.
The feedback I’ve received from CEA on this point is that, while the mere fact of a waitlist wasn’t necessarily surprising, the quality of the people in that waitlist was much higher than anticipated. This is consistent with the various metric updates from CEA’s Q3 updates; however, as such I think it should still not have been hugely surprising (though hindsight is of course 20⁄20).
(I previously had some discussion of COVID news in September here, but I’ve cut it since it wasn’t raised by CEA as a major factor in their responses to me.)
Based on communication with CEA, it is also my understanding that many of the most promising applicants only applied after the initial early-bird deadline – and thus after many places at the conference had already been allocated, making the need for expansion much more pressing. I don’t see any reason to doubt this – but it suggests to me that CEA’s systems for deciding how and when to allocate conference places was quite badly flawed.
[High confidence] The decision to expand the event was underjustified and untransparent.
As with most COVID-related decisions made by the CEA events team over the course of the pandemic, the change in policy was justified solely by an appeal to authority – namely, that they consulted with their COVID advisory board. Although my understanding is that more people were consulted, no further justification or discussion was made available.
In particular, no estimate was given of how much this change in policy would alter the risk of the event – the expansion email advised attendees to use microCOVID to help estimate their risk, but didn’t provide any modelling or estimates they had made themselves as part of making this decision.
When one prominent Forum user (Saulius) asked for estimates of the risk, this went unresponded to for over a week. And in my discussions with CEA over this issue, I have yet to be linked to any pre-existing in-house models or cost-benefit estimates CEA used to make this decision.
In general, it’s currently my impression that CEA did relatively little in the way of in-house, quantitative cost-benefit modelling before making this decision. This is very surprising to me – the least I would expect to see as part of a decision like this is an estimate of how it would affect the COVID exposure of attendees, and once such an estimate had been made I’m quite confident it should have been shared with the community.
I’m also fairly confident that more transparency around the decision would have enabled others in the community to make, argue about, and refine these models, giving the community as a whole a much better idea of the relative costs and benefits of attending the conference.
The above is a specific version of a general charge, which is that this decision, and other important COVID-related decisions made by the CEA events team, have been made in an untransparent and uncommunicative manner that (i) makes those decisions worse in expectation, and (ii) makes it difficult for the wider community to evaluate those decisions and make good decisions themselves about how to respond. Apart from those few people privileged to have close links with CEA or the Oxford office, most attendees didn’t have access to the reasoning used to make the decision to expand. The arguments and data used to make that consequential decision should have been made available to the community, so that we could evaluate whether we found them convincing, challenge them if not, and update our beliefs and plans accordingly. This was not done.
(This does not necessarily imply that the decision to expand or not expand should have been left up to to the community – this doesn’t sound great to me for various reasons. Rather, I think CEA should have made its decision, then shared the in-depth reasoning for that decision with the community. I would make similar claims for all COVID-related events decisions by CEA.)
[Low confidence] The decision to expand EA Global was inconsistent with CEA’s previous claims about conference size.
As mentioned above, EA Global conferences have been oversubscribed for many years now, in part because of a decision to intentionally reduce the size of the conferences: “The largest EA Global was about 1000 people in 2016, and we got feedback that it was too big and that it was easy to get lost in the shuffle. Our recent events have been between 500 − 650 people including speakers, volunteers, and staff.”
While I was never super confident in these arguments, they seem sufficient to make expanding the event ambiguously good even in the absence of COVID; it seems especially strange to reverse this policy at the time when the downside risk of doing so is drastically larger than it was in the past.
My understanding now is that there are various reasons why CEA no longer believes these arguments – for example, the community is now larger, the number of high-quality applicants is higher, and software for connecting attendees and facilitating a good conference experience has improved.
These seem like reasonable arguments to me – but I only learned about them by sharing a draft of this post with CEA! The decision to expand EAG London 2021 would have been a great opportunity to lay out those updates for the community, but this was not done.
[Medium confidence] CEA undercompensated for the increase in risk.
Based on my communications with CEA, I believe they did take some steps to increase the safety of the venue in response to the expansion, including communicating with the venue about increasing ventilation and setting up an additional marquee.
These were good changes! The increase in ventilation in particular seems great to me.
However, at the event itself I didn’t get the sense that the event organisers were trying especially hard to reduce COVID risks from the event.
The marquees, for example, were very well-sealed and robust, which is good for keeping atteendees warm but not for improving safety. One of the marquees had a large flap at the front which I and some others opened on Friday afternoon; however, this was closed on Friday night and never re-opened. Most windows at the event were also kept shut throughout the event, even when the weather was good. And as far as I know, no real effort was made to make sure attendees were getting tested – probably the most valuable intervention in terms of reducing risk without impairing the conference experience.
(I’m not including masks here, because I think requiring masks would have made the experience of the event for many attendees much worse, which doesn’t seem worth it to me.)
(In the event, the weather was better than forecast and I was able to have a lot of meetings outside, circumventing some of these concerns. But that was sheer luck.)
My loose impression from communicating with various people at or close to CEA is that they (or at least some of them) were operating under a model where the expected costs from COVID just weren’t really something to worry too much about, relative to the other costs of operating such an event, and as such not worth putting too much effort or resources into mitigating. This explains many of my observations above, and might even be true – however, I don’t think this was made at all clear to the community, and I think a number of attendees would have been surprised, and perhaps alarmed (not necessarily correctly!), to learn this was the case.
[Fairly high confidence] Expanding after the fact was substantially worse than deciding on a larger conference from the beginning.
Upon making the decision to expand the event, CEA offered to allow those who had signed up when it was due to have an attendance of 500 a free refund if they decided they didn’t want to attend, as well as compensation for travel and accommodation. While this is all good, and I admit I’m not sure what else they should have done at that point given their other decisions, it’s far from a great solution.
People are generally much less likely to cancel something they no longer think is a good idea, than to book it in the first place. I think these sorts of psychological lock-in effects will have led to many less people deciding not to come than might, on deep reflection, have done so. Even if they did do so, overcoming these biases is much more costly in terms of time and effort than deciding not to attend ex ante would have been; in the words of Algorithms to Live By, it was computationally unkind.
All these effects are exacerbated if they also have to cancel & seek compensation for travel & accommodation.
Given this, being transparent about the reasoning underlying the decision was especially important.
In addition, only deciding to expand the event a month in advance reduced CEA’s (and other actors’) ability to make changes to the event to compensate for the increased size. (Though as discussed above, I don’t think they took full advantage of the opportunities they did have, so possibly this wouldn’t have mattered too much.)
For these and other reasons, I think changing the size of the event like this was significantly worse than deciding in advance on the size of the event and sticking to it.
In almost all things non-COVID, I believe that the CEA events team are some of the best events organisers in the world. They consistently put on events that far exceed the quality of any other conference series I have experience of, and EA Global London 2021 continued that trend. I attended the conference, and am confident it was good for the world for me to do so. But I wish the decision to expand the conference had been made more carefully, more transparently, and sooner.