Thanks for doing this and for sharing the code! It’s really great to have reproducible models – I know how much work it can take to do this, and I appreciate you being willing to go through the labor. I have submitted a couple small PR’s with documentation improvements.
I wanted to better understand what you were doing, so I looked at the NeurIPS Social. If I understand correctly: the value from this social comes from attendees increasing their research relevance from 0.206 to 0.2574, an increase of ~25%.
At first glance, a 25% increase just from a social seems pretty surprising; digging in further: the major difference appears to be this line:
(0.2 * 3 + (2 /​ 10) * 100)
Which I guess means that you are assuming that 103 participants had an increase of 0.2 QARYs. Is that right? If so I would be interested to hear more about where that number came from.[1]
Also, you are dividing all these numbers by 400, which makes me think that the numerator is the total amount of QARY increase, and there were 400 attendees, so you are dividing by 400 to get the average increase, but the parameters you chose for the gamma distribution result in an average number of attendees around 312. So my guess is that one of these numbers is off?
I dug into this a bit more; I am not at all confident that I’m understanding things correctly (my code is here) but I printed out a few sanity checks on the numbers:
Program
Cost (USD)
Benefit (counterfactual expected QARYs)
Cost-effectiveness (QARYs per $1M)
Attendees
Attendees (weighted)
QARYs/​Attendee
Trojan Detection Challenge
65,000
21
320
210
65
0.098
NeurIPS Social
5200
150
29,000
310
680
0.49
NeurIPS Workshop
110,000
360
3200
590
1100
0.6
It continues to feel surprising to me that a social would cause 0.5 QARYs/​attendee, but maybe there’s something I’m missing (e.g. maybe you happen to know about a particularly valuable collaboration which was sparked at this event).
Thanks for doing this and for sharing the code! It’s really great to have reproducible models – I know how much work it can take to do this, and I appreciate you being willing to go through the labor. I have submitted a couple small PR’s with documentation improvements.
I wanted to better understand what you were doing, so I looked at the NeurIPS Social. If I understand correctly: the value from this social comes from attendees increasing their research relevance from 0.206 to 0.2574, an increase of ~25%.
At first glance, a 25% increase just from a social seems pretty surprising; digging in further: the major difference appears to be this line:
Which I guess means that you are assuming that 103 participants had an increase of 0.2 QARYs. Is that right? If so I would be interested to hear more about where that number came from.[1]
Also, you are dividing all these numbers by 400, which makes me think that the numerator is the total amount of QARY increase, and there were 400 attendees, so you are dividing by 400 to get the average increase, but the parameters you chose for the gamma distribution result in an average number of attendees around 312. So my guess is that one of these numbers is off?
You give more information on the NeurIPS workshop here, but I didn’t see anything about the social – sorry if I missed it
I dug into this a bit more; I am not at all confident that I’m understanding things correctly (my code is here) but I printed out a few sanity checks on the numbers:
It continues to feel surprising to me that a social would cause 0.5 QARYs/​attendee, but maybe there’s something I’m missing (e.g. maybe you happen to know about a particularly valuable collaboration which was sparked at this event).