In my view the basic problem with this analysis is you probably can’t lump all the camps together as one thing and evaluate them together as one entity. Format, structure, leadership and participants seem to have been very different.
Yes, we were particularly concerned with the fact that earlier camps were in-person and likely had a stronger selection bias for people interested in AIS (due to AI/AIS being more niche at the time) as well as a geographic selection bias. That’s why I have more trust in the participant tracking data for camps 4-6 which were more recent, virtual and had a more consistent format.
Since AISC 8 is so big, it will be interesting to re-do this analysis with a single group under the same format and degree of selection.
When producing the main estimates, Sam already uses just the virtual camps, for this reason. Could emphasise more that this probably doesn’t generalise.
The key thing about AISC for me was probably the “hero licence” (social encouragement, uncertainty reduction) the camp gave me. I imagine this specific impact works 20x better in person. I don’t know how many attendees need any such thing (in my cohort, maybe 25%) or what impact adjustment to give this type of attendee (probably a discount, since independence and conviction is so valuable in a lot of research).
Another wrinkle is the huge difference in acceptance rates between programmes. IIRC the admission rate for AISC 2018 was 80% (only possible because of the era’s heavy self-selection for serious people, as Sam notes). IIRC, 2023 MATS is down around ~3%. Rejections have some cost for applicants, mostly borne by the highly uncertain ones who feel they need licencing. So this is another way AISC and MATS aren’t doing the same thing, and so I wouldn’t directly compare them (without noting this). Someone should be there to catch ~80% of seriously interested people. So, despite appearances, AGISF is a better comparison for AISC on this axis.
In my view the basic problem with this analysis is you probably can’t lump all the camps together as one thing and evaluate them together as one entity. Format, structure, leadership and participants seem to have been very different.
Yes, we were particularly concerned with the fact that earlier camps were in-person and likely had a stronger selection bias for people interested in AIS (due to AI/AIS being more niche at the time) as well as a geographic selection bias. That’s why I have more trust in the participant tracking data for camps 4-6 which were more recent, virtual and had a more consistent format.
Since AISC 8 is so big, it will be interesting to re-do this analysis with a single group under the same format and degree of selection.
When producing the main estimates, Sam already uses just the virtual camps, for this reason. Could emphasise more that this probably doesn’t generalise.
The key thing about AISC for me was probably the “hero licence” (social encouragement, uncertainty reduction) the camp gave me. I imagine this specific impact works 20x better in person. I don’t know how many attendees need any such thing (in my cohort, maybe 25%) or what impact adjustment to give this type of attendee (probably a discount, since independence and conviction is so valuable in a lot of research).
Another wrinkle is the huge difference in acceptance rates between programmes. IIRC the admission rate for AISC 2018 was 80% (only possible because of the era’s heavy self-selection for serious people, as Sam notes). IIRC, 2023 MATS is down around ~3%. Rejections have some cost for applicants, mostly borne by the highly uncertain ones who feel they need licencing. So this is another way AISC and MATS aren’t doing the same thing, and so I wouldn’t directly compare them (without noting this). Someone should be there to catch ~80% of seriously interested people. So, despite appearances, AGISF is a better comparison for AISC on this axis.