Why wouldn’t randomisation be a good fit for EAG? I suspect that the ability of the organisers to finely distinguish between similarly-promising applicants is minimal anyway*, so a strategy of, say, roughly scoring applicants into buckets and then randomising among those who fall between “obvious shoo-in” and “clearly unsuitable” could work quite well, as well as being much quicker and easier for the organisers.
(This is roughly how the proposal to randomise scientific grantmaking would work: apply a basic check for suitability/competence and then randomise among those who make that cut. I think this would be a big improvement over the current system and would apply the same reasoning in many other domains with similar features, such as university admissions.)
* Not because I have a low opinion of the organisers, just because I think this is generally true.
If time is an issue, organisers can make quick snap judgements. It’s not clear to me that randomisation would be much faster, particularly since you anyway have to make a first rough scoring on your approach. And it seems reasonable, in my view, that organisers are better than chance at picking the better applicants, even when using snap judgements, and even among applicants in the same bucket.
I’m confused why you don’t think randomisation would be faster than producing a complete ranking of candidates, but I also don’t currently have reason to think the ranking is a limiting factor, so unless we get information to the contrary this isn’t the main point of contention.
More importantly, I think we disagree on the last sentence. I think snap judgements between candidates that don’t clearly differ dramatically in suitability are likely to be not significantly better than, and possibly worse than, chance.
You don’t have to provide a complete ranking of candidates. You only have to decide which candidates to accept and which not to in the bucket that you would prefer to randomise. And it seems to me that such decisions could in principle be made extremely quickly, particularly since you must already have assimilated some information about the candidates in order to put them in the right bucket (though speed probably affects quality adversely; but I still think some signal will remain).
Thanks, this is worth thinking about. Among similar applicants (e.g. students from the same university group) we’re choosing fairly arbitrarily but we’ve never tried true randomization.
Why wouldn’t randomisation be a good fit for EAG? I suspect that the ability of the organisers to finely distinguish between similarly-promising applicants is minimal anyway*, so a strategy of, say, roughly scoring applicants into buckets and then randomising among those who fall between “obvious shoo-in” and “clearly unsuitable” could work quite well, as well as being much quicker and easier for the organisers.
(This is roughly how the proposal to randomise scientific grantmaking would work: apply a basic check for suitability/competence and then randomise among those who make that cut. I think this would be a big improvement over the current system and would apply the same reasoning in many other domains with similar features, such as university admissions.)
* Not because I have a low opinion of the organisers, just because I think this is generally true.
If time is an issue, organisers can make quick snap judgements. It’s not clear to me that randomisation would be much faster, particularly since you anyway have to make a first rough scoring on your approach. And it seems reasonable, in my view, that organisers are better than chance at picking the better applicants, even when using snap judgements, and even among applicants in the same bucket.
I’m confused why you don’t think randomisation would be faster than producing a complete ranking of candidates, but I also don’t currently have reason to think the ranking is a limiting factor, so unless we get information to the contrary this isn’t the main point of contention.
More importantly, I think we disagree on the last sentence. I think snap judgements between candidates that don’t clearly differ dramatically in suitability are likely to be not significantly better than, and possibly worse than, chance.
You don’t have to provide a complete ranking of candidates. You only have to decide which candidates to accept and which not to in the bucket that you would prefer to randomise. And it seems to me that such decisions could in principle be made extremely quickly, particularly since you must already have assimilated some information about the candidates in order to put them in the right bucket (though speed probably affects quality adversely; but I still think some signal will remain).
Thanks, this is worth thinking about. Among similar applicants (e.g. students from the same university group) we’re choosing fairly arbitrarily but we’ve never tried true randomization.