I just want to add, on top of Haydn’s comment to your comment, that:
-
You don’t need the treatment and the control group to be of the same size, so you could, for instance, randomize among the top 300 candidates.
-
In my experience, when there isn’t a clear metric for ordering, it is extremely hard to make clear judgements. Therefore, I think that in practice, it is very likely that let’s say places 100-200 in their ranking seem very similar.
I think that these two factors, combined with Haydn’s suggestion to take the top candidates and exclude them from the study, make it very reasonable, and of very low cost.
I wrote a response post Even More Ambitious Altruistic Tech Efforts, and I would love to spinoff relevant discussion there. The tl;dr is that I think we should have even more ambitious goals, and try to initiate projects that potentially have a very large direct impact (rather than focus on tools and infrastructure for other efforts).
Also, thanks for writing this post Ozzie. Despite my disagreements with your post, I mostly agree with your opinions and think that more attention should be steered towards such efforts.