I just want to add, on top of Haydn’s comment to your comment, that:
You don’t need the treatment and the control group to be of the same size, so you could, for instance, randomize among the top 300 candidates.
In my experience, when there isn’t a clear metric for ordering, it is extremely hard to make clear judgements. Therefore, I think that in practice, it is very likely that let’s say places 100-200 in their ranking seem very similar.
I think that these two factors, combined with Haydn’s suggestion to take the top candidates and exclude them from the study, make it very reasonable, and of very low cost.
I just want to add, on top of Haydn’s comment to your comment, that:
You don’t need the treatment and the control group to be of the same size, so you could, for instance, randomize among the top 300 candidates.
In my experience, when there isn’t a clear metric for ordering, it is extremely hard to make clear judgements. Therefore, I think that in practice, it is very likely that let’s say places 100-200 in their ranking seem very similar.
I think that these two factors, combined with Haydn’s suggestion to take the top candidates and exclude them from the study, make it very reasonable, and of very low cost.