Yeah, I agree that for forecasting setups self-fulfilling prophesies/feedback loops can be a problem, but it seems likely that they can be mitigated with a certain amount of exploration (e.g., occasionally try things you’d expect to fail in order to test your system.)
It’s also not clear that this type of evaluation is worse than the alternative informal evaluation system. For example, with a formal evaluation system you’d be able to pick up high quality outputs even if they come from otherwise low status people (and then give them further support.)
Thanks for the thoughtful answer. I agree that it’s not clear that it is worse than other alternatives, in my comment I didn’t give a reference solution to compare it to after all.
I just wanted to highlight the potential for problems that ought to be looked at while designing such solutions. So, if you consider working more on this in the future, it might be fruitful to think about how it would influence such feedback loops.
Yeah, I agree that for forecasting setups self-fulfilling prophesies/feedback loops can be a problem, but it seems likely that they can be mitigated with a certain amount of exploration (e.g., occasionally try things you’d expect to fail in order to test your system.)
It’s also not clear that this type of evaluation is worse than the alternative informal evaluation system. For example, with a formal evaluation system you’d be able to pick up high quality outputs even if they come from otherwise low status people (and then give them further support.)
Thanks for the thoughtful answer. I agree that it’s not clear that it is worse than other alternatives, in my comment I didn’t give a reference solution to compare it to after all.
I just wanted to highlight the potential for problems that ought to be looked at while designing such solutions. So, if you consider working more on this in the future, it might be fruitful to think about how it would influence such feedback loops.