I think that that seems promisingly fast to me, given that this was an early attempt and could probably be sped up (holding quality/​rigour constant) by experience, tools, templates, etc. So that updates me a bit further towards enthusiasm about this general idea.
I’d also note that the larger goals are to scale in non-human ways. If we have a bunch of examples, we could:
1) Open this up to a prediction-market style setup, with a mix of volunteers and possibly inexpensive hires. 2) As we get samples, some people could use data analysis to make simple algorithms to estimate the value of many more documents. 3) We could later use ML and similar to scale this further.
So even if each item were rather time-costly right now, this might be an important step for later. If we can’t even do this, with a lot of work, that would be a significant blocker.
I think that that seems promisingly fast to me, given that this was an early attempt and could probably be sped up (holding quality/​rigour constant) by experience, tools, templates, etc. So that updates me a bit further towards enthusiasm about this general idea.
I’d also note that the larger goals are to scale in non-human ways. If we have a bunch of examples, we could:
1) Open this up to a prediction-market style setup, with a mix of volunteers and possibly inexpensive hires.
2) As we get samples, some people could use data analysis to make simple algorithms to estimate the value of many more documents.
3) We could later use ML and similar to scale this further.
So even if each item were rather time-costly right now, this might be an important step for later. If we can’t even do this, with a lot of work, that would be a significant blocker.
https://​​www.lesswrong.com/​​posts/​​kMmNdHpQPcnJgnAQF/​​prediction-augmented-evaluation-systems