I’d be interested to hear roughly how long this whole process took you (or how long it took minus writing the actual post, or something)? This seems relevant to how worthwhile and scalable this sort of thing is.
Maybe an afternoon for the initial version, and then two weeks of occasional tweaks. Say 10h to 30h in total? I imagine that if one wanted to scale this, one could get it to 30 mins to an hour for each estimate.
I think that that seems promisingly fast to me, given that this was an early attempt and could probably be sped up (holding quality/rigour constant) by experience, tools, templates, etc. So that updates me a bit further towards enthusiasm about this general idea.
I’d also note that the larger goals are to scale in non-human ways. If we have a bunch of examples, we could:
1) Open this up to a prediction-market style setup, with a mix of volunteers and possibly inexpensive hires. 2) As we get samples, some people could use data analysis to make simple algorithms to estimate the value of many more documents. 3) We could later use ML and similar to scale this further.
So even if each item were rather time-costly right now, this might be an important step for later. If we can’t even do this, with a lot of work, that would be a significant blocker.
Maybe an afternoon for the initial version, and then two weeks of occasional tweaks. Say 10h to 30h in total? I imagine that if one wanted to scale this, one could get it to 30 mins to an hour for each estimate.
I think that that seems promisingly fast to me, given that this was an early attempt and could probably be sped up (holding quality/rigour constant) by experience, tools, templates, etc. So that updates me a bit further towards enthusiasm about this general idea.
I’d also note that the larger goals are to scale in non-human ways. If we have a bunch of examples, we could:
1) Open this up to a prediction-market style setup, with a mix of volunteers and possibly inexpensive hires.
2) As we get samples, some people could use data analysis to make simple algorithms to estimate the value of many more documents.
3) We could later use ML and similar to scale this further.
So even if each item were rather time-costly right now, this might be an important step for later. If we can’t even do this, with a lot of work, that would be a significant blocker.
https://www.lesswrong.com/posts/kMmNdHpQPcnJgnAQF/prediction-augmented-evaluation-systems