I also don’t see how complex discussion on the forum with the high quality reviews you imagine would cost 5 hours.
I think an initial version of the process, in which you plus maybe one or two close collaborators, would play the role of evaluators and participate in an EA Forum thread, would take less than 5 hours to set up and less than 15 hours of time to actually execute and write reviews on, and I think would give you significant evidence about what kind of evaluations will be valuable and what the current bottlenecks in this space are.
I would be curious about you model why the open discussion we currently have does not work well—like here, where user nonzerosum proposed a project, the post was heavily downvoted (at some point to negative karma) without substantial discussion of the problems.
I think that post is actually a good example of why a multi-stage process like this will cause a lot of problems. I think the best thing for nonzerosum to do would have been to create a short comment or post, maybe two to three paragraphs, in which he explained the basic idea of a donor list. At this point, he would have not been super invested in it, and I think if he had posted only a short document, people would have reacted with openness and told him that there has been a pretty long history of people trying to make lots of EA donor coordination platforms, and that there are significant problems with unilateralist curse-like problems. I think the downvotes and negative reaction came primarily from people perceiving him to be prematurely charging ahead with a project.
I do think you need some additional incentive for people to actually write up their thoughts in addition to just voting on stuff, which is why a volunteer evaluator group, or maybe some kind of financial incentive, or maybe just some kind of modifications to the forum software (which I recognize is not something you can easily do but which I have affordances for), is a good idea. But I do think you want to be very hesitant to batch the reviews too much, because as I mentioned elsewhere in the thread, there is a lot of value from fast feedback loops in this evaluation process, as well as allowing experts in different domains to chime in with their thoughts.
And we did see exactly that. I think the best comment (next to yours) on that post is Ben West’s comment and Aaron Gertler’s comments that were both written relatively soon after the post was written (and I think would have been written even if you hadn’t written yours) and concisely explained the problems with the proposal. I don’t think a delay of 2-3 days is that bad, and overall I think nonzerosum successfully received the feedback that the project needed. I do think I would like to ensure that people proposing projects feel less punished by doing so, but I think that can easily be achieved by establishing a space in which there is common knowledge that a lot of proposals will be bad and have problems, and that a proposal being proposed in that space does not mean that everyone has to be scared that someone will rush ahead with that proposal and potentially cause a lot of damage.
If I understood your setup correctly, it would have potentially taken multiple weeks for nonzerosum to get feedback on their proposal, and the response would have come in the form of an evaluation that took multiple hours to write, which I don’t think would have benefited anyone in this situation.
I think an initial version of the process, in which you plus maybe one or two close collaborators, would play the role of evaluators and participate in an EA Forum thread, would take less than 5 hours to set up and less than 15 hours of time to actually execute and write reviews on, and I think would give you significant evidence about what kind of evaluations will be valuable and what the current bottlenecks in this space are.
I think that post is actually a good example of why a multi-stage process like this will cause a lot of problems. I think the best thing for nonzerosum to do would have been to create a short comment or post, maybe two to three paragraphs, in which he explained the basic idea of a donor list. At this point, he would have not been super invested in it, and I think if he had posted only a short document, people would have reacted with openness and told him that there has been a pretty long history of people trying to make lots of EA donor coordination platforms, and that there are significant problems with unilateralist curse-like problems. I think the downvotes and negative reaction came primarily from people perceiving him to be prematurely charging ahead with a project.
I do think you need some additional incentive for people to actually write up their thoughts in addition to just voting on stuff, which is why a volunteer evaluator group, or maybe some kind of financial incentive, or maybe just some kind of modifications to the forum software (which I recognize is not something you can easily do but which I have affordances for), is a good idea. But I do think you want to be very hesitant to batch the reviews too much, because as I mentioned elsewhere in the thread, there is a lot of value from fast feedback loops in this evaluation process, as well as allowing experts in different domains to chime in with their thoughts.
And we did see exactly that. I think the best comment (next to yours) on that post is Ben West’s comment and Aaron Gertler’s comments that were both written relatively soon after the post was written (and I think would have been written even if you hadn’t written yours) and concisely explained the problems with the proposal. I don’t think a delay of 2-3 days is that bad, and overall I think nonzerosum successfully received the feedback that the project needed. I do think I would like to ensure that people proposing projects feel less punished by doing so, but I think that can easily be achieved by establishing a space in which there is common knowledge that a lot of proposals will be bad and have problems, and that a proposal being proposed in that space does not mean that everyone has to be scared that someone will rush ahead with that proposal and potentially cause a lot of damage.
If I understood your setup correctly, it would have potentially taken multiple weeks for nonzerosum to get feedback on their proposal, and the response would have come in the form of an evaluation that took multiple hours to write, which I don’t think would have benefited anyone in this situation.