On the other hand, making evaluations public is more informative for readers, who may acquire better models of reality if the evaluations are correct,
I am in agreement. Please, let me note that people can still get a good model of reality even if they do not know the names of the people involved.
If evaluations did not contain the name of the subjects, do you think it would still be easy for readers to connect the evaluation to the organisations being evaluated? Perhaps you could frame the evaluation so that links are not clear.
or be able to point out flaws if the evaluation has some errors.
Although this is the reviewer’s responsibility, it would be nice to have extra help indeed. (Is this you goal?) Though, the quality of feedback you receive is linked to the amount of information you share, and specific organisation details might be important here. Perhaps, you could share the detailed information to a limited set of interested people working while asking them to sign a confidentiality agreement.
I’d also be curious about whether evaluators generally should or shouldn’t give the people and organizations being evaluated the chance to respond before publication.
Would that make the reviewers change their mind?
If there is a specific issue the reviewer is worried about, I believe the reviewer can query the organisation directly.
If it is a more general issue, it is likely to be something the reviewer need to do further research about. Probably the reviewer does not have enough time to carry out the needed research, and a rushed evaluation does not help.
Nonetheless, it is important to give the organisations an opportunity to give a post-evaluation feedback, so that the reviewer has chance to address the general issue before the next round of reviews.
Furthermore, let’s not forget that one of the evaluation criteria is the ability of the applicants to introduce the problem, describe clearly the plans and address risks and contingencies. If something big is missing, it is generally a sign that the applicant needs a bit more time to complete the idea, and the reviewer should probably advise waiting for the next round.
I am in agreement. Please, let me note that people can still get a good model of reality even if they do not know the names of the people involved.
If evaluations did not contain the name of the subjects, do you think it would still be easy for readers to connect the evaluation to the organisations being evaluated? Perhaps you could frame the evaluation so that links are not clear.
Although this is the reviewer’s responsibility, it would be nice to have extra help indeed. (Is this you goal?) Though, the quality of feedback you receive is linked to the amount of information you share, and specific organisation details might be important here. Perhaps, you could share the detailed information to a limited set of interested people working while asking them to sign a confidentiality agreement.
Would that make the reviewers change their mind?
If there is a specific issue the reviewer is worried about, I believe the reviewer can query the organisation directly.
If it is a more general issue, it is likely to be something the reviewer need to do further research about. Probably the reviewer does not have enough time to carry out the needed research, and a rushed evaluation does not help.
Nonetheless, it is important to give the organisations an opportunity to give a post-evaluation feedback, so that the reviewer has chance to address the general issue before the next round of reviews.
Furthermore, let’s not forget that one of the evaluation criteria is the ability of the applicants to introduce the problem, describe clearly the plans and address risks and contingencies. If something big is missing, it is generally a sign that the applicant needs a bit more time to complete the idea, and the reviewer should probably advise waiting for the next round.