IMO the risks you state are much less severe relative to missing key information about a specific charity (as likely happened with your Sinergia work) and therefore misleading people. This also makes people less likely to take your claims seriously in all future reviews.
Risk 2: Unconscious biases from interacting with charity staff.
When we evaluate a charity, we want to evaluate them based on their work, not based on how much we like their employees. Accordingly, we do not want to acquire unconscious biases.
If anyone has solutions to this problem, please let us know below, as it would make us more open to showing reviews to charities before releasing them. We would also like to acknowledge that we may be misunderstanding what people are suggesting when they say they’d like us to show our reviews to the charities before publishing them. If this simply entails sending them an email and nothing more, we are more open to that than having meetings with charity employees to discuss their review.
To clarify, yes I think most people think you should just share a Google Doc of your review and give the organisation time to leave comments about factual inaccuracies or other relevant context. I don’t think anyone is suggesting you meet with organisations and discuss things via a call. If anything, it’s best to share your review relatively early so you don’t spend 100s of hours, as you claim you did with Sinergia, down some rabbit hole which may just be a lack of understanding or context-specific issues.
I strongly agree that the benefits of sharing the evaluation greatly outweigh the risks, but I’m not sure if sharing the it relatively early is best
There is a risk of starting a draining back-and-forth which could block or massively delay publication. See e.g. Research Deprioritizing External Communication which was delayed by one year
It would cost more time for the org to review a very early draft and point out mistakes that would be fixed anyway
It could cause the org to take the evaluation less seriously, and be less likely to take action based on the feedback
I think the minimal version proposed by @Jason of just sending an advance copy a week or two in advance is an extremely low-cost policy that mitigates most of the risks and provides most of the benefits (but some limited back-and-forth would be ideal)
From first hand experience, I think critics should make more of an effort to ensure that the charities have actually received your communications and had a chance to review it. When my organization was the subject of a critical post, the email from the critic had landed in my spam, so I only learned of the critique when reading it on the forum.
What’s more, as an organization of 2 people, one of which was on leave at the time of the notice, we couldn’t have reasonably responded to it without stopping critical work to keep the lights on. The smaller the organization being critiqued is, the less flex capacity they have to respond quickly to these sorts of things, and so they should be given a longer grace period.
Yep fair enough! That was one bit I wasn’t sure about and can definitely see the downsides of sharing too early. I guess the trade-off I was considering was Vetted Causes’ time spent on the evaluation but definitely think an advance finished version with two weeks notice would be something most groups would be happy with.
IMO the risks you state are much less severe relative to missing key information about a specific charity (as likely happened with your Sinergia work) and therefore misleading people. This also makes people less likely to take your claims seriously in all future reviews.
To clarify, yes I think most people think you should just share a Google Doc of your review and give the organisation time to leave comments about factual inaccuracies or other relevant context. I don’t think anyone is suggesting you meet with organisations and discuss things via a call. If anything, it’s best to share your review relatively early so you don’t spend 100s of hours, as you claim you did with Sinergia, down some rabbit hole which may just be a lack of understanding or context-specific issues.
I strongly agree that the benefits of sharing the evaluation greatly outweigh the risks, but I’m not sure if sharing the it relatively early is best
There is a risk of starting a draining back-and-forth which could block or massively delay publication. See e.g. Research Deprioritizing External Communication which was delayed by one year
It would cost more time for the org to review a very early draft and point out mistakes that would be fixed anyway
It could cause the org to take the evaluation less seriously, and be less likely to take action based on the feedback
I think the minimal version proposed by @Jason of just sending an advance copy a week or two in advance is an extremely low-cost policy that mitigates most of the risks and provides most of the benefits (but some limited back-and-forth would be ideal)
From first hand experience, I think critics should make more of an effort to ensure that the charities have actually received your communications and had a chance to review it. When my organization was the subject of a critical post, the email from the critic had landed in my spam, so I only learned of the critique when reading it on the forum.
What’s more, as an organization of 2 people, one of which was on leave at the time of the notice, we couldn’t have reasonably responded to it without stopping critical work to keep the lights on. The smaller the organization being critiqued is, the less flex capacity they have to respond quickly to these sorts of things, and so they should be given a longer grace period.
Yep fair enough! That was one bit I wasn’t sure about and can definitely see the downsides of sharing too early. I guess the trade-off I was considering was Vetted Causes’ time spent on the evaluation but definitely think an advance finished version with two weeks notice would be something most groups would be happy with.