Risk 2: Unconscious biases from interacting with charity staff.
You can try correcting for this by not updating as much towards being positive about the organisation as naively warranted by the feedback you received from people working there. In addition, you can avoid meeting with people from the organisation, restricting yourself to sharing a doc with your review, and discussing it via messages.
These are fair points. We agree the risk of bias is likely minimal if this is how you handle it.
We agree the risk of bias is likely minimal if this is how you handle it.
I think there is still some significant risk. My reviews of organisations very often become more positive towards the organisations after I integrate their feedback, although I try to update the reviews based on what makes sense to me instead of what appeals to the organisations. I try to counter the selection bias naturally present in the feedback I receive from organisations by actively searching for evidence against them after receiving their feedback, especially if I updated significantly as a result of their feedback. I search for contrary evidence not only in the updated parts of the review, but also elsewhere. In addition, you can ask feedback from people who you think are pessimistic about the organisations or their interventions.
Thanks for the feedback, Vasco!
These are fair points. We agree the risk of bias is likely minimal if this is how you handle it.
Thanks!
I think there is still some significant risk. My reviews of organisations very often become more positive towards the organisations after I integrate their feedback, although I try to update the reviews based on what makes sense to me instead of what appeals to the organisations. I try to counter the selection bias naturally present in the feedback I receive from organisations by actively searching for evidence against them after receiving their feedback, especially if I updated significantly as a result of their feedback. I search for contrary evidence not only in the updated parts of the review, but also elsewhere. In addition, you can ask feedback from people who you think are pessimistic about the organisations or their interventions.