I was surprised by your “unpleasant to a lot of communities” comment. By that, are you referring to the dynamic where if you have to place value on outcomes, some people/orgs will be disappointed with the value you place on their work?
Not really. I was more referring that any attempt to quantify the likely impact someone will have is (a) inaccurate (b) likely to create some sort of hierarchy and unhealthy community dynamics.
This seems like another area where control groups would be helpful in making the exercise an actual experiment. Seems like a fairly easy place to introduce at least some randomization into
I agree with this, I like the idea of successful groups joining existing mentorship programs such that there is a natural control group of “average of all the other mentors.” (There are many ways this experiment would be imperfect, as I’m sure you can imagine) - I think the main implementation challenge here so far has been “getting groups to actually want to do this.” We are very careful to preserve the groups’ autonomy, I think this acts as a check on our behaviour. If groups engage on programs with us voluntarily, and we don’t make that engagement a condition of funding, it demonstrates that our programs are at least delivering value in the eyes of the organizers. If we started trying to claim more autonomy and started designating groups into experiments, we’d lose one of our few feedback measures. On balance I think I would prefer to have the feedback mechanism rather than the experiment. (The previous paragraph does contain some simplifications, it would certainly be possible to find examples of where we haven’t optimised purely for group autonomy)
Thanks for clarifying these points Rob. Agree that group autonomy is an important feedback loop, and that this feedback is more important than the experiment I suggested. But to the extent its possible to do experimentation on a voluntary basis, I do think that’d be valuable.
Not really. I was more referring that any attempt to quantify the likely impact someone will have is (a) inaccurate (b) likely to create some sort of hierarchy and unhealthy community dynamics.
I agree with this, I like the idea of successful groups joining existing mentorship programs such that there is a natural control group of “average of all the other mentors.” (There are many ways this experiment would be imperfect, as I’m sure you can imagine) - I think the main implementation challenge here so far has been “getting groups to actually want to do this.” We are very careful to preserve the groups’ autonomy, I think this acts as a check on our behaviour. If groups engage on programs with us voluntarily, and we don’t make that engagement a condition of funding, it demonstrates that our programs are at least delivering value in the eyes of the organizers. If we started trying to claim more autonomy and started designating groups into experiments, we’d lose one of our few feedback measures. On balance I think I would prefer to have the feedback mechanism rather than the experiment. (The previous paragraph does contain some simplifications, it would certainly be possible to find examples of where we haven’t optimised purely for group autonomy)
Thanks for clarifying these points Rob. Agree that group autonomy is an important feedback loop, and that this feedback is more important than the experiment I suggested. But to the extent its possible to do experimentation on a voluntary basis, I do think that’d be valuable.
I agree with this statement entirely.
Go team!