Thanks, Yonatan, this is helpful! I agree that the impact accounting is hard here. I still think that encouraging a norm of making imprecise & short-term impact measurements and reporting them with the right level of uncertainty is probably better than the current situation.
Itās similar to how I think it is good, on aggregate, that EA community organisers evaluate the impact of conferences and events with surveys, despite also believing that such feedback data is relatively inaccurate, not particularly suited to assess counterfactual impact and open to misinterpretation.
I may change my mind on all that, though! On that note, I am interested to know what your evaluation system involves and why you worry that you might be deluding yourself?
I ask at the beginning of almost every conversation āwhatās your default plan? what would you do if we wouldnāt talk today?ā, and in most conversations I ask once or twice later āwhatās your default plan now?ā
To be clear, my standard of āmade a differenceā is something like ā[expect to have] saved a few months of the career of the personā
Almost all my conversations are recorded and also in a Google Doc, which means
I can go over my own work (in practice I only did this once, when I had a small breakdown and thought Iām maybe deluding myself, so I went over lots of calls)
Itās ready for a 3rd party to do an impact analysis on my work
I think this has a psychological effect on me, somewhat thinking what someone else would say about this
I sometimes ask for (or spontaneously get) feedback after the call, mostly in the comments to one of my posts
In about half the calls I ask for feedback at the end of the call (and I donāt take anything vague like āthis was usefulā as an indication of impact, only things like āthis is where I changed my planā)
I think I āshouldā do (Iād do this if I had more resources, or a 3rd party interested)
Tidy impact analysis, going over my calls, contacting people I spoke to, more similar to what 80k do
The crazy version here is a RCT. I specifically expect my CV exercise might get impressive results in raising peopleās salary if someone would run an RCT on it
The thing that changed my mind during the small-breakdown and got me to believe that yes, Iām useful:
Specific stories of pretty big impact that people told me about months after speaking to me. Hereās one I got yesterday by surprise and made me really happy, even trying hard to analyze counterfactuals which I really appreciate
Thanks, Yonatan, this is helpful! I agree that the impact accounting is hard here. I still think that encouraging a norm of making imprecise & short-term impact measurements and reporting them with the right level of uncertainty is probably better than the current situation.
Itās similar to how I think it is good, on aggregate, that EA community organisers evaluate the impact of conferences and events with surveys, despite also believing that such feedback data is relatively inaccurate, not particularly suited to assess counterfactual impact and open to misinterpretation.
I may change my mind on all that, though! On that note, I am interested to know what your evaluation system involves and why you worry that you might be deluding yourself?
Things I do to measure myself:
I ask at the beginning of almost every conversation āwhatās your default plan? what would you do if we wouldnāt talk today?ā, and in most conversations I ask once or twice later āwhatās your default plan now?ā
To be clear, my standard of āmade a differenceā is something like ā[expect to have] saved a few months of the career of the personā
Almost all my conversations are recorded and also in a Google Doc, which means
I can go over my own work (in practice I only did this once, when I had a small breakdown and thought Iām maybe deluding myself, so I went over lots of calls)
Itās ready for a 3rd party to do an impact analysis on my work
I think this has a psychological effect on me, somewhat thinking what someone else would say about this
I sometimes ask for (or spontaneously get) feedback after the call, mostly in the comments to one of my posts
In about half the calls I ask for feedback at the end of the call (and I donāt take anything vague like āthis was usefulā as an indication of impact, only things like āthis is where I changed my planā)
I think I āshouldā do (Iād do this if I had more resources, or a 3rd party interested)
Tidy impact analysis, going over my calls, contacting people I spoke to, more similar to what 80k do
The crazy version here is a RCT. I specifically expect my CV exercise might get impressive results in raising peopleās salary if someone would run an RCT on it
The thing that changed my mind during the small-breakdown and got me to believe that yes, Iām useful:
Specific stories of pretty big impact that people told me about months after speaking to me. Hereās one I got yesterday by surprise and made me really happy, even trying hard to analyze counterfactuals which I really appreciate