Thanks, Yonatan, this is helpful! I agree that the impact accounting is hard here. I still think that encouraging a norm of making imprecise & short-term impact measurements and reporting them with the right level of uncertainty is probably better than the current situation.
It’s similar to how I think it is good, on aggregate, that EA community organisers evaluate the impact of conferences and events with surveys, despite also believing that such feedback data is relatively inaccurate, not particularly suited to assess counterfactual impact and open to misinterpretation.
I may change my mind on all that, though! On that note, I am interested to know what your evaluation system involves and why you worry that you might be deluding yourself?
I ask at the beginning of almost every conversation “what’s your default plan? what would you do if we wouldn’t talk today?”, and in most conversations I ask once or twice later “what’s your default plan now?”
To be clear, my standard of “made a difference” is something like “[expect to have] saved a few months of the career of the person”
Almost all my conversations are recorded and also in a Google Doc, which means
I can go over my own work (in practice I only did this once, when I had a small breakdown and thought I’m maybe deluding myself, so I went over lots of calls)
It’s ready for a 3rd party to do an impact analysis on my work
I think this has a psychological effect on me, somewhat thinking what someone else would say about this
I sometimes ask for (or spontaneously get) feedback after the call, mostly in the comments to one of my posts
In about half the calls I ask for feedback at the end of the call (and I don’t take anything vague like “this was useful” as an indication of impact, only things like “this is where I changed my plan”)
I think I “should” do (I’d do this if I had more resources, or a 3rd party interested)
Tidy impact analysis, going over my calls, contacting people I spoke to, more similar to what 80k do
The crazy version here is a RCT. I specifically expect my CV exercise might get impressive results in raising people’s salary if someone would run an RCT on it
The thing that changed my mind during the small-breakdown and got me to believe that yes, I’m useful:
Specific stories of pretty big impact that people told me about months after speaking to me. Here’s one I got yesterday by surprise and made me really happy, even trying hard to analyze counterfactuals which I really appreciate
Thanks, Yonatan, this is helpful! I agree that the impact accounting is hard here. I still think that encouraging a norm of making imprecise & short-term impact measurements and reporting them with the right level of uncertainty is probably better than the current situation.
It’s similar to how I think it is good, on aggregate, that EA community organisers evaluate the impact of conferences and events with surveys, despite also believing that such feedback data is relatively inaccurate, not particularly suited to assess counterfactual impact and open to misinterpretation.
I may change my mind on all that, though! On that note, I am interested to know what your evaluation system involves and why you worry that you might be deluding yourself?
Things I do to measure myself:
I ask at the beginning of almost every conversation “what’s your default plan? what would you do if we wouldn’t talk today?”, and in most conversations I ask once or twice later “what’s your default plan now?”
To be clear, my standard of “made a difference” is something like “[expect to have] saved a few months of the career of the person”
Almost all my conversations are recorded and also in a Google Doc, which means
I can go over my own work (in practice I only did this once, when I had a small breakdown and thought I’m maybe deluding myself, so I went over lots of calls)
It’s ready for a 3rd party to do an impact analysis on my work
I think this has a psychological effect on me, somewhat thinking what someone else would say about this
I sometimes ask for (or spontaneously get) feedback after the call, mostly in the comments to one of my posts
In about half the calls I ask for feedback at the end of the call (and I don’t take anything vague like “this was useful” as an indication of impact, only things like “this is where I changed my plan”)
I think I “should” do (I’d do this if I had more resources, or a 3rd party interested)
Tidy impact analysis, going over my calls, contacting people I spoke to, more similar to what 80k do
The crazy version here is a RCT. I specifically expect my CV exercise might get impressive results in raising people’s salary if someone would run an RCT on it
The thing that changed my mind during the small-breakdown and got me to believe that yes, I’m useful:
Specific stories of pretty big impact that people told me about months after speaking to me. Here’s one I got yesterday by surprise and made me really happy, even trying hard to analyze counterfactuals which I really appreciate