The health interventions seem very different to me than the productivity interventions.
The health interventions have issues with long time-scales, which productivity interventions don’t have as much.
However, productivity interventions have major challenges with generality. When I’ve looked into studies around productivity interventions, often they’re done in highly constrained environments, or environments very different from mine, and I have very little clue what to really make of them. If the results are highly promising, I’m particularly skeptical, so it would take multiple strong studies to make the case.
I think it’s really telling that Google and Amazon don’t have internal testing teams to study productivity/management techniques in isolation. In practice, I just don’t think you learn that much, for the cost of it.
What these companies do do, is to allow different managers to try things out, survey them, and promote the seemingly best practices throughout. This happens very quickly. I’m sure we could make tools to make this process go much faster. (Better elicitation, better data collection of what already happens, lots of small estimates of impact to see what to focus more on, etc).
In general, I think traditional scientific experimentation on humans is very inefficient, and we should be aiming for much more efficient setups. (But we should be working on these!)
This all makes sense to me overall. I’m still excited about this idea (slightly less so than before) but I think/agree there should be careful considerations on which interventions make the most sense to test.
I think it’s really telling that Google and Amazon don’t have internal testing teams to study productivity/management techniques in isolation. In practice, I just don’t think you learn that much, for the cost of it.
What these companies do do, is to allow different managers to try things out, survey them, and promote the seemingly best practices throughout. This happens very quickly. I’m sure we could make tools to make this process go much faster. (Better elicitation, better data collection of what already happens, lots of small estimates of impact to see what to focus more on, etc).
A few things come to mind here:
The point on the amount of evidence Google/Amazon not doing it provides feels related to the discussion around our corporate prediction market analysis. Note that I was the author who probably took the evidence that most corporations discontinued their prediction markets as the most weak (see my conclusion), though I still think it’s fairly substantial.
I also agree with the point in your reply that setting up prediction markets and learning from them has positive externalities, and a similar thing should apply here.
I agree that more data collection tools for what already happens and other innovations in that vein seem good as well!
The health interventions seem very different to me than the productivity interventions.
The health interventions have issues with long time-scales, which productivity interventions don’t have as much.
However, productivity interventions have major challenges with generality. When I’ve looked into studies around productivity interventions, often they’re done in highly constrained environments, or environments very different from mine, and I have very little clue what to really make of them. If the results are highly promising, I’m particularly skeptical, so it would take multiple strong studies to make the case.
I think it’s really telling that Google and Amazon don’t have internal testing teams to study productivity/management techniques in isolation. In practice, I just don’t think you learn that much, for the cost of it.
What these companies do do, is to allow different managers to try things out, survey them, and promote the seemingly best practices throughout. This happens very quickly. I’m sure we could make tools to make this process go much faster. (Better elicitation, better data collection of what already happens, lots of small estimates of impact to see what to focus more on, etc).
In general, I think traditional scientific experimentation on humans is very inefficient, and we should be aiming for much more efficient setups. (But we should be working on these!)
This post is relevant: https://www.lesswrong.com/posts/vCQpJLNFpDdHyikFy/are-the-social-sciences-challenging-because-of-fundamental
This all makes sense to me overall. I’m still excited about this idea (slightly less so than before) but I think/agree there should be careful considerations on which interventions make the most sense to test.
A few things come to mind here:
The point on the amount of evidence Google/Amazon not doing it provides feels related to the discussion around our corporate prediction market analysis. Note that I was the author who probably took the evidence that most corporations discontinued their prediction markets as the most weak (see my conclusion), though I still think it’s fairly substantial.
I also agree with the point in your reply that setting up prediction markets and learning from them has positive externalities, and a similar thing should apply here.
I agree that more data collection tools for what already happens and other innovations in that vein seem good as well!