I think the obvious answer is that doing controlled trials in these areas is a whole lot of work/expense for the benefit.
Some things like health effects can take a long time to play out; maybe 10-50 years. And I wouldn’t expect the difference to be particularly amazing. (I’d be surprised if the average person could increase their productivity by more than ~20% with any of those)
On “challenge trials”; I imagine the big question is how difficult it would be to convince people to accept a very different lifestyle for a long time. I’m not sure if it’s called “challenge trial” in this case.
I think the obvious answer is that doing controlled trials in these areas is a whole lot of work/expense for the benefit.
Some things like health effects can take a long time to play out; maybe 10-50 years. And I wouldn’t expect the difference to be particularly amazing. (I’d be surprised if the average person could increase their productivity by more than ~20% with any of those)
I think our main disagreement is around the likely effect sizes; e.g. I think blocking out focused work could easily have an effect size of >50% (but am pretty uncertain which is why I want the trial!). I agree about long-term effects being a concern, particularly depending on one’s TAI timelines.
On “challenge trials”; I imagine the big question is how difficult it would be to convince people to accept a very different lifestyle for a long time. I’m not sure if it’s called “challenge trial” in this case.
Yeah, I’m most excited about challenges that last more like a few months to a year, though this isn’t ideal in all domains (e.g. veganism), so maybe this wasn’t best as the top example. I have no strong views on terminology.
The health interventions seem very different to me than the productivity interventions.
The health interventions have issues with long time-scales, which productivity interventions don’t have as much.
However, productivity interventions have major challenges with generality. When I’ve looked into studies around productivity interventions, often they’re done in highly constrained environments, or environments very different from mine, and I have very little clue what to really make of them. If the results are highly promising, I’m particularly skeptical, so it would take multiple strong studies to make the case.
I think it’s really telling that Google and Amazon don’t have internal testing teams to study productivity/management techniques in isolation. In practice, I just don’t think you learn that much, for the cost of it.
What these companies do do, is to allow different managers to try things out, survey them, and promote the seemingly best practices throughout. This happens very quickly. I’m sure we could make tools to make this process go much faster. (Better elicitation, better data collection of what already happens, lots of small estimates of impact to see what to focus more on, etc).
In general, I think traditional scientific experimentation on humans is very inefficient, and we should be aiming for much more efficient setups. (But we should be working on these!)
This all makes sense to me overall. I’m still excited about this idea (slightly less so than before) but I think/agree there should be careful considerations on which interventions make the most sense to test.
I think it’s really telling that Google and Amazon don’t have internal testing teams to study productivity/management techniques in isolation. In practice, I just don’t think you learn that much, for the cost of it.
What these companies do do, is to allow different managers to try things out, survey them, and promote the seemingly best practices throughout. This happens very quickly. I’m sure we could make tools to make this process go much faster. (Better elicitation, better data collection of what already happens, lots of small estimates of impact to see what to focus more on, etc).
A few things come to mind here:
The point on the amount of evidence Google/Amazon not doing it provides feels related to the discussion around our corporate prediction market analysis. Note that I was the author who probably took the evidence that most corporations discontinued their prediction markets as the most weak (see my conclusion), though I still think it’s fairly substantial.
I also agree with the point in your reply that setting up prediction markets and learning from them has positive externalities, and a similar thing should apply here.
I agree that more data collection tools for what already happens and other innovations in that vein seem good as well!
I think the obvious answer is that doing controlled trials in these areas is a whole lot of work/expense for the benefit.
Some things like health effects can take a long time to play out; maybe 10-50 years. And I wouldn’t expect the difference to be particularly amazing. (I’d be surprised if the average person could increase their productivity by more than ~20% with any of those)
On “challenge trials”; I imagine the big question is how difficult it would be to convince people to accept a very different lifestyle for a long time. I’m not sure if it’s called “challenge trial” in this case.
It wouldn’t shock me if an average vegan diet decreased lifetime productivity by more than 20% by malnutrition → mental health link.
I think our main disagreement is around the likely effect sizes; e.g. I think blocking out focused work could easily have an effect size of >50% (but am pretty uncertain which is why I want the trial!). I agree about long-term effects being a concern, particularly depending on one’s TAI timelines.
Yeah, I’m most excited about challenges that last more like a few months to a year, though this isn’t ideal in all domains (e.g. veganism), so maybe this wasn’t best as the top example. I have no strong views on terminology.
The health interventions seem very different to me than the productivity interventions.
The health interventions have issues with long time-scales, which productivity interventions don’t have as much.
However, productivity interventions have major challenges with generality. When I’ve looked into studies around productivity interventions, often they’re done in highly constrained environments, or environments very different from mine, and I have very little clue what to really make of them. If the results are highly promising, I’m particularly skeptical, so it would take multiple strong studies to make the case.
I think it’s really telling that Google and Amazon don’t have internal testing teams to study productivity/management techniques in isolation. In practice, I just don’t think you learn that much, for the cost of it.
What these companies do do, is to allow different managers to try things out, survey them, and promote the seemingly best practices throughout. This happens very quickly. I’m sure we could make tools to make this process go much faster. (Better elicitation, better data collection of what already happens, lots of small estimates of impact to see what to focus more on, etc).
In general, I think traditional scientific experimentation on humans is very inefficient, and we should be aiming for much more efficient setups. (But we should be working on these!)
This post is relevant: https://www.lesswrong.com/posts/vCQpJLNFpDdHyikFy/are-the-social-sciences-challenging-because-of-fundamental
This all makes sense to me overall. I’m still excited about this idea (slightly less so than before) but I think/agree there should be careful considerations on which interventions make the most sense to test.
A few things come to mind here:
The point on the amount of evidence Google/Amazon not doing it provides feels related to the discussion around our corporate prediction market analysis. Note that I was the author who probably took the evidence that most corporations discontinued their prediction markets as the most weak (see my conclusion), though I still think it’s fairly substantial.
I also agree with the point in your reply that setting up prediction markets and learning from them has positive externalities, and a similar thing should apply here.
I agree that more data collection tools for what already happens and other innovations in that vein seem good as well!