Instead I would specifically look at its output and approach to external engagement: if they’re not publishing research I would take that as a strong negative signal for the project. Likewise, in participating in a research project I would want to ensure that we were writing publicly and opening our work to engaged and critical feedback.
I’m curious about why your conclusion is about the importance of public engagement instead of about the importance (and difficulty) of setting up good feedback loops for research.
It seems to me that it is possible to have good feedback loops without good public engagement (e.g., the Manhattan Project) and good public engagement without good feedback loops (e.g., many areas of academic research). But, whereas important research progress seems possible in the former case, it seems all but impossible in the latter case.
I think feedback loops are the important thing, but public engagement is a powerful way to strengthen them which Leverage seemed to have suffered from deprioritizing.
In the example of the Manhattan Project, they were studying and engineering physical things, which makes it a lot harder to be wrong about whether you’re making progress. My understanding is also that they brought a shockingly high fraction of the experts in the field into the project, which might mean you could get some of what you’d normally get from public presentation internally?
The degree to which public presentation is likely to strengthen your feedback loops seems to depend quite a lot on the state of the field that you are investigating. In highly functional fields like those found in modern physics, it seems quite likely to be helpful. In less functional fields or those with fewer relevant researchers, this seems less helpful.
To my mind, one strong consideration in favor of publicly presenting your research if you’re working in a less functional field is that even if you’re right, causing future researchers to build on your work is extremely difficult. Indeed, promising research avenues that are presented publicly die all the time (e.g., muscle reading or phlogiston c.f. Chang in Is Water H2O?). Presenting your research publicly is the best way to engage with other researchers and ensure that, if you do succeed, a research tradition can be built on top of your work.
I’m curious about why your conclusion is about the importance of public engagement instead of about the importance (and difficulty) of setting up good feedback loops for research.
It seems to me that it is possible to have good feedback loops without good public engagement (e.g., the Manhattan Project) and good public engagement without good feedback loops (e.g., many areas of academic research). But, whereas important research progress seems possible in the former case, it seems all but impossible in the latter case.
I think feedback loops are the important thing, but public engagement is a powerful way to strengthen them which Leverage seemed to have suffered from deprioritizing.
In the example of the Manhattan Project, they were studying and engineering physical things, which makes it a lot harder to be wrong about whether you’re making progress. My understanding is also that they brought a shockingly high fraction of the experts in the field into the project, which might mean you could get some of what you’d normally get from public presentation internally?
The degree to which public presentation is likely to strengthen your feedback loops seems to depend quite a lot on the state of the field that you are investigating. In highly functional fields like those found in modern physics, it seems quite likely to be helpful. In less functional fields or those with fewer relevant researchers, this seems less helpful.
To my mind, one strong consideration in favor of publicly presenting your research if you’re working in a less functional field is that even if you’re right, causing future researchers to build on your work is extremely difficult. Indeed, promising research avenues that are presented publicly die all the time (e.g., muscle reading or phlogiston c.f. Chang in Is Water H2O?). Presenting your research publicly is the best way to engage with other researchers and ensure that, if you do succeed, a research tradition can be built on top of your work.