Thank you for writing this. I was very curious about Leverage and Iām excited to see more clearly what you are going for.
Some off the bat skepticism. It seems a priori that the research on early stage science is motivated by early stage research directions and tools in Psychology. Iām wary of motivated reasoning when coming to conclusions regarding the resulting models in early stage, especially as it seems to me that this kind of research (like historical research) is very malleable and can be inadvertently argued to almost any conclusions one is initially inclined to.
Whatās your take on it?
Also, Iām not quite sure where do you put the line on what is an early stage research. To take some familiar examples, Einsteinās theory of relativity, Turingās cryptanalysis research on the enigma (with new computing tools), Wilesās proof of Fermatās last theorem, EAās work on longtermism, Current research on String theoryāare they early stage scientific research?
Some off the bat skepticism. It seems a priori that the research on early stage science is motivated by early stage research directions and tools in Psychology. Iām wary of motivated reasoning when coming to conclusions regarding the resulting models in early stage, especially as it seems to me that this kind of research (like historical research) is very malleable and can be inadvertently argued to almost any conclusions one is initially inclined to.
Whatās your take on it?
Thanks for the question. This seems like the right kind of thing to be skeptical about. Here are a few thoughts.
First, I want to emphasize that we hypothesize that there may be a pattern here. Part of our initial reasoning for thinking that the hypothesis is plausible comes from both the historical case studies and our results from attempting early stage psychology research, but it could very well turn out that science doesnāt follow phases in the way weāve hypothesized or that we arenāt able to find a single justified, describable pattern in the development of functional knowledge acquisition programs. If this happens weād abandon or change the research program depending on what we find.
I expect that claims we make about early stage science will ultimately involve three justification types. The first is whether we can make abstractly plausible claims that fit the fact pattern from historical cases. The second is that our claims will need to follow a coherent logic of discovery that makes sense given the obstacles that scientists face in understanding new phenomena. Finally, if our research program goes well, I expect us to be able to make claims about how scientists should conduct early stage science today and then see whether those claims help scientists achieve more scientific progress. The use of multiple justification types makes it more difficult to simply argue for whatever conclusion one is already inclined towards.
Finally, I should note that the epistemic status of claims made on the basis of historical cases is something of an open question. Thereās an active debate in academia about the use of history for reaching methodological conclusions, but at least one camp holds that historical cases can be used in an epistemically sound way. Working through the details of this debate is one of the topics Iām researching at the moment.
Also, Iām not quite sure where do you put the line on what is an early stage research. To take some familiar examples, Einsteinās theory of relativity, Turingās cryptanalysis research on the enigma (with new computing tools), Wilesās proof of Fermatās last theorem, EAās work on longtermism, Current research on String theoryāare they early stage scientific research?
I donāt yet have a precise answer to the question of which instances of scientific progress count as early stage science although I expect to work out a more detailed account in the future. Figuring out whether a case of intellectual progress counts as early stage science involves both figuring out whether it is science and then figuring out whether it is early stage science. I probably wouldnāt consider Wilesās proof of Fermatās last theorem and the development of cryptography as early stage science because I wouldnāt consider mathematical research of this type as science. Similarly, I probably wouldnāt consider EA work on longtermism as early stage science because I would consider it philosophy instead of science.
In terms of whether a particular work of science is early stage science, in our paper we gesture at the characteristics one might look for by identifying the following cluster of attributes:
A relative absence of established theories and well-understood instruments in the area of investigation, the appearance of strange or unexplained phenomena, and lack of theoretical and practical consensus among researchers. Progress seems to occur despite (and sometimes enabled by) flawed theories, individual researchers use imprecise measurement tools that are frequently new and difficult to share, and there exists a bi-directional cycle of improvement between increasingly sophisticated theories and increasingly precise measurement tools.
I donāt know enough about the details of how Einstein arrived at his general theory of relativity to say whether it fits this attribute cluster, but it appears to be missing the experimentation and improvement of measurements tools, and disagreements among researchers. Similarly, while there is significant disagreement among researchers working on theories in modern physics, I think there is substantial agreement on which phenomena need to be explained, how the relevant instruments work and so on.
Great, this helps me understand my confusion regarding what counts as early stage science. I come from a math background, and I feel that the cluster of attributes above represent a lot of how I see some of the progress there. There are clear examples where the language, intuitions and background facts are understood to be very far from grasping an observed phenomenon.
Instruments and measurement tools in Math can be anything from intuitions of experts to familiar simplifications to technical tools that helps (graduate students) to tackle subcases (which would themselves be considered as āobservationsā).
Different researchers may be in complete disagreement on what are the relevant tools (in the above sense) and directions to solve the problem. There is a constant feeling of progress even though it may be completely unrelated to the goal. Some tools require deep expertise in a specific subbranch of mathematics that makes it harder to collaborate and reach consensus.
So Iām curious if intellectual progress which is dependent on physical tools is really that much different. Iād naively expect your results to translate to math as well.
So Iām curious if intellectual progress which is dependent on physical tools is really that much different. Iād naively expect your results to translate to math as well.
This is an interesting point, and itās useful to know that your experience indicates there might be a similar phenomenon in math.
My initial reaction is that I wouldnāt expect models of early stage science to straightforwardly apply to mathematics because observations are central to scientific inquiry and donāt appear to have a straightforward analogue in the mathematical case (observations are obviously involved in math, but the role and type seems possibly different).
Iāll keep the question of whether the models apply to mathematics in mind as we start specifying the early stage science hypotheses in more detail.
Thank you for writing this. I was very curious about Leverage and Iām excited to see more clearly what you are going for.
Some off the bat skepticism. It seems a priori that the research on early stage science is motivated by early stage research directions and tools in Psychology. Iām wary of motivated reasoning when coming to conclusions regarding the resulting models in early stage, especially as it seems to me that this kind of research (like historical research) is very malleable and can be inadvertently argued to almost any conclusions one is initially inclined to.
Whatās your take on it?
Also, Iām not quite sure where do you put the line on what is an early stage research. To take some familiar examples, Einsteinās theory of relativity, Turingās cryptanalysis research on the enigma (with new computing tools), Wilesās proof of Fermatās last theorem, EAās work on longtermism, Current research on String theoryāare they early stage scientific research?
Hi edoarad,
Thanks for the question. This seems like the right kind of thing to be skeptical about. Here are a few thoughts.
First, I want to emphasize that we hypothesize that there may be a pattern here. Part of our initial reasoning for thinking that the hypothesis is plausible comes from both the historical case studies and our results from attempting early stage psychology research, but it could very well turn out that science doesnāt follow phases in the way weāve hypothesized or that we arenāt able to find a single justified, describable pattern in the development of functional knowledge acquisition programs. If this happens weād abandon or change the research program depending on what we find.
I expect that claims we make about early stage science will ultimately involve three justification types. The first is whether we can make abstractly plausible claims that fit the fact pattern from historical cases. The second is that our claims will need to follow a coherent logic of discovery that makes sense given the obstacles that scientists face in understanding new phenomena. Finally, if our research program goes well, I expect us to be able to make claims about how scientists should conduct early stage science today and then see whether those claims help scientists achieve more scientific progress. The use of multiple justification types makes it more difficult to simply argue for whatever conclusion one is already inclined towards.
Finally, I should note that the epistemic status of claims made on the basis of historical cases is something of an open question. Thereās an active debate in academia about the use of history for reaching methodological conclusions, but at least one camp holds that historical cases can be used in an epistemically sound way. Working through the details of this debate is one of the topics Iām researching at the moment.
I donāt yet have a precise answer to the question of which instances of scientific progress count as early stage science although I expect to work out a more detailed account in the future. Figuring out whether a case of intellectual progress counts as early stage science involves both figuring out whether it is science and then figuring out whether it is early stage science. I probably wouldnāt consider Wilesās proof of Fermatās last theorem and the development of cryptography as early stage science because I wouldnāt consider mathematical research of this type as science. Similarly, I probably wouldnāt consider EA work on longtermism as early stage science because I would consider it philosophy instead of science.
In terms of whether a particular work of science is early stage science, in our paper we gesture at the characteristics one might look for by identifying the following cluster of attributes:
I donāt know enough about the details of how Einstein arrived at his general theory of relativity to say whether it fits this attribute cluster, but it appears to be missing the experimentation and improvement of measurements tools, and disagreements among researchers. Similarly, while there is significant disagreement among researchers working on theories in modern physics, I think there is substantial agreement on which phenomena need to be explained, how the relevant instruments work and so on.
Great, this helps me understand my confusion regarding what counts as early stage science. I come from a math background, and I feel that the cluster of attributes above represent a lot of how I see some of the progress there. There are clear examples where the language, intuitions and background facts are understood to be very far from grasping an observed phenomenon.
Instruments and measurement tools in Math can be anything from intuitions of experts to familiar simplifications to technical tools that helps (graduate students) to tackle subcases (which would themselves be considered as āobservationsā).
Different researchers may be in complete disagreement on what are the relevant tools (in the above sense) and directions to solve the problem. There is a constant feeling of progress even though it may be completely unrelated to the goal. Some tools require deep expertise in a specific subbranch of mathematics that makes it harder to collaborate and reach consensus.
So Iām curious if intellectual progress which is dependent on physical tools is really that much different. Iād naively expect your results to translate to math as well.
This is an interesting point, and itās useful to know that your experience indicates there might be a similar phenomenon in math.
My initial reaction is that I wouldnāt expect models of early stage science to straightforwardly apply to mathematics because observations are central to scientific inquiry and donāt appear to have a straightforward analogue in the mathematical case (observations are obviously involved in math, but the role and type seems possibly different).
Iāll keep the question of whether the models apply to mathematics in mind as we start specifying the early stage science hypotheses in more detail.