I have to say, I rather like putting a name to this concept. I know this wasn’t the upshot of the article, but it immediately struck me, on reading this, that it would be a good idea for the effective altruist community to engage in some minimal trust investigations of each other’s analyses and frame them as such.
I’m worried about there being too much deference and actually not very much criticism of the received wisdom. Part of the issue is that to criticise the views of smart, thoughtful, well-intentioned people in leadership positions might imply either that you don’t trust them (which is rude) or that you’re not smart and well-informed enough to ‘get it’; there are also the normal fears associated with criticising those with greater power.
These issues are somewhat addressed by saying “look, I have a lot of respect for X and assume there are right about lots of things, but I wanted to get to the bottom of this issue myself and not take anything they said for granted. So I did a ‘minimal-trust investigation’. Here’s what I found...”
I worry that, if adopted, an annoying fraction of people will use this term to mean “I looked at the citations for an article” rather than “I exhaustively looked at the evidence for X from multiple angles over a long period of time.”
An “X-hour investigation” is a more precise claim. Including the references and sources they looked at, and a description of why they chose these, is a complement to saying how much time they’ve spent. In general, I like that this post illustrates what raising one’s research ambitions looks like.
Holden: how many hours, roughly, do you think you spent on some of these minimal-trust investigations? And how many hours would you spend reading a given paper?
I wish I had a better answer, but it varies hugely by topic (and especially by how open-ended the question is). The example I give in the post was an early GiveWell investigation that played out over years, and took at least dozens of hours, maybe hundreds. Something like “checking attribution” can be under an hour. For a short-end “empirical social science” case, I can think of personal medical topics I’ve researched in a handful of hours (especially when I had previously researched similar topics and knew what I was looking for in the abstracts). I also don’t have a good answer to how long I spend on a particular study: I’ve definitely spent double-digit hours on an individual study before (and David Roodman has often gone much deeper, lowering the “trust” factor more than I ever have via things like reproducing someone’s calculations), but these are only for key studies—many studies can quickly be identified as having only small relevance to the question at hand.
I don’t think I’ve defined “minimal-trust investigation” tightly enough to make it a hard term to abuse :) but I think it could be a helpful term nonetheless, including for the purpose Michael Plant proposes.
I would include the productivity of the reviewers and the scope of the investigations as factors of the time spent evaluating the evidence. For example, an investigator who analyzes the accuracy of key assumptions 10x faster and incorporates a 10x wider viewpoint can get 100x better conclusions than another reviewer spending the same time.
I would also conduct an expected value cost-benefit analysis in deciding to what extent minimal-trust investigations’ insights are shared. For example, if EA can lose $1 billion because of outlining the questions regarding LLIN effectiveness with a 50% chance, because it loses appeal to some funders, but can gain $2 billion with 10% chance which can be used 3x more cost-effectively, then the investigation should be shared.
If a better solution exists, such as keeping the LLIN cost-effectiveness as a cool entry point while later motivating people to devise solutions which generate high wellbeing impact across futures, then the LLIN questions can be shared on a medium accessible to more senior people while the impressive numbers exhibited publicly.
Then, using the above example, EA can lose $1 billion invested in malaria with 90% likelihood, develop a solution that sustainably addresses the fundamental issues (astronomically greater cost-effectiveness than LLINs because of the scale of the future), and gain $10 billion to find further solutions.
The question can be: can you keep speaking about systemic change intentions but difficulties with OPP while dropping questions so that the development and scale up of universally beneficial systemic solutions is supported?
I have to say, I rather like putting a name to this concept. I know this wasn’t the upshot of the article, but it immediately struck me, on reading this, that it would be a good idea for the effective altruist community to engage in some minimal trust investigations of each other’s analyses and frame them as such.
I’m worried about there being too much deference and actually not very much criticism of the received wisdom. Part of the issue is that to criticise the views of smart, thoughtful, well-intentioned people in leadership positions might imply either that you don’t trust them (which is rude) or that you’re not smart and well-informed enough to ‘get it’; there are also the normal fears associated with criticising those with greater power.
These issues are somewhat addressed by saying “look, I have a lot of respect for X and assume there are right about lots of things, but I wanted to get to the bottom of this issue myself and not take anything they said for granted. So I did a ‘minimal-trust investigation’. Here’s what I found...”
I worry that, if adopted, an annoying fraction of people will use this term to mean “I looked at the citations for an article” rather than “I exhaustively looked at the evidence for X from multiple angles over a long period of time.”
An “X-hour investigation” is a more precise claim. Including the references and sources they looked at, and a description of why they chose these, is a complement to saying how much time they’ve spent. In general, I like that this post illustrates what raising one’s research ambitions looks like.
Holden: how many hours, roughly, do you think you spent on some of these minimal-trust investigations? And how many hours would you spend reading a given paper?
I wish I had a better answer, but it varies hugely by topic (and especially by how open-ended the question is). The example I give in the post was an early GiveWell investigation that played out over years, and took at least dozens of hours, maybe hundreds. Something like “checking attribution” can be under an hour. For a short-end “empirical social science” case, I can think of personal medical topics I’ve researched in a handful of hours (especially when I had previously researched similar topics and knew what I was looking for in the abstracts). I also don’t have a good answer to how long I spend on a particular study: I’ve definitely spent double-digit hours on an individual study before (and David Roodman has often gone much deeper, lowering the “trust” factor more than I ever have via things like reproducing someone’s calculations), but these are only for key studies—many studies can quickly be identified as having only small relevance to the question at hand.
I don’t think I’ve defined “minimal-trust investigation” tightly enough to make it a hard term to abuse :) but I think it could be a helpful term nonetheless, including for the purpose Michael Plant proposes.
I would include the productivity of the reviewers and the scope of the investigations as factors of the time spent evaluating the evidence. For example, an investigator who analyzes the accuracy of key assumptions 10x faster and incorporates a 10x wider viewpoint can get 100x better conclusions than another reviewer spending the same time.
I would also conduct an expected value cost-benefit analysis in deciding to what extent minimal-trust investigations’ insights are shared. For example, if EA can lose $1 billion because of outlining the questions regarding LLIN effectiveness with a 50% chance, because it loses appeal to some funders, but can gain $2 billion with 10% chance which can be used 3x more cost-effectively, then the investigation should be shared.
If a better solution exists, such as keeping the LLIN cost-effectiveness as a cool entry point while later motivating people to devise solutions which generate high wellbeing impact across futures, then the LLIN questions can be shared on a medium accessible to more senior people while the impressive numbers exhibited publicly.
Then, using the above example, EA can lose $1 billion invested in malaria with 90% likelihood, develop a solution that sustainably addresses the fundamental issues (astronomically greater cost-effectiveness than LLINs because of the scale of the future), and gain $10 billion to find further solutions.
The question can be: can you keep speaking about systemic change intentions but difficulties with OPP while dropping questions so that the development and scale up of universally beneficial systemic solutions is supported?