My guess is that longtermist EAs ( like almost all humans) have never been that close to purely quantitative models guiding decisions
I agree with the literal meaning of that, because it is generally a terrible idea to just do what a purely quantitative model tells you (and I’ll note that even GiveWell isn’t doing this). But imagining the spirit of what you meant, I suspect I disagree.
I don’t think you should collapse it into the single dimension of “how much do you use quantitative models in your decisions”. It also matters how amenable the decisions are to quantitative modeling. I’m not sure how you’re distinguishing between the two hypotheses:
Longtermists don’t like quantitative modeling in general.
Longtermist questions are not amenable to quantitative modeling, and so longtermists don’t do much quantitative modeling, but they would if they tackled questions that were amenable to quantitative modeling.
(Unless you want to defend the position that longtermist questions are just as easy to model as, say, those in global poverty? That would be… an interesting position.)
Also, just for the sake of actual evidence, here are some attempts at modeling, biased towards AI since that’s the space I know. Not all are quantitative, and none of them are cost effectiveness analyses.
going from my stereotype of weeatquince(2020)’s views
Fwiw, my understanding is that weeatquince(2020) is very pro modeling, and is only against the negation of the motte. The first piece of advice in that post is to use techniques like assumption based planning, exploratory modeling, and scenario planning, all of which sound to me like “explicit modeling”. I think I personally am a little more against modeling than weeatquince(2020).
(I’m exaggerating my views here to highlight the differences, I think my all-things-considered opinion on these positions are much closer to yours than the rest of the comment will make it sound)
I think my strongest disagreement with your comment is the framing here:
I’m not sure how you’re distinguishing between the two hypotheses:
Longtermists don’t like quantitative modeling in general.
Longtermist questions are not amenable to quantitative modeling, and so longtermists don’t do much quantitative modeling, but they would if they tackled questions that were amenable to quantitative modeling.
(Unless you want to defend the position that longtermist questions are just as easy to model as, say, those in global poverty? That would be… an interesting position.)
If we peel away the sarcasm, I think the implicit framing is that
If X is less amenable than Y to method A of obtaining truth, and X is equally or more amenable to methods B, C, and D relative to Y, we should do less method A to obtain truth in X (relative to Y), and more methods B, C, and D.
X is less amenable than Y to method A of obtaining truth.
Thus, we should use method A less in X than in Y.
Unless I’m missing something, I think this is logically invalid. The obvious response here is that I don’t think longtermist questions are more amenable to explicit quantitative modeling than global poverty, but I’m even more suspicious of other methodologies here.
Medicine is less amenable to empirical testing than physics, but that doesn’t mean that clinical intuition is a better source of truth for the outcomes of drugs than RCTs. (But medicine is relatively much less amenable to theorems than physics, so it’s correct to use less proofs in medicine than physics.)
More minor gripes:
(and I’ll note that even GiveWell isn’t doing this).
I think I’m willing to bite the bullet and say that GiveWell (or at least my impression of them from a few years back) should be more rigorous in their modeling. Eg, weird to use median staff member’s views as a proxy for truth, weird to have so few well-specified forecasts, and so forth.
The first piece of advice in that post is to use techniques like assumption based planning, exploratory modeling, and scenario planning, all of which sound to me like “explicit modeling
I think we might just be arguing about different things here? Like to me, these seem more like verbal arguments of questionable veracity than something that has a truth-value like cost-effectiveness analyses or forecasting. (In contrast, Open Phil’s reports on AI, or at least the ones I’ve read, would count as modeling).
because it is generally a terrible idea to just do what a purely quantitative model tells you
What’s the actual evidence for this? I feel like this type of reasoning (and other general things like it in the rough cluster of “injunctions against naive consequentialism”) are pretty common in our community and tend to be strongly held, but when I ask people to defend it, I just see weird thought experiments and handwaved intuitions (rather than a model or a track record)?
This type of view also maps in my head to being the type of view that’s a) high-status and b) diplomatic/”plays nicely” with high-prestige non-EA Western intellectuals, which makes me doubly suspicious that views of this general shape are arrived at through impartial truth-seeking means.
I also think in practice if you have a model telling you to do one thing but your intuitions tell you to do something else, it’s often worth making enough updates to form a reflective equilibrium. There are at least two ways to go about this:
1) Use the model to maybe update your intuitions, and go with your intuitions in the final decision, being explicit about how your final decisions may have gone against the naive model.
2) Use your intuitions to probe which pieces your model is making, update your model accordingly, and then go with your (updated) model in the final decision, being explicit about how your final model may have been updated for unprincipled reasons.
I think you (and by revealed preferences, the EA community, including myself) usually goes with 1) as the correct form of intuition vs model reflective equilibrium. But I don’t think this is backed by too much evidence, and I think we haven’t really given 2) a fair shake.
Now I think in practice 1) and 2) might end up getting the same result much of the time anyway. But a) probably not all the time and b) this is an empirical question.
The obvious response here is that I don’t think longtermist questions are more amenable to explicit quantitative modeling than global poverty, but I’m even more suspicious of other methodologies here.
Yeah, I’m just way, way more suspicious of quantitative modeling relative to other methodologies for most longtermist questions.
I think we might just be arguing about different things here?
Makes sense, I’m happy to ignore those sorts of methods for the purposes of this discussion.
Medicine is less amenable to empirical testing than physics, but that doesn’t mean that clinical intuition is a better source of truth for the outcomes of drugs than RCTs.
You can’t run an RCT on arms races between countries, whether or not AGI leads to extinction, whether totalitarian dictatorships are stable, whether civilizational collapse would be a permanent trajectory change vs. a temporary blip, etc.
What’s the actual evidence for this?
It just seems super obvious in almost every situation that comes up? I also don’t really know how you expect to get evidence; it seems like you can’t just “run an RCT” here, when a typical quantitative model for a longtermist question takes ~a year to develop (and that’s in situations that are selected for being amenable to quantitative modeling).
For example, here’s a subset of the impact-related factors I considered when I was considering where to work:
Lack of non-xrisk-related demands on my time
Freedom to work on what I want
Ability to speak publicly
Career flexibility
Salary
I think incorporating just these factors into a quantitative model is a hell of an ask (and there are others I haven’t listed here—I haven’t even included the factors for the academia vs industry question). A selection of challenges:
I need to make an impact calculation for the research I would do by default.
I need to make that impact calculation comparable with donations (so somehow putting them in the same units).
I need to predict the counterfactual research I would do at each of the possible organizations if I didn’t have the freedom to work on what I wanted, and quantify its impact, again in similar units.
I need to model the relative importance of technical research that tries to solve the problem vs. communication.
To model the benefits of communication, I need to model field-building benefits, legitimizing benefits, and the benefit of convincing key future decision-makers.
I need to quantify the probability of various kinds of “risks” (the org I work at shuts down, we realize AI risk isn’t actually a problem, a different AI lab reveals that they’re going to get to AGI in 2 years, unknown unknowns) in order to quantify the importance of career flexibility.
I think just getting a framework that incorporates all of these things is already a Herculean effort that really isn’t worth it, and even if you did make such a framework, I would be shocked if you could set the majority of the inputs based on actually good reference classes rather than just “what my gut says”. (And that’s all assuming I don’t notice a bunch more effects I failed to mention initially that my intuitions were taking into account but that I hadn’t explicitly verbalized.)
It seems blatantly obvious that the correct choice here is not to try to get to the point of “quantitative model that captures the large majority of the relevant considerations with inputs that have some basis in reference classes / other forms of legible evidence”, and I’d be happy to take a 100:1 bet that you wouldn’t be able to produce a model that meets that standard (as I evaluate it) in 1000 person-hours.
I have similar reactions for most other cost effectiveness analyses in longtermism. (For quantitative modeling in general, it depends on the question, but I expect I would still often have this reaction.)
Eg, weird to use median staff member’s views as a proxy for truth
If you mean that the weighting on saving vs. improving lives comes from the median staff member, note that GiveWell has been funding research that aims to set these weights in a manner with more legible evidence, because the evidence didn’t exist. In some sense this is my point—that if you want to get legible evidence, you need to put in large amounts of time and money in order to generate that evidence; this problem is worse in the longtermist space and is rarely worth it.
I agree with the literal meaning of that, because it is generally a terrible idea to just do what a purely quantitative model tells you (and I’ll note that even GiveWell isn’t doing this). But imagining the spirit of what you meant, I suspect I disagree.
I don’t think you should collapse it into the single dimension of “how much do you use quantitative models in your decisions”. It also matters how amenable the decisions are to quantitative modeling. I’m not sure how you’re distinguishing between the two hypotheses:
Longtermists don’t like quantitative modeling in general.
Longtermist questions are not amenable to quantitative modeling, and so longtermists don’t do much quantitative modeling, but they would if they tackled questions that were amenable to quantitative modeling.
(Unless you want to defend the position that longtermist questions are just as easy to model as, say, those in global poverty? That would be… an interesting position.)
Also, just for the sake of actual evidence, here are some attempts at modeling, biased towards AI since that’s the space I know. Not all are quantitative, and none of them are cost effectiveness analyses.
Open Phil’s reports on AI timelines: Biological anchors, Modeling the Human Trajectory, Semi-informative priors, brain computation, probability of power-seeking x-risk
Races: racing to the precipice, followup
Mapping out arguments: MTAIR and its inspiration
Fwiw, my understanding is that weeatquince(2020) is very pro modeling, and is only against the negation of the motte. The first piece of advice in that post is to use techniques like assumption based planning, exploratory modeling, and scenario planning, all of which sound to me like “explicit modeling”. I think I personally am a little more against modeling than weeatquince(2020).
Thanks so much for the response! Upvoted.
(I’m exaggerating my views here to highlight the differences, I think my all-things-considered opinion on these positions are much closer to yours than the rest of the comment will make it sound)
I think my strongest disagreement with your comment is the framing here:
If we peel away the sarcasm, I think the implicit framing is that
If X is less amenable than Y to method A of obtaining truth, and X is equally or more amenable to methods B, C, and D relative to Y, we should do less method A to obtain truth in X (relative to Y), and more methods B, C, and D.
X is less amenable than Y to method A of obtaining truth.
Thus, we should use method A less in X than in Y.
Unless I’m missing something, I think this is logically invalid. The obvious response here is that I don’t think longtermist questions are more amenable to explicit quantitative modeling than global poverty, but I’m even more suspicious of other methodologies here.
Medicine is less amenable to empirical testing than physics, but that doesn’t mean that clinical intuition is a better source of truth for the outcomes of drugs than RCTs. (But medicine is relatively much less amenable to theorems than physics, so it’s correct to use less proofs in medicine than physics.)
More minor gripes:
I think I’m willing to bite the bullet and say that GiveWell (or at least my impression of them from a few years back) should be more rigorous in their modeling. Eg, weird to use median staff member’s views as a proxy for truth, weird to have so few well-specified forecasts, and so forth.
I think we might just be arguing about different things here? Like to me, these seem more like verbal arguments of questionable veracity than something that has a truth-value like cost-effectiveness analyses or forecasting. (In contrast, Open Phil’s reports on AI, or at least the ones I’ve read, would count as modeling).
What’s the actual evidence for this? I feel like this type of reasoning (and other general things like it in the rough cluster of “injunctions against naive consequentialism”) are pretty common in our community and tend to be strongly held, but when I ask people to defend it, I just see weird thought experiments and handwaved intuitions (rather than a model or a track record)?
This type of view also maps in my head to being the type of view that’s a) high-status and b) diplomatic/”plays nicely” with high-prestige non-EA Western intellectuals, which makes me doubly suspicious that views of this general shape are arrived at through impartial truth-seeking means.
I also think in practice if you have a model telling you to do one thing but your intuitions tell you to do something else, it’s often worth making enough updates to form a reflective equilibrium. There are at least two ways to go about this:
1) Use the model to maybe update your intuitions, and go with your intuitions in the final decision, being explicit about how your final decisions may have gone against the naive model.
2) Use your intuitions to probe which pieces your model is making, update your model accordingly, and then go with your (updated) model in the final decision, being explicit about how your final model may have been updated for unprincipled reasons.
I think you (and by revealed preferences, the EA community, including myself) usually goes with 1) as the correct form of intuition vs model reflective equilibrium. But I don’t think this is backed by too much evidence, and I think we haven’t really given 2) a fair shake.
Now I think in practice 1) and 2) might end up getting the same result much of the time anyway. But a) probably not all the time and b) this is an empirical question.
Yeah, I’m just way, way more suspicious of quantitative modeling relative to other methodologies for most longtermist questions.
Makes sense, I’m happy to ignore those sorts of methods for the purposes of this discussion.
You can’t run an RCT on arms races between countries, whether or not AGI leads to extinction, whether totalitarian dictatorships are stable, whether civilizational collapse would be a permanent trajectory change vs. a temporary blip, etc.
It just seems super obvious in almost every situation that comes up? I also don’t really know how you expect to get evidence; it seems like you can’t just “run an RCT” here, when a typical quantitative model for a longtermist question takes ~a year to develop (and that’s in situations that are selected for being amenable to quantitative modeling).
For example, here’s a subset of the impact-related factors I considered when I was considering where to work:
Lack of non-xrisk-related demands on my time
Freedom to work on what I want
Ability to speak publicly
Career flexibility
Salary
I think incorporating just these factors into a quantitative model is a hell of an ask (and there are others I haven’t listed here—I haven’t even included the factors for the academia vs industry question). A selection of challenges:
I need to make an impact calculation for the research I would do by default.
I need to make that impact calculation comparable with donations (so somehow putting them in the same units).
I need to predict the counterfactual research I would do at each of the possible organizations if I didn’t have the freedom to work on what I wanted, and quantify its impact, again in similar units.
I need to model the relative importance of technical research that tries to solve the problem vs. communication.
To model the benefits of communication, I need to model field-building benefits, legitimizing benefits, and the benefit of convincing key future decision-makers.
I need to quantify the probability of various kinds of “risks” (the org I work at shuts down, we realize AI risk isn’t actually a problem, a different AI lab reveals that they’re going to get to AGI in 2 years, unknown unknowns) in order to quantify the importance of career flexibility.
I think just getting a framework that incorporates all of these things is already a Herculean effort that really isn’t worth it, and even if you did make such a framework, I would be shocked if you could set the majority of the inputs based on actually good reference classes rather than just “what my gut says”. (And that’s all assuming I don’t notice a bunch more effects I failed to mention initially that my intuitions were taking into account but that I hadn’t explicitly verbalized.)
It seems blatantly obvious that the correct choice here is not to try to get to the point of “quantitative model that captures the large majority of the relevant considerations with inputs that have some basis in reference classes / other forms of legible evidence”, and I’d be happy to take a 100:1 bet that you wouldn’t be able to produce a model that meets that standard (as I evaluate it) in 1000 person-hours.
I have similar reactions for most other cost effectiveness analyses in longtermism. (For quantitative modeling in general, it depends on the question, but I expect I would still often have this reaction.)
If you mean that the weighting on saving vs. improving lives comes from the median staff member, note that GiveWell has been funding research that aims to set these weights in a manner with more legible evidence, because the evidence didn’t exist. In some sense this is my point—that if you want to get legible evidence, you need to put in large amounts of time and money in order to generate that evidence; this problem is worse in the longtermist space and is rarely worth it.