Iām not sure if I understand where youāre coming from, but Iād be curious to know: do you think similarly of EAs who are Superforecasters or have a robust forecasting record?
In my mind, updating may as well be a ritual, but if itās a ritual that allows us to better track reality then thereās little to dislike about it. As an example of how precise numerical reasoning could help, the book Superforcasting describes how rounding Superforecasters predictions (interpreting .67 probability of X happens as .7 probability of X happening) increases the error of the prediction. The book also includes many other examples where I think numerical reasoning confers a sizable advantage to its user.
In my understanding, superforecasters claim to do well at short-term predictions, in the range of several years, and with respect to a few domains. That is not me speaking about my judgement, thatās some discussion that I read about them. They have no reason to update on their own forecasts outside a certain topic and time domain, so to speak.I can track down the references and offer them if you like, but I think the problem Iām referring to is well-known.
I want to learn more about the narrow context and math where superforecasters are considered āaccurateā versus in error, and why that is so. Offering odds as predictions is not the same as offering a straightforward prediction, and interpretation and use of odds as a prediction is not the same as acting on a prediction. I suspect that thereās a mistaken analogy about what superforcasterās actually do and what it means for assigning subjective probabilities to credences in general.
EAās offer probabilities for just about any credence, but especially credences whose truth is very hard to ever determine, such as their belief in an upcoming existential harm. Accordingly, I donāt believe that the mystique that supeforecasters have can rub off on EAās, and certainly superforecaster success cannot.
Other approaches for different purposes, like fermi estimates, where you attempt to estimate the size of something by breaking it down into components and multiplying, are good ways to get a better estimate of whatever is being estimated, but I donāt consider that an attempt to assign a probability to a credence by an EA in a typical context, and that is all I was focused on with my critique.
Statistical estimation is used in a lot of domains, but not in the domain of beliefs. If I were taking samples of runs in a factory, looking to estimate numbers of widgets with construction errors coming off the line with some random sampling, I wouldnāt be thinking as I do here about EA updating. EAās donāt do much random sampling as part of offering their subjective probabilities. While it might be useful to check for background distributions relevant to event occurrences under the assumption that some outcomes are determined by a population distribution, most of what happens in the world is not random in that sense, and our feelings and beliefs are not about interpretation of such random events, for the most part. An EA working in a factory would do what everybody else does, some random sampling, not use their āpriorsā and guess a percentage and thus a probability of a random widget having a construction error.
I donāt object to numerical reasoning. Itās really the āupdatingā that I think is dubious. There are a lot of great equations in mathematics, any of which you could build a a community around, and Bayeās theorem is, I guess, one of them, but itās gone wrong in EA.
Thanks for taking the time to respond! I find your point of view more plausible now that I understand it a little bit better (though Iām still not sure of how convincing I find it overall).
Sure, and thank you for being interested in what I have written here.
I didnāt offer an argument meant to convince, more a listing of perspectives on what is actually happening around EA āupdatingā. For example, to know that an EA is confusing judgments of feeling intensity and truth probability, I would have to have evidence that they are acting in good faith toā updateā in the first place, rather than (unconsciously) pursuing some other agenda. As another example, to know that an EA has a betting problem, the psychological kind, I would have see pathological betting behavior on their part (for example, that ruined their finances or their relationships). Different EAās are typically doing different things with their āupdatingā at different times. Some of them are sure to have a bit of risk-seeking that is unhealthy and distorts their perspective, others are doing their best to measure their feelings of certainty as a fractional number, and still others are being cynical with their offerings of numbers. If the superforecasters among the EAās are doing so well with their probability estimates, I wish they would offer some commentary on free will, game theory, and how they model indeterminism. I would learn something.
If there were a tight feedback loop about process, if everyone understood not only the math, but also the evidence that moves your credence numbers, if there were widespread agreement that a new bit of evidence X should move probability of credence Y an amount Z , then I could believe that there was systematic care about epistemics involved in the community. I could believe that EA folks are always training to improve their credence probability assignments. I could believe that EA folks train their guts, hone their rationality, and sharpen their research skills, all at the same time.
But what actually goes on in EA seems very subjective, anyone can make up any number and claim their evidence justifies it and donāt really have to prove anything, and in fact, itās hard to make a case that is not subjective, and thereās a spread of numbers on most credences, or there really isnāt but Iām looking at the numbers and seeing most people quoting somebody else then, while everyday credences are just treated as regular beliefs (binary or constrained with experience) plus there is plenty of secondary gain potential, so I donāt believe that EA folks are using their probability estimates well at all, no.
NOTE: By āsecondary gain potentialā, I mean gain from an action other than the actionās ostensible gains. A person doing updating for its secondary gains might be developing:
feelings of belonging in the EA community.
a sense of control over their feelings of doubt or certainty.
an addiction to betting or other financial risk-taking.
a (false) sense of efficacy in rational arguments.
or something else that they would have to tell you.
I suspect that most EAās are being sloppy in their offerings of probabilities and credence levels. They offer something closer to epistemic status metrics for their own statements, which rely on poor evidence or no evidence, just a feeling, maybe of worry or disappointment at failing otherās expectations, or the converse, a high metric corresponding to a poor judgement of the quality of their own work, typically basing that on other work in EA, unsurprisingly. EA folks choose numbers to emphasize a message to their audience that reflects what theyāre comfortable sharing, not what they actually believe. Thereās some real controversy among EAās, but it comes from outside the community, apparently, through people like me.
So as I said, this whole updating thing has gone wrong in EA. EA as a research community would do better to drop updating and the bayesian kick until they have a better handle on contexts suitable for it. If superforecasting offers those contexts, then great!
Lets see superforecasters address existential risk with any success. Lets just see them do the research first so they that have some clear preliminary answers to what the dangers can shape up to be. Presuming that they are not using foresight but forecasting techniques, that should be interesting.
NOTE: Once I learn more about superforecasting, I might constrain my beliefs about how EAās update to exclude superforecasters working in their successful forecasting domains. I would do that right now, but Iām suspicious of the superforecaster reputation, so Iāll wait until I can learn more.
Iām not sure if I understand where youāre coming from, but Iād be curious to know: do you think similarly of EAs who are Superforecasters or have a robust forecasting record?
In my mind, updating may as well be a ritual, but if itās a ritual that allows us to better track reality then thereās little to dislike about it. As an example of how precise numerical reasoning could help, the book Superforcasting describes how rounding Superforecasters predictions (interpreting .67 probability of X happens as .7 probability of X happening) increases the error of the prediction. The book also includes many other examples where I think numerical reasoning confers a sizable advantage to its user.
Thank you for the question!
In my understanding, superforecasters claim to do well at short-term predictions, in the range of several years, and with respect to a few domains. That is not me speaking about my judgement, thatās some discussion that I read about them. They have no reason to update on their own forecasts outside a certain topic and time domain, so to speak.I can track down the references and offer them if you like, but I think the problem Iām referring to is well-known.
I want to learn more about the narrow context and math where superforecasters are considered āaccurateā versus in error, and why that is so. Offering odds as predictions is not the same as offering a straightforward prediction, and interpretation and use of odds as a prediction is not the same as acting on a prediction. I suspect that thereās a mistaken analogy about what superforcasterās actually do and what it means for assigning subjective probabilities to credences in general.
EAās offer probabilities for just about any credence, but especially credences whose truth is very hard to ever determine, such as their belief in an upcoming existential harm. Accordingly, I donāt believe that the mystique that supeforecasters have can rub off on EAās, and certainly superforecaster success cannot.
Other approaches for different purposes, like fermi estimates, where you attempt to estimate the size of something by breaking it down into components and multiplying, are good ways to get a better estimate of whatever is being estimated, but I donāt consider that an attempt to assign a probability to a credence by an EA in a typical context, and that is all I was focused on with my critique.
Statistical estimation is used in a lot of domains, but not in the domain of beliefs. If I were taking samples of runs in a factory, looking to estimate numbers of widgets with construction errors coming off the line with some random sampling, I wouldnāt be thinking as I do here about EA updating. EAās donāt do much random sampling as part of offering their subjective probabilities. While it might be useful to check for background distributions relevant to event occurrences under the assumption that some outcomes are determined by a population distribution, most of what happens in the world is not random in that sense, and our feelings and beliefs are not about interpretation of such random events, for the most part. An EA working in a factory would do what everybody else does, some random sampling, not use their āpriorsā and guess a percentage and thus a probability of a random widget having a construction error.
I donāt object to numerical reasoning. Itās really the āupdatingā that I think is dubious. There are a lot of great equations in mathematics, any of which you could build a a community around, and Bayeās theorem is, I guess, one of them, but itās gone wrong in EA.
Thanks for taking the time to respond! I find your point of view more plausible now that I understand it a little bit better (though Iām still not sure of how convincing I find it overall).
Sure, and thank you for being interested in what I have written here.
I didnāt offer an argument meant to convince, more a listing of perspectives on what is actually happening around EA āupdatingā. For example, to know that an EA is confusing judgments of feeling intensity and truth probability, I would have to have evidence that they are acting in good faith toā updateā in the first place, rather than (unconsciously) pursuing some other agenda. As another example, to know that an EA has a betting problem, the psychological kind, I would have see pathological betting behavior on their part (for example, that ruined their finances or their relationships). Different EAās are typically doing different things with their āupdatingā at different times. Some of them are sure to have a bit of risk-seeking that is unhealthy and distorts their perspective, others are doing their best to measure their feelings of certainty as a fractional number, and still others are being cynical with their offerings of numbers. If the superforecasters among the EAās are doing so well with their probability estimates, I wish they would offer some commentary on free will, game theory, and how they model indeterminism. I would learn something.
If there were a tight feedback loop about process, if everyone understood not only the math, but also the evidence that moves your credence numbers, if there were widespread agreement that a new bit of evidence X should move probability of credence Y an amount Z , then I could believe that there was systematic care about epistemics involved in the community. I could believe that EA folks are always training to improve their credence probability assignments. I could believe that EA folks train their guts, hone their rationality, and sharpen their research skills, all at the same time.
But what actually goes on in EA seems very subjective, anyone can make up any number and claim their evidence justifies it and donāt really have to prove anything, and in fact, itās hard to make a case that is not subjective, and thereās a spread of numbers on most credences, or there really isnāt but Iām looking at the numbers and seeing most people quoting somebody else then, while everyday credences are just treated as regular beliefs (binary or constrained with experience) plus there is plenty of secondary gain potential, so I donāt believe that EA folks are using their probability estimates well at all, no.
NOTE: By āsecondary gain potentialā, I mean gain from an action other than the actionās ostensible gains. A person doing updating for its secondary gains might be developing:
feelings of belonging in the EA community.
a sense of control over their feelings of doubt or certainty.
an addiction to betting or other financial risk-taking.
a (false) sense of efficacy in rational arguments.
or something else that they would have to tell you.
I suspect that most EAās are being sloppy in their offerings of probabilities and credence levels. They offer something closer to epistemic status metrics for their own statements, which rely on poor evidence or no evidence, just a feeling, maybe of worry or disappointment at failing otherās expectations, or the converse, a high metric corresponding to a poor judgement of the quality of their own work, typically basing that on other work in EA, unsurprisingly. EA folks choose numbers to emphasize a message to their audience that reflects what theyāre comfortable sharing, not what they actually believe. Thereās some real controversy among EAās, but it comes from outside the community, apparently, through people like me.
So as I said, this whole updating thing has gone wrong in EA. EA as a research community would do better to drop updating and the bayesian kick until they have a better handle on contexts suitable for it. If superforecasting offers those contexts, then great!
Lets see superforecasters address existential risk with any success. Lets just see them do the research first so they that have some clear preliminary answers to what the dangers can shape up to be. Presuming that they are not using foresight but forecasting techniques, that should be interesting.
NOTE: Once I learn more about superforecasting, I might constrain my beliefs about how EAās update to exclude superforecasters working in their successful forecasting domains. I would do that right now, but Iām suspicious of the superforecaster reputation, so Iāll wait until I can learn more.