Hm… Some of this would take a lot more writing than would make sense in a blog post.
On overconfidence in probabilities vs. intuitions:
I think I mostly agree with you. One cool thing about probabilities is that they can be much more straightforwardly verified/falsified and measured using metrics for calibration. If we had much larger systems, I believe we could do a great deal of work to better ensure calibration with defined probabilities.
“should eventually be used for most things”
I’m not saying that humans should come up with unique probabilities for most things on most days. One example I’d consider “used for most things” is a case where an AI uses probabilities to tell humans which actions seem the best, and humans go with what the AI states. Similar could be said for “a trusted committee” that uses probabilities as an in-between.
“we could learn to get much better than them later”
I think there are strong claims that topics like Bayes, Causality, Rationality even, are still relatively poorly understood, and may be advanced a lot in the next 30-100 years. As we get better with them, I predict we would get better at formal modeling.
I reject expected utility theory (and related stances such as cost-benefit analysis), at least if it comes as a formal way of spelling out a maximizing consequentialist moral stance which does not properly incorporate rights.
This is a complicated topic. It think a lot of Utilitarians/Consequentialists wouldn’t deem many interpretations of rights as metaphysical or terminally-valuable things. Another way to look at it would be to attempt to map the rights to a utility function. Utility functions require very, very few conditions. I’m personally a bit cynical of values that can’t be mapped to utility functions, if even in a highly-uncertain way.
But Chris Smith quotes the proposed solution, and then writes…
It’s clear Chris Smith has thought about some of this topic a fair bit, but my impression is that I disagree with him. It’s quite possible that much of the disagreement is semantic; where he says ‘this solution is unworkable’ I may say, ‘the solution results in a very wide amount of uncertainty’. I think it’s clear to everyone (the main researchers anyway) that there’s little data about many of these topics, and that Bayesian or any kind of statistical manipulations can’t fundamentally convert “very little data” into “a great deal of confidence”.
Kudos for identifing that post. The main solution I was referring to was the one described in the second comment:
In statistics the solution you describe is called Hierarchical or Multilevel Modeling. You assume that you data is drawn from a set of distributions which have their parameters drawn from another distribution. This automatically shrinks your estimates of the distributions towards the mean. I think it’s a pretty useful trick to know and I think it would be good to do a writeup but I think you might need to have a decent grasp of bayesian statistics first.
I’m not saying that these are easy to solve, but rather, there is a mathematical strategy to generally fix them in ways that would make sense intuitively. There’s no better approach than to try to approximate the mathematical approach, or go with an approach that in-expectation does a decent job at approximating the mathematical approach.
Just found this post, coming in to comment a year late—Thanks Michael for the thoughtful post and Ozzie for the thoughtful comments!
I’m not saying that these are easy to solve, but rather, there is a mathematical strategy to generally fix them in ways that would make sense intuitively. There’s no better approach than to try to approximate the mathematical approach, or go with an approach that in-expectation does a decent job at approximating the mathematical approach.
I might agree with you about what’s (in some sense) mathematically possible (in principle). In practice, I don’t think people trying to approximate the ideal mathematical approach are going to have a ton of success (for reasons discussed in my post and quoted in Michael’s previous comment).
I don’t think searching for “an approach that in-expectation does a decent job at approximating the mathematical approach” is pragmatic.
In most important scenarios, we’re uncertain what approaches work well in-expectation. Our uncertainty about what works well in-expectation is the kind of uncertainty that’s hard to hash out in probabilities. A strict Bayesian might say, “That’s not a problem—with even more math, the uncertainty can be handled....”
While you can keep adding more math and technical patches to try and ground decision making in Bayesianism, pragmatism eventually pushes me in other directions. I think David Chapman explains this idea a hell of a lot better than I can in Rationalism’s Responses To Trouble.
Getting more concrete: Trusting my gut or listening to domain experts might turn out to be approaches that work well in some situation. If one of these approaches works, I’m sure someone could argue in hindsight that an approach works because it approximates an idealized mathematical approach. But I’m skeptical of the merits of work done in the reverse (i.e., trying to discover non-math approaches by looking for things that will approximate idealized mathematical approaches).
Hmm, I feel like you may be framing things quite differently to how I would, or something. My initial reaction to your comment is something like:
It seems usefully to conceptually separate data collection from data processing, where by the latter I mean using that data to arrive at probability estimates and decisions.
I think Bayesianism (in the sense of using Bayes’ theorem and a Bayesian interpretation of probability) and “math and technical patches” might tend to be part of the data processing, not the data collection. (Though they could also guide what data to look for. And this is just a rough conceptual divide.)
When Ozzie wrote about going with “an approach that in-expectation does a decent job at approximating the mathematical approach”, he was specifically referring to dealing with the optimizer’s curse. I’d consider this part of data processing.
Meanwhile, my intuitions (i.e., gut reactions) and what experts say are data. Attending to them is data collection, and then we have to decide how to integrate that with other things to arrive at probability estimates and decisions.
I don’t think we should see ourselves as deciding between either Bayesianism and “math and technical patches” or paying attention to my intuitions and domain experts. You can feed all sorts of evidence into Bayes theorem. I doubt any EA would argue we should form conclusions from “Bayesianism and math alone”, without using any data from the world (including even their intuitive sense of what numbers to plug in, or whether people they share their findings with seem skeptical). I’m not even sure what that’d look like.
And I think my intuitions or what domain experts says can very easily be made sense of as valid data within a Bayesian framework. Generally, my intuitions and experts are more likely to indicate X is true in worlds where X is true than where it’s not. This effect is stronger when the conditions for intuitive expertise are met, when experts’ incentives seem to be well aligned with seeking and sharing truth, etc. This effect is weaker when it seems that there are strong biases or misaligned incentives at play, or when it seems there might be.
(Perhaps this is talking past you? I’m not sure I understood your argument.)
I largely agree with what you said in this comment, though I’d say the line between data collection and data processing is often blurred in real-world scenarios.
I think we are talking past each other (not in a bad faith way though!), so I want to stop myself from digging us deeper into an unproductive rabbit hole.
Hm… Some of this would take a lot more writing than would make sense in a blog post.
On overconfidence in probabilities vs. intuitions: I think I mostly agree with you. One cool thing about probabilities is that they can be much more straightforwardly verified/falsified and measured using metrics for calibration. If we had much larger systems, I believe we could do a great deal of work to better ensure calibration with defined probabilities.
I’m not saying that humans should come up with unique probabilities for most things on most days. One example I’d consider “used for most things” is a case where an AI uses probabilities to tell humans which actions seem the best, and humans go with what the AI states. Similar could be said for “a trusted committee” that uses probabilities as an in-between.
I think there are strong claims that topics like Bayes, Causality, Rationality even, are still relatively poorly understood, and may be advanced a lot in the next 30-100 years. As we get better with them, I predict we would get better at formal modeling.
This is a complicated topic. It think a lot of Utilitarians/Consequentialists wouldn’t deem many interpretations of rights as metaphysical or terminally-valuable things. Another way to look at it would be to attempt to map the rights to a utility function. Utility functions require very, very few conditions. I’m personally a bit cynical of values that can’t be mapped to utility functions, if even in a highly-uncertain way.
Kudos for identifing that post. The main solution I was referring to was the one described in the second comment:
The optimizer’s curse arguably is basically within the class of Goodhart-like problems https://www.lesswrong.com/posts/5bd75cc58225bf06703754b2/the-three-levels-of-goodhart-s-curse
I’m not saying that these are easy to solve, but rather, there is a mathematical strategy to generally fix them in ways that would make sense intuitively. There’s no better approach than to try to approximate the mathematical approach, or go with an approach that in-expectation does a decent job at approximating the mathematical approach.
That all seems to make sense to me. Thanks for the interesting reply!
Just found this post, coming in to comment a year late—Thanks Michael for the thoughtful post and Ozzie for the thoughtful comments!
I might agree with you about what’s (in some sense) mathematically possible (in principle). In practice, I don’t think people trying to approximate the ideal mathematical approach are going to have a ton of success (for reasons discussed in my post and quoted in Michael’s previous comment).
I don’t think searching for “an approach that in-expectation does a decent job at approximating the mathematical approach” is pragmatic.
In most important scenarios, we’re uncertain what approaches work well in-expectation. Our uncertainty about what works well in-expectation is the kind of uncertainty that’s hard to hash out in probabilities. A strict Bayesian might say, “That’s not a problem—with even more math, the uncertainty can be handled....”
While you can keep adding more math and technical patches to try and ground decision making in Bayesianism, pragmatism eventually pushes me in other directions. I think David Chapman explains this idea a hell of a lot better than I can in Rationalism’s Responses To Trouble.
Getting more concrete:
Trusting my gut or listening to domain experts might turn out to be approaches that work well in some situation. If one of these approaches works, I’m sure someone could argue in hindsight that an approach works because it approximates an idealized mathematical approach. But I’m skeptical of the merits of work done in the reverse (i.e., trying to discover non-math approaches by looking for things that will approximate idealized mathematical approaches).
Hmm, I feel like you may be framing things quite differently to how I would, or something. My initial reaction to your comment is something like:
It seems usefully to conceptually separate data collection from data processing, where by the latter I mean using that data to arrive at probability estimates and decisions.
I think Bayesianism (in the sense of using Bayes’ theorem and a Bayesian interpretation of probability) and “math and technical patches” might tend to be part of the data processing, not the data collection. (Though they could also guide what data to look for. And this is just a rough conceptual divide.)
When Ozzie wrote about going with “an approach that in-expectation does a decent job at approximating the mathematical approach”, he was specifically referring to dealing with the optimizer’s curse. I’d consider this part of data processing.
Meanwhile, my intuitions (i.e., gut reactions) and what experts say are data. Attending to them is data collection, and then we have to decide how to integrate that with other things to arrive at probability estimates and decisions.
I don’t think we should see ourselves as deciding between either Bayesianism and “math and technical patches” or paying attention to my intuitions and domain experts. You can feed all sorts of evidence into Bayes theorem. I doubt any EA would argue we should form conclusions from “Bayesianism and math alone”, without using any data from the world (including even their intuitive sense of what numbers to plug in, or whether people they share their findings with seem skeptical). I’m not even sure what that’d look like.
And I think my intuitions or what domain experts says can very easily be made sense of as valid data within a Bayesian framework. Generally, my intuitions and experts are more likely to indicate X is true in worlds where X is true than where it’s not. This effect is stronger when the conditions for intuitive expertise are met, when experts’ incentives seem to be well aligned with seeking and sharing truth, etc. This effect is weaker when it seems that there are strong biases or misaligned incentives at play, or when it seems there might be.
(Perhaps this is talking past you? I’m not sure I understood your argument.)
I largely agree with what you said in this comment, though I’d say the line between data collection and data processing is often blurred in real-world scenarios.
I think we are talking past each other (not in a bad faith way though!), so I want to stop myself from digging us deeper into an unproductive rabbit hole.