Just found this post, coming in to comment a year late—Thanks Michael for the thoughtful post and Ozzie for the thoughtful comments!
I’m not saying that these are easy to solve, but rather, there is a mathematical strategy to generally fix them in ways that would make sense intuitively. There’s no better approach than to try to approximate the mathematical approach, or go with an approach that in-expectation does a decent job at approximating the mathematical approach.
I might agree with you about what’s (in some sense) mathematically possible (in principle). In practice, I don’t think people trying to approximate the ideal mathematical approach are going to have a ton of success (for reasons discussed in my post and quoted in Michael’s previous comment).
I don’t think searching for “an approach that in-expectation does a decent job at approximating the mathematical approach” is pragmatic.
In most important scenarios, we’re uncertain what approaches work well in-expectation. Our uncertainty about what works well in-expectation is the kind of uncertainty that’s hard to hash out in probabilities. A strict Bayesian might say, “That’s not a problem—with even more math, the uncertainty can be handled....”
While you can keep adding more math and technical patches to try and ground decision making in Bayesianism, pragmatism eventually pushes me in other directions. I think David Chapman explains this idea a hell of a lot better than I can in Rationalism’s Responses To Trouble.
Getting more concrete: Trusting my gut or listening to domain experts might turn out to be approaches that work well in some situation. If one of these approaches works, I’m sure someone could argue in hindsight that an approach works because it approximates an idealized mathematical approach. But I’m skeptical of the merits of work done in the reverse (i.e., trying to discover non-math approaches by looking for things that will approximate idealized mathematical approaches).
Hmm, I feel like you may be framing things quite differently to how I would, or something. My initial reaction to your comment is something like:
It seems usefully to conceptually separate data collection from data processing, where by the latter I mean using that data to arrive at probability estimates and decisions.
I think Bayesianism (in the sense of using Bayes’ theorem and a Bayesian interpretation of probability) and “math and technical patches” might tend to be part of the data processing, not the data collection. (Though they could also guide what data to look for. And this is just a rough conceptual divide.)
When Ozzie wrote about going with “an approach that in-expectation does a decent job at approximating the mathematical approach”, he was specifically referring to dealing with the optimizer’s curse. I’d consider this part of data processing.
Meanwhile, my intuitions (i.e., gut reactions) and what experts say are data. Attending to them is data collection, and then we have to decide how to integrate that with other things to arrive at probability estimates and decisions.
I don’t think we should see ourselves as deciding between either Bayesianism and “math and technical patches” or paying attention to my intuitions and domain experts. You can feed all sorts of evidence into Bayes theorem. I doubt any EA would argue we should form conclusions from “Bayesianism and math alone”, without using any data from the world (including even their intuitive sense of what numbers to plug in, or whether people they share their findings with seem skeptical). I’m not even sure what that’d look like.
And I think my intuitions or what domain experts says can very easily be made sense of as valid data within a Bayesian framework. Generally, my intuitions and experts are more likely to indicate X is true in worlds where X is true than where it’s not. This effect is stronger when the conditions for intuitive expertise are met, when experts’ incentives seem to be well aligned with seeking and sharing truth, etc. This effect is weaker when it seems that there are strong biases or misaligned incentives at play, or when it seems there might be.
(Perhaps this is talking past you? I’m not sure I understood your argument.)
I largely agree with what you said in this comment, though I’d say the line between data collection and data processing is often blurred in real-world scenarios.
I think we are talking past each other (not in a bad faith way though!), so I want to stop myself from digging us deeper into an unproductive rabbit hole.
Just found this post, coming in to comment a year late—Thanks Michael for the thoughtful post and Ozzie for the thoughtful comments!
I might agree with you about what’s (in some sense) mathematically possible (in principle). In practice, I don’t think people trying to approximate the ideal mathematical approach are going to have a ton of success (for reasons discussed in my post and quoted in Michael’s previous comment).
I don’t think searching for “an approach that in-expectation does a decent job at approximating the mathematical approach” is pragmatic.
In most important scenarios, we’re uncertain what approaches work well in-expectation. Our uncertainty about what works well in-expectation is the kind of uncertainty that’s hard to hash out in probabilities. A strict Bayesian might say, “That’s not a problem—with even more math, the uncertainty can be handled....”
While you can keep adding more math and technical patches to try and ground decision making in Bayesianism, pragmatism eventually pushes me in other directions. I think David Chapman explains this idea a hell of a lot better than I can in Rationalism’s Responses To Trouble.
Getting more concrete:
Trusting my gut or listening to domain experts might turn out to be approaches that work well in some situation. If one of these approaches works, I’m sure someone could argue in hindsight that an approach works because it approximates an idealized mathematical approach. But I’m skeptical of the merits of work done in the reverse (i.e., trying to discover non-math approaches by looking for things that will approximate idealized mathematical approaches).
Hmm, I feel like you may be framing things quite differently to how I would, or something. My initial reaction to your comment is something like:
It seems usefully to conceptually separate data collection from data processing, where by the latter I mean using that data to arrive at probability estimates and decisions.
I think Bayesianism (in the sense of using Bayes’ theorem and a Bayesian interpretation of probability) and “math and technical patches” might tend to be part of the data processing, not the data collection. (Though they could also guide what data to look for. And this is just a rough conceptual divide.)
When Ozzie wrote about going with “an approach that in-expectation does a decent job at approximating the mathematical approach”, he was specifically referring to dealing with the optimizer’s curse. I’d consider this part of data processing.
Meanwhile, my intuitions (i.e., gut reactions) and what experts say are data. Attending to them is data collection, and then we have to decide how to integrate that with other things to arrive at probability estimates and decisions.
I don’t think we should see ourselves as deciding between either Bayesianism and “math and technical patches” or paying attention to my intuitions and domain experts. You can feed all sorts of evidence into Bayes theorem. I doubt any EA would argue we should form conclusions from “Bayesianism and math alone”, without using any data from the world (including even their intuitive sense of what numbers to plug in, or whether people they share their findings with seem skeptical). I’m not even sure what that’d look like.
And I think my intuitions or what domain experts says can very easily be made sense of as valid data within a Bayesian framework. Generally, my intuitions and experts are more likely to indicate X is true in worlds where X is true than where it’s not. This effect is stronger when the conditions for intuitive expertise are met, when experts’ incentives seem to be well aligned with seeking and sharing truth, etc. This effect is weaker when it seems that there are strong biases or misaligned incentives at play, or when it seems there might be.
(Perhaps this is talking past you? I’m not sure I understood your argument.)
I largely agree with what you said in this comment, though I’d say the line between data collection and data processing is often blurred in real-world scenarios.
I think we are talking past each other (not in a bad faith way though!), so I want to stop myself from digging us deeper into an unproductive rabbit hole.