I asked an AI safety researcher āSuppose your research project went as well as it could possibly go; how would it make it easier to align powerful AI systems?ā, and they said that they hadnāt really thought about that. I think that this makes your work less useful.
This seems like a deeper disagreement than youāre describing. A lot of research in academia (ex: much of math) involves playing with ideas that seem poorly understood, trying to figure out whatās going on. Itās not really goal directed, especially not the kind of goal you can chain back to world improvement, itās more understanding directed.
For AI safety your view may still be right: one major way I could see the field going wrong is getting really into interesting problems that arenāt useful. But on the other hand itās also possible that the best path involves highly productive interest-following understanding-building research where most individual projects donāt seem promising from an end to end view. And maybe even where most arenāt useful from an end to end view!
Again, Iām not sure here at all, but I donāt think itās obvious youāre right.
This comment is a general reply to this whole thread.
Some clarifications:
I donāt think that we should require that people working in AI safety have arguments for their research which are persuasive to anyone else. Iām saying I think they should have arguments which are persuasive to them.
I think that good plans involve doing things like playing around with ideas that excite you, and learning subjects which are only plausibly related if you have a hunch it could be helpful; I do these things a lot myself.
I think thereās a distinction between having an end-to-end story for your solution strategy vs the problem youāre trying to solveāI think itās much more tractable to choose unusually important problems than to choose unusually effective research strategies.
In most fields, the reason you can pick more socially important problems is that people arenāt trying very hard to do useful work. Itās a little more surprising that you can beat the average in AI safety by trying intentionally to do useful work, but my anecdotal impression is that people who choose what problems to work on based on a model of what problems would be important to solve are still noticeably more effective.
Hereās my summary of my position here:
I think that being goal directed is very helpful to making progress on problems on a week-by-week or month-by-month scale.
I think that within most fields, some directions are much more promising than others, and backchaining is required in order to work on the promising directions. AI safety is a field like this. Math is anotherāif I decided to try to do good by going into math, Iād end up doing research which was really different from normal mathematicians. I agree with Paul Christianoās old post about this.
If I wanted to maximize my probability of solving the Riemann hypothesis, Iād probably try to pursue some crazy plan involving weird strengths of mine and my impression of blind spots of the field. However, I donāt think this is actually that relevant, because I think that the important work in AI safety (and most other fields of relevance to EA) is less competitive than solving the Riemann hypothesis, and also a less challenging mathematical problem.
I think that in my experience, people who do the best work on AI safety generally have a clear end-to-end picture of the story for what work they need to do, and people who donāt have such a clear picture rarely do work Iām very excited about. Eg I think Nate Soares and Paul Christiano are both really good AI safety researchers, and both choose their research directions very carefully based on their sense of what problems are important to solve.
Sometimes I talk to people who are skeptical of EA because they have a stronger version of the position youāre presenting hereāthey think that nothing useful ever comes of people intentionally pursuing research that they think is important, and the right strategy is to pursue what youāre most interested in.
One way of thinking about this is to imagine that there are different problems in a field, and different researchers have different comparative advantages at the problems. In one extreme case, the problems vary wildly in importance, and so the comparative advantage basically doesnāt matter and you should work on whatās most important. In the other extreme, itās really hard to get a sense of which things are likely to be more useful than other things, and your choices should be dominated by comparative advantage.
(Incidentally, you could also apply this to the more general problem of deciding what to work on as an EA. My personal sense is that the differences in values between different cause areas are big enough to basically dwarf comparative advantage arguments, but within a cause area comparative advantage is the dominant consideration.)
I would love to see a high quality investigation of historical examples here.
I mostly share your position, except that I think that you would perhaps maximize the probability of solving the Riemann hypothesis by going into paths on the frontline of current research instead of starting something new (but I imagine that there are many promising paths currently, which may be the difference).
This planners vs Hayekian genre of dilemmas seems very important to me, and it might be a crux in my career trajectory or at least impact possible projects Iām taking. I intuitively think that this question can be dissolved quite easily to make it obvious when each strategy is better, how parts of the EA world-view influences the answer and perhaps how this impacts how we think about academic research. There is also a lot of existing literature on this matter, so there might already be a satisfying argument.
If someone here is up to a (possibly adversarial) collaboration on the topic, letās do it!
The Planners vs Hayekian dillema seems related to some of the discussion in Realism about rationality, and especially this crux for Abram Demski and Rohin Shah.
Broadly, two types of strategies in technical AI alignment work are
Build a solid mathematical foundations on which to build further knowledge which would eventually serve to reason more clearly on AI alignment.
Focus on targeted problems we can see today which are directly related to risks from advanced AI, and do our best to solve these (by heuristics or tracing back to related mathematical questions).
Borrowing Vanessaās analogy of understanding the world as a castle, each floor built on the one underneath representing knowledge hierarchically built, when one wants to build a castle with unknown materials and unknown set of rules for itās construction with a specific tower top in mind, one can either start by building the groundwork well or by starting with some ideas of what can by directly below the tower top.
Planners start from the towers top, while Hayekians want to build a solid ground and add on as many well placed floors as they can.
I agree, and also immediately thought of pure mathematics as a counterexample. E.g., if oneās most important goal was to prove the Riemann hypothesis, then I claim (based on my personal experience of doing maths, though e.g. Terence Tao seems to agree) that itād be a very bad strategy to only do things where one has an end-to-end story for how they might contribute to a proof of the Riemann hypothesis. This is true especially if one is junior, but I claim it would be true even for a hypothetical person eventually proving the Riemann conjecture, except maybe in some of the very last stages of them actually figuring out the proof.
I think the history of maths also provides some suggestive examples of the dangers of requiring end-to-end stories. E.g., consider some famous open questions in Ancient mathematics that were phrased in the language of geometric constructions with ruler and compass, such as whether itās possible to āsquare the circleā. It was solved 2,000 years after it was posed using modern number theory. But if you had insisted that everyone working on it has an end-to-end story for how what theyāre doing contributes to solving that problem, I think there would have been a real risk that people continue thinking purely in ruler-and-compass terms and we never develop modern number theory in the first place.
--
The Planners vs. Hayekians distinction seems related. The way Iām understanding Buck is that he thinks that, at least within AI alignment, a Planning strategy is superior to a Hayekian one (i.e. roughly one based on optimizing robust heuristics rather than an end-to-end story).
--
One of the strongest defenses of Buckās original claim I can think of would appeal specifically to the āpreparadigmaticā stage of AI alignment. I.e. roughly the argument would be: sure, perhaps in areas where we know of heuristics that are robustly good to pursue it can sometimes be best to do so; however, the challenge with AI alignment precisely is that we do not know of such heuristics, hence there simply is no good alternative to having an end-to-end story.
I just looked up the proof of Fermatās Last Theorem, and it came about from Andrew Wiles spotting that someone else had recently proven something which could plausibly be turned into a proof, and then working on it for seven years. This seems like a data point in favor of the end-to-end models approach.
Yes, agree. Though anecdotally my impression is that Wiles is an exception, and that his strategy was seen as quite weird and unusual by his peers.
I think I agree that in general there will almost always be a point at which itās optimal to switch to a more end-to-end strategy. In Wilesās case, I donāt think his strategy would have worked if he had switched as an undergraduate, and I donāt think it would have worked if he had lived 50 years earlier (because the conceptual foundations used in the proof had not been developed yet).
This can also be a back and forth. E.g. for Fermatās Last Theorem, perhaps number theorists were justified in taking a more end-to-end approach in the 19th century because there had been little effort using then-modern tools; and indeed, I think partly stimulated by attempts to prove FLT (and actually proving it in some special cases), they developed some of the foundations of classical algebraic number theory. Maybe then people had understood that the conjecture resists attempts to prove it directly given then-current conceptual tools, and at this point it would have become more fruitful to spend more time on less direct approaches, though they could still be guided by heuristics like āitās useful to further develop the foundations of this area of maths /ā our understanding of this kind of mathematical object because we know of a certain connection to FLT, even though we wouldnāt know how exactly this could help in a proof of FLTā. Then, perhaps in Wilesās time, it was time again for more end-to-end attempts etc.
Iām not confident that this is a very accurate history of FLT, but reasonably confident that the rough pattern applies to a lot of maths.
The paper Architecting Discovery by Ed Boyden and Adam Marblestone also discusses how one can methodologically go about producing better scientific tools (which they used for Expansion Microscopy and Optogenetics).
I think the history of maths also provides some suggestive examples of the dangers of requiring end-to-end stories. E.g., consider some famous open questions in Ancient mathematics that were phrased in the language of geometric constructions with ruler and compass, such as whether itās possible to āsquare the circleā. It was solved 2,000 years after it was posed using modern number theory. But if you had insisted that everyone working on it has an end-to-end story for how what theyāre doing contributes to solving that problem, I think there would have been a real risk that people continue thinking purely in ruler-and-compass terms and we never develop modern number theory in the first place.
I think youāre interpreting me to say that people ought to have an externally validated end-to-end story; Iām actually just saying that they should have an approach which they think might be useful, which is weaker.
Thanks, I think this is a useful clarification. Iām actually not sure if I even clearly distinguished these cases in my thinking when I wrote my previous comments, but I agree the thing you quoted is primarily relevant to when end-to-end stories will be externally validated. (By which I think you mean something like: they would lead to an āobjectiveā solution, e.g. maths proof, if executed without major changes.)
The extent to which we agree depends on what counts as end-to-end story. For example, consider someone working on ML transparency claiming their research is valuable for AI alignment. My guess is:
If literally everything they can say when queried is āI donāt know how transparency helps with AI alignment, I just saw the term in some list of relevant research directionsā, then we both are quite pessimistic about the value of that work.
If they say something like āIāve made the deliberate decision not to focus on research for which I can fully argue it will be relevant to AI alignment right now. Instead, I just focus on understanding ML transparency as best as I can because I think there are many scenarios in which understanding transparency will be beneficial.ā, and then they say something showing they understand longtermist thought on AI risk, then Iām not necessarily pessimistic. Iād think they wonāt come up with their own research agenda in the next two years, but depending on the circumstances I might well be optimistic about that personās impact over their whole career, and I wouldnāt necessarily recommend them to change their approach. Iām not sure what youād think, but I think initially I read you as being pessimistic in such a case, and this was partly what I was reacting against.
If they give an end-to-end story for how their work fits within AI alignment, then all else equal I consider that to be a good sign. However, depending on the circumstances I might still think the best long-term strategy for that person is to postpone the direct pursuit of that end-to-end story and instead focus on targeted deliberate practice of some of the relevant skills, or at least complement the direct pursuit with such deliberate practice. For example, if someone is very junior, and their story says that mathematical logic is important for their work, I might recommend they grab a logic textbook and work through all the exercises. My guess is we disagree on such cases, but that the disagreement is somewhat gradual; i.e. we both agree about extreme cases, but Iād more often recommend more substantial deliberate practice.
Similar with what youāre saying about AI alignment being preparadigmatic, a major reason why trying to prove the Riemann conjecture head-on would be a bad idea is that people have already been trying to do that for a long time without success. I expect the first people to consider the conjecture approached it directly, and were reasonable to do so.
Yes, good points. I basically agree. I guess this could provide another argument in favor of Buckās original view, namely that the AI alignment problem is young and so worth attacking directly. (Though there are differences between attacking a problem directly and having an end-to-end story for how to solve it, which may be worth paying attention to.)
I think your view is also born out by some examples from the history of maths. For example, the Weil conjectures were posed in 1949, and it took āonlyā a few decades to prove them. However, some of the key steps were known from the start, it just required a lot of work and innovation to complete them. And so I think itās fair to characterize the process as a relatively direct, and ultimately successful, attempt to solve a big problem. (Indeed, this is an example of the effect where the targeted pursuit of a specific problem led to a lot of foundational/ātheoretical innovation, which has much wider uses.)
For clarity, Terry Tao argues that it is a bad strategy to work on one open problem because one should skill up first, lose some naivety and get a higher status within the community. Not because it is a better problem solving strategy.
My reading is that career/āstatus considerations are only one of at least two major reasons Tao mentions. I agree those may be less relevant in the AI alignment case, and are not centrally a criterion for how good a problem solving strategy is.
However, Tao also appeals to the required āmathematical preparationā, which fits with you mentioning skilling up and losing naivety. I do think these are central criteria for how good a problem solving strategy is. If I want to build a house, it would be a bad strategy to start putting it together with my bare hands; it would be better to first build a hammer and other tools, and understand how to use them. Similarly, it would be better to acquire and understand the relevant tools before attempting to solve a mathematical problem.
I agree with this. Perhaps we are on the same page.
But I think that this is in an important way orthogonal to the Planner vs Hayekian distinction which I think is the more crucial point here.
Iād argue that if one wants to solve a problem, it would be better to have a sort of a roadmap and to learn stuff on the way. I agree that it might be great to choose subproblems if they give you some relevant tools, but there should be a good argument as to why these tools are likely to help. In many cases, Iād expect choosing subproblems which are closer to what you really want to accomplish to help you learn more relevant tools. If you want to get better at climbing stairs, you should practice climbing stairs.
I think having a roadmap, and choosing subproblems as close as possible to the final problem, are often good strategies, perhaps in a large majority of cases.
However, I think there at least three important types of exceptions:
When itās not possible to identify any clear subproblems or their closeness to the final problem is unclear (perhaps AI alignment is an example, though I think itās less true today than it was, say, 10 years agoāat least if you buy e.g. Paul Christianoās broad agenda).
When the close, or even all known, subproblems have resisted solutions for a long time, e.g. Riemann hypothesis.
When one needs tools/āsubproblems that seem closely related only after having invested a lot of effort investigating them, rather than in advance. E.g. squaring the circleāāif you want to understand constructions with ruler and compass, do a lot of constructions with ruler and compassā was a bad strategy. Though admittedly itās unclear if one can identify examples of this type in advance unless they are also examples of one of the previous two types.
Also, I of course acknowledge that there are limits to the idea of exploring subproblems that are less closely related. For example, I think no matter what mathematical problem you want to solve, I think it would be a very bad strategy to study dung beetles or to become a priest. And to be fair, I think at least in hindsight the idea of studying close subproblems will almost always appear to be correct. To return to the example of squaring the circle: once people had realized that the set of points you can construct with ruler and compass are closed under basic algebraic operations in the complex plane, it was possible and relatively easy to see how certain problems in algebra number theory were closely related. So the problem was less that itās intrinsically better to focus on less related subproblems, but more that people didnāt properly understand what would count as helpfully related.
Regarding the first two types, I think that itās practically never the case and one can always make progressāeven if that progress is in work done on analogies or heuristically relevant techniques. The Riemann hypothesis is actually a great example of that; there are many paths currently pursued to help us understand it better, even if there arenāt any especially promising reductions (not sure if thatās the case). But I guess that your point here is that this are distinct markers for how easy is it to make progress.
What is the alternative strategy you are suggesting in those exceptions? Is it to work on problems that are weakly related and the connection is not clear but are more tractable?
If so, I think that two alternative strategies are to just try harder to find something more related or to move to a different project altogether. Of course, this all lies on a continuum so itās a matter of degree.
This seems like a deeper disagreement than youāre describing. A lot of research in academia (ex: much of math) involves playing with ideas that seem poorly understood, trying to figure out whatās going on. Itās not really goal directed, especially not the kind of goal you can chain back to world improvement, itās more understanding directed.
It reminds me of Sarah Constantinās post about the trade-off between output and external direction: https://āāsrconstantin.wordpress.com/āā2019/āā07/āā20/āāthe-costs-of-reliability/āā
For AI safety your view may still be right: one major way I could see the field going wrong is getting really into interesting problems that arenāt useful. But on the other hand itās also possible that the best path involves highly productive interest-following understanding-building research where most individual projects donāt seem promising from an end to end view. And maybe even where most arenāt useful from an end to end view!
Again, Iām not sure here at all, but I donāt think itās obvious youāre right.
This comment is a general reply to this whole thread.
Some clarifications:
I donāt think that we should require that people working in AI safety have arguments for their research which are persuasive to anyone else. Iām saying I think they should have arguments which are persuasive to them.
I think that good plans involve doing things like playing around with ideas that excite you, and learning subjects which are only plausibly related if you have a hunch it could be helpful; I do these things a lot myself.
I think thereās a distinction between having an end-to-end story for your solution strategy vs the problem youāre trying to solveāI think itās much more tractable to choose unusually important problems than to choose unusually effective research strategies.
In most fields, the reason you can pick more socially important problems is that people arenāt trying very hard to do useful work. Itās a little more surprising that you can beat the average in AI safety by trying intentionally to do useful work, but my anecdotal impression is that people who choose what problems to work on based on a model of what problems would be important to solve are still noticeably more effective.
Hereās my summary of my position here:
I think that being goal directed is very helpful to making progress on problems on a week-by-week or month-by-month scale.
I think that within most fields, some directions are much more promising than others, and backchaining is required in order to work on the promising directions. AI safety is a field like this. Math is anotherāif I decided to try to do good by going into math, Iād end up doing research which was really different from normal mathematicians. I agree with Paul Christianoās old post about this.
If I wanted to maximize my probability of solving the Riemann hypothesis, Iād probably try to pursue some crazy plan involving weird strengths of mine and my impression of blind spots of the field. However, I donāt think this is actually that relevant, because I think that the important work in AI safety (and most other fields of relevance to EA) is less competitive than solving the Riemann hypothesis, and also a less challenging mathematical problem.
I think that in my experience, people who do the best work on AI safety generally have a clear end-to-end picture of the story for what work they need to do, and people who donāt have such a clear picture rarely do work Iām very excited about. Eg I think Nate Soares and Paul Christiano are both really good AI safety researchers, and both choose their research directions very carefully based on their sense of what problems are important to solve.
Sometimes I talk to people who are skeptical of EA because they have a stronger version of the position youāre presenting hereāthey think that nothing useful ever comes of people intentionally pursuing research that they think is important, and the right strategy is to pursue what youāre most interested in.
One way of thinking about this is to imagine that there are different problems in a field, and different researchers have different comparative advantages at the problems. In one extreme case, the problems vary wildly in importance, and so the comparative advantage basically doesnāt matter and you should work on whatās most important. In the other extreme, itās really hard to get a sense of which things are likely to be more useful than other things, and your choices should be dominated by comparative advantage.
(Incidentally, you could also apply this to the more general problem of deciding what to work on as an EA. My personal sense is that the differences in values between different cause areas are big enough to basically dwarf comparative advantage arguments, but within a cause area comparative advantage is the dominant consideration.)
I would love to see a high quality investigation of historical examples here.
I mostly share your position, except that I think that you would perhaps maximize the probability of solving the Riemann hypothesis by going into paths on the frontline of current research instead of starting something new (but I imagine that there are many promising paths currently, which may be the difference).
This planners vs Hayekian genre of dilemmas seems very important to me, and it might be a crux in my career trajectory or at least impact possible projects Iām taking. I intuitively think that this question can be dissolved quite easily to make it obvious when each strategy is better, how parts of the EA world-view influences the answer and perhaps how this impacts how we think about academic research. There is also a lot of existing literature on this matter, so there might already be a satisfying argument.
If someone here is up to a (possibly adversarial) collaboration on the topic, letās do it!
The Planners vs Hayekian dillema seems related to some of the discussion in Realism about rationality, and especially this crux for Abram Demski and Rohin Shah.
Broadly, two types of strategies in technical AI alignment work are
Build a solid mathematical foundations on which to build further knowledge which would eventually serve to reason more clearly on AI alignment.
Focus on targeted problems we can see today which are directly related to risks from advanced AI, and do our best to solve these (by heuristics or tracing back to related mathematical questions).
Borrowing Vanessaās analogy of understanding the world as a castle, each floor built on the one underneath representing knowledge hierarchically built, when one wants to build a castle with unknown materials and unknown set of rules for itās construction with a specific tower top in mind, one can either start by building the groundwork well or by starting with some ideas of what can by directly below the tower top.
Planners start from the towers top, while Hayekians want to build a solid ground and add on as many well placed floors as they can.
I agree, and also immediately thought of pure mathematics as a counterexample. E.g., if oneās most important goal was to prove the Riemann hypothesis, then I claim (based on my personal experience of doing maths, though e.g. Terence Tao seems to agree) that itād be a very bad strategy to only do things where one has an end-to-end story for how they might contribute to a proof of the Riemann hypothesis. This is true especially if one is junior, but I claim it would be true even for a hypothetical person eventually proving the Riemann conjecture, except maybe in some of the very last stages of them actually figuring out the proof.
I think the history of maths also provides some suggestive examples of the dangers of requiring end-to-end stories. E.g., consider some famous open questions in Ancient mathematics that were phrased in the language of geometric constructions with ruler and compass, such as whether itās possible to āsquare the circleā. It was solved 2,000 years after it was posed using modern number theory. But if you had insisted that everyone working on it has an end-to-end story for how what theyāre doing contributes to solving that problem, I think there would have been a real risk that people continue thinking purely in ruler-and-compass terms and we never develop modern number theory in the first place.
--
The Planners vs. Hayekians distinction seems related. The way Iām understanding Buck is that he thinks that, at least within AI alignment, a Planning strategy is superior to a Hayekian one (i.e. roughly one based on optimizing robust heuristics rather than an end-to-end story).
--
One of the strongest defenses of Buckās original claim I can think of would appeal specifically to the āpreparadigmaticā stage of AI alignment. I.e. roughly the argument would be: sure, perhaps in areas where we know of heuristics that are robustly good to pursue it can sometimes be best to do so; however, the challenge with AI alignment precisely is that we do not know of such heuristics, hence there simply is no good alternative to having an end-to-end story.
I just looked up the proof of Fermatās Last Theorem, and it came about from Andrew Wiles spotting that someone else had recently proven something which could plausibly be turned into a proof, and then working on it for seven years. This seems like a data point in favor of the end-to-end models approach.
Yes, agree. Though anecdotally my impression is that Wiles is an exception, and that his strategy was seen as quite weird and unusual by his peers.
I think I agree that in general there will almost always be a point at which itās optimal to switch to a more end-to-end strategy. In Wilesās case, I donāt think his strategy would have worked if he had switched as an undergraduate, and I donāt think it would have worked if he had lived 50 years earlier (because the conceptual foundations used in the proof had not been developed yet).
This can also be a back and forth. E.g. for Fermatās Last Theorem, perhaps number theorists were justified in taking a more end-to-end approach in the 19th century because there had been little effort using then-modern tools; and indeed, I think partly stimulated by attempts to prove FLT (and actually proving it in some special cases), they developed some of the foundations of classical algebraic number theory. Maybe then people had understood that the conjecture resists attempts to prove it directly given then-current conceptual tools, and at this point it would have become more fruitful to spend more time on less direct approaches, though they could still be guided by heuristics like āitās useful to further develop the foundations of this area of maths /ā our understanding of this kind of mathematical object because we know of a certain connection to FLT, even though we wouldnāt know how exactly this could help in a proof of FLTā. Then, perhaps in Wilesās time, it was time again for more end-to-end attempts etc.
Iām not confident that this is a very accurate history of FLT, but reasonably confident that the rough pattern applies to a lot of maths.
The paper Architecting Discovery by Ed Boyden and Adam Marblestone also discusses how one can methodologically go about producing better scientific tools (which they used for Expansion Microscopy and Optogenetics).
I think youāre interpreting me to say that people ought to have an externally validated end-to-end story; Iām actually just saying that they should have an approach which they think might be useful, which is weaker.
Thanks, I think this is a useful clarification. Iām actually not sure if I even clearly distinguished these cases in my thinking when I wrote my previous comments, but I agree the thing you quoted is primarily relevant to when end-to-end stories will be externally validated. (By which I think you mean something like: they would lead to an āobjectiveā solution, e.g. maths proof, if executed without major changes.)
The extent to which we agree depends on what counts as end-to-end story. For example, consider someone working on ML transparency claiming their research is valuable for AI alignment. My guess is:
If literally everything they can say when queried is āI donāt know how transparency helps with AI alignment, I just saw the term in some list of relevant research directionsā, then we both are quite pessimistic about the value of that work.
If they say something like āIāve made the deliberate decision not to focus on research for which I can fully argue it will be relevant to AI alignment right now. Instead, I just focus on understanding ML transparency as best as I can because I think there are many scenarios in which understanding transparency will be beneficial.ā, and then they say something showing they understand longtermist thought on AI risk, then Iām not necessarily pessimistic. Iād think they wonāt come up with their own research agenda in the next two years, but depending on the circumstances I might well be optimistic about that personās impact over their whole career, and I wouldnāt necessarily recommend them to change their approach. Iām not sure what youād think, but I think initially I read you as being pessimistic in such a case, and this was partly what I was reacting against.
If they give an end-to-end story for how their work fits within AI alignment, then all else equal I consider that to be a good sign. However, depending on the circumstances I might still think the best long-term strategy for that person is to postpone the direct pursuit of that end-to-end story and instead focus on targeted deliberate practice of some of the relevant skills, or at least complement the direct pursuit with such deliberate practice. For example, if someone is very junior, and their story says that mathematical logic is important for their work, I might recommend they grab a logic textbook and work through all the exercises. My guess is we disagree on such cases, but that the disagreement is somewhat gradual; i.e. we both agree about extreme cases, but Iād more often recommend more substantial deliberate practice.
Similar with what youāre saying about AI alignment being preparadigmatic, a major reason why trying to prove the Riemann conjecture head-on would be a bad idea is that people have already been trying to do that for a long time without success. I expect the first people to consider the conjecture approached it directly, and were reasonable to do so.
Yes, good points. I basically agree. I guess this could provide another argument in favor of Buckās original view, namely that the AI alignment problem is young and so worth attacking directly. (Though there are differences between attacking a problem directly and having an end-to-end story for how to solve it, which may be worth paying attention to.)
I think your view is also born out by some examples from the history of maths. For example, the Weil conjectures were posed in 1949, and it took āonlyā a few decades to prove them. However, some of the key steps were known from the start, it just required a lot of work and innovation to complete them. And so I think itās fair to characterize the process as a relatively direct, and ultimately successful, attempt to solve a big problem. (Indeed, this is an example of the effect where the targeted pursuit of a specific problem led to a lot of foundational/ātheoretical innovation, which has much wider uses.)
For clarity, Terry Tao argues that it is a bad strategy to work on one open problem because one should skill up first, lose some naivety and get a higher status within the community. Not because it is a better problem solving strategy.
My reading is that career/āstatus considerations are only one of at least two major reasons Tao mentions. I agree those may be less relevant in the AI alignment case, and are not centrally a criterion for how good a problem solving strategy is.
However, Tao also appeals to the required āmathematical preparationā, which fits with you mentioning skilling up and losing naivety. I do think these are central criteria for how good a problem solving strategy is. If I want to build a house, it would be a bad strategy to start putting it together with my bare hands; it would be better to first build a hammer and other tools, and understand how to use them. Similarly, it would be better to acquire and understand the relevant tools before attempting to solve a mathematical problem.
I agree with this. Perhaps we are on the same page.
But I think that this is in an important way orthogonal to the Planner vs Hayekian distinction which I think is the more crucial point here.
Iād argue that if one wants to solve a problem, it would be better to have a sort of a roadmap and to learn stuff on the way. I agree that it might be great to choose subproblems if they give you some relevant tools, but there should be a good argument as to why these tools are likely to help. In many cases, Iād expect choosing subproblems which are closer to what you really want to accomplish to help you learn more relevant tools. If you want to get better at climbing stairs, you should practice climbing stairs.
I think having a roadmap, and choosing subproblems as close as possible to the final problem, are often good strategies, perhaps in a large majority of cases.
However, I think there at least three important types of exceptions:
When itās not possible to identify any clear subproblems or their closeness to the final problem is unclear (perhaps AI alignment is an example, though I think itās less true today than it was, say, 10 years agoāat least if you buy e.g. Paul Christianoās broad agenda).
When the close, or even all known, subproblems have resisted solutions for a long time, e.g. Riemann hypothesis.
When one needs tools/āsubproblems that seem closely related only after having invested a lot of effort investigating them, rather than in advance. E.g. squaring the circleāāif you want to understand constructions with ruler and compass, do a lot of constructions with ruler and compassā was a bad strategy. Though admittedly itās unclear if one can identify examples of this type in advance unless they are also examples of one of the previous two types.
Also, I of course acknowledge that there are limits to the idea of exploring subproblems that are less closely related. For example, I think no matter what mathematical problem you want to solve, I think it would be a very bad strategy to study dung beetles or to become a priest. And to be fair, I think at least in hindsight the idea of studying close subproblems will almost always appear to be correct. To return to the example of squaring the circle: once people had realized that the set of points you can construct with ruler and compass are closed under basic algebraic operations in the complex plane, it was possible and relatively easy to see how certain problems in algebra number theory were closely related. So the problem was less that itās intrinsically better to focus on less related subproblems, but more that people didnāt properly understand what would count as helpfully related.
Regarding the first two types, I think that itās practically never the case and one can always make progressāeven if that progress is in work done on analogies or heuristically relevant techniques. The Riemann hypothesis is actually a great example of that; there are many paths currently pursued to help us understand it better, even if there arenāt any especially promising reductions (not sure if thatās the case). But I guess that your point here is that this are distinct markers for how easy is it to make progress.
What is the alternative strategy you are suggesting in those exceptions? Is it to work on problems that are weakly related and the connection is not clear but are more tractable?
If so, I think that two alternative strategies are to just try harder to find something more related or to move to a different project altogether. Of course, this all lies on a continuum so itās a matter of degree.