In Tetlock’s book Superforecasting, he distinguishes between two skills related to forecasting: generating questions, and answering them. This “disentanglement research” business sounds more like the first sort of work. Unfortunately, Tetlock’s book focuses on the second skill, but I do believe he talks some about the first skill (e.g. giving examples of people who are good at it).
I would imagine that for generating questions, curiosity and creativity are useful. Unfortunately, the Effective Altruism movement seems to be bad at creativity.
John Cleese gave this great talk about creativity in which he distinguishes between two mental modes, “open mode” and “closed mode”. Open mode is good for generating ideas, whereas closed mode is good for accomplishing well-defined tasks. It seems to me that for a lot of different reasons, the topic of AI strategy might put a person in closed mode:
Ethical obligation—Effective altruism is often framed as an ethical obligation. If I recall correctly, a Facebook poll indicated that around half of the EA community sees EA as more of an obligation than an opportunity. Obligations don’t typically create a feeling of playfulness.
Size of the problem—Paul Graham writes: “Big problems are terrifying. There’s an almost physical pain in facing them.” AI safety strategy is almost the biggest problem imaginable.
Big names—People like Nick Bostrom, Eliezer Yudkowsky, and Eric Drexler have a very high level of prestige within the EA community. (The status difference between them and your average EA is greater than what I’ve observed between the students & the professor in any college class I remember taking.) Eliezer in particular can get very grumpy with you if you disagree with him. I’ve noticed that I’m much more apt to generate ideas if I see myself as being at the top of the status hierarchy, and if there is no penalty for coming up with a “bad” idea (even a bad idea can be a good starting point). One idea for solving the EA community’s creativity problem is to encourage more EAs to develop Richard Feynman-level indifference to our local status norms.
Urgency—As you state in this post, every second counts! Unfortunately urgency typically has the effect of triggering closed mode.
Difficulty—As you state in this post, many brilliant people have tried & failed. For some people, this fact is likely to create a sense of intimidation which precludes creativity.
For curiosity, one useful exercise I’ve found is Anna Salamon’s practice of setting a 7-minute timer and trying to think of as many questions as possible within that period. The common pattern here seems to be “quantity over quality”. If you’re in a mental state where you feel a small amount of reinforcement for a bad idea, and a large amount of reinforcement for a good idea, don’t be surprised if a torrent of ideas follows (some of which are good).
Another practice I’ve found useful is keeping a notebook. Harnessing “ambient thought” and recording ideas as they come to me, in the appropriate notebook page, seems to be much more efficient on a per-minute basis than dedicated brainstorming.
If I was attacking this problem, my overall strategic approach would differ a little from what you are describing here.
I would place less emphasis on intellectual centralization and more emphasis on encouraging people to develop idiosyncraticperspectives/form their own ontologies. Rationale: if many separately developed idiosyncratic perspectives all predict that a particular action X is desirable, that is good evidence that we should do X. There’s an analogy to stock trading here. (Relatedly, the finance/venture capital industry might be the segment of society that has the most domain expertise related to predicting the future, modulo principle-agent problems that come with investing other peoples’ money. Please let me know if you can think of other candidates… perhaps the intelligence community?)
Discipline could be useful for reading books & passing classes which expand one’s library of concepts, but once you get to the original reasoning part, discipline gets less useful. Centralization could be useful for making sure that the space of ideas relevant to AI strategy gets thoroughly covered through our collective study, and for helping people find intellectual collaborators. But I would go for beers, whiteboards, and wikis with long lists of crowdsourced pros and cons, structured to maximize the probability that usefully related ideas will at one point or another be co-located in someone’s working memory, before any kind of standard curriculum. I suspect it’s better to see AI strategy as a fundamentally interdisciplinary endeavor. (It might be useful to look at successful interdisciplinary research groups such as the Santa Fe Institute for ideas.) And forget all that astronomical waste nonsense for a moment. We are in a simulation. We score 1 point if we get a positive singularity, 0 points otherwise. Where is the loophole in the game’s rules that the designers didn’t plan for?
[Disclaimer: I haven’t made a serious effort to survey the literature or systematically understand the recommendations of experts on either creativity or curiosity, and everything in this comment is just made up of bits and pieces I picked up here and there. If you agree with my hunch that creativity/curiosity are a core part of the problem, it might be worth doing a serious lit review/systematically reading authors who write about this stuff such as Thomas Kuhn, plus reading innovators in various fields who have written about their creative process.]
Another thought: Given the nature of this problem, I wonder why the focus is on enabling EAs to discover AI strategy vs trying to gather ideas from experts who are outside the community. Most college professors have office hours you can go to and ask questions. Existing experts aren’t suffering from any of the issues that might put EAs in closed mode, and they already have the deep expertise it would take years for us to accumulate. I could imagine an event like the Asilomar AI conference, but for AI safety strategy, where you invite leading experts in every field that seems relevant, do the beer and whiteboards thing, and see what people come up with. (A gathering size much smaller than the Asilomar conference might be optimal for idea generation. I think it’d be interesting to sponsor independent teams where each team consists of one deep learning expert, one AI venture capitalist, one game theory person, one policy person, one historian, one EA/rationalist, etc. and then see if the teams end up agreeing about anything.)
Are there any best practices for getting academics interested in problems?
I run a group for creatives on Facebook called Altruistic Ideas. In it, I have worked to foster a creative culture. I’ve also written about the differences between the EA and rationality cultures vs. the culture creatives need. If this might be useful for anyone’s EA goals, please feel free to message me.
I’d would point out that you may need discipline to do experiments based upon your creative thoughts (if the information you need is not available). If you can’t check your original reasoning against the world, you are adrift in a sea of possibilities.
By “independent” I mean “thinking about something without considering others’ thoughts on it” or something to that effect… it seems easy for people’s thoughts to converge too much if they aren’t allowed to develop in isolation.
Thinking about it now, though, I wonder if there isn’t some even better middle ground; in my experience, group brainstorming can be much more productive than independent thought as I’ve described it.
There is a very high-level analogy with evolution: I imagine sexual reproduction might create more diversity in a population than horizontal gene transfer, since in the latter case, an idea(=gene) which seems good could rapidly become universal, and thus “local optima” might be more of a problem for the population (I have no idea if that’s actually how this works biologically… in fact, it seems like it might not be, since at least some viruses/bacteria seem to do a great job of rapidly mutating to become resistant to defences/treatments.)
Carrick, this is an excellent post. I agree with most of the points that you make. I would, however, like to call attention to the wide consensus that exists in relation to acting prematurely.
As you observe, there are often path dependencies at play in AI strategy. Ill-conceived early actions can amplify the difficulty of taking corrective action at a later date. Under ideal circumstances, we would act under as close to certainty as possible. Achieving this ideal, however, is impractical for several interrelated reasons:
AI strategy is replete with wicked problems. The confidence that we can have in many (most?) of our policy recommendations must necessarily be relatively low. If the marginal costs of further research are high, then undertaking that research may not be worthwhile.
Delaying policy recommendations can sometimes be as harmful as or more harmful than making sub-par policy recommendations. There are several reasons for this. First, there are direct costs (e.g., lives lost prior to implementing sanitary standards). Second, delays allow other actors—most of whom are less concerned with rigor and welfare—to make relative gains in implementing their favored policies. If outcomes are path dependent, then inaction from AI strategists can lead to worse effects than missteps. Third, other actors are likely to gain influence if AI strategists delay. Opaque incentive structures and informal networks litter the path from ideation to policymaking. Even if there are not path dependencies baked into the policies themselves, there are sociopolitical path dependencies in the policymaking process. Gaining clout at an early stage tends to increase later influence. If AI strategists are unwilling to recommend policies, others will do so and reap the reputational gains entailed. Inversely, increased visibility may confer legitimacy to AI strategy as a discipline.
Policy communities in multiple countries are becoming more aware of AI, and policymaking activity is poised to increase. China’s national AI strategy, released several months ago, is a long-range plan, the implementation of which is being carried out by top officials. For the CCP, AI is not a marginal issue. Westerners will look to Chinese policies to inform their own decisions. In Washington, think tanks are increasingly recognizing the importance of AI. The Center for a New American Security, for example, now has a dedicated AI program (https://www.cnas.org/research/technology-and-national-security/artificial-intelligence) and is actively hiring. Other influential organizations are following suit. While DC policymakers paid little attention to AlphaGo, they definitely noticed Putin’s comments on AI’s strategic importance earlier this month. As someone with an inside vantage point, I can say with a high degree of confidence that AI will not remain neglected for long. Inaction on the part of AI strategists will not mean an absence of policy; it will mean the implementation of less considered policy.
As policy discussions in relation to AI become more commonplace and more ideologically motivated, EAs will likely have less ability to influence outcomes, ceteris paribus (hence Carrick’s call for individuals to build career capital). Even if we are uncertain about specific recommendations—uncertainty that may be intractable—we will need to claim a seat at the table or risk being sidelined.
There are also many advantages to starting early. To offer a few:
If AI strategists are early movers, they can wield disproportionate influence in framing the discourse. Since anchoring effects can be large, introducing policymakers to AI through the lens of safety rather than, say, national military advantage is probably quite positive in expectation.
Making policy recommendations can be useful in outsourcing cognitive labor. Once an idea becomes public, others can begin working on it. Research rarely becomes policy overnight. In the interim period, proponents and critics alike can refine thinking and increase the analytical power brought to bear on a topic. This enables greater scrutiny for longer-range thought that has no realistic path to near-term implementation, and may result in fewer unidentified considerations.
Taking reversible harmful actions at an early stage allows us to learn from our mistakes. If these mistakes are difficult to avoid ex ante, and we wait until later to make them, the consequences are likely to be more severe. Of course, we may not know which actions are reversible. This indicates to me that researching path dependence in policymaking would be valuable.
This is not a call for immediate action, and it is not to suggest that we should be irresponsible in making recommendations. I do, however, think that we should increasingly question the consensus around inaction and begin to consider more seriously how much uncertainty we are willing to accept, as well as when and how to take a more proactive approach to implementation.
I think it is important to note that in the political world there is the vision of two phases of AI development, narrow AI and general AI.
Narrow AI is happening now. The 30+% job loss predictions in the next 20 years, all narrow AI. This is what people in the political sphere are preparing for, from my exposure to it.
General AI is conveniently predicted more that 20 years away, so people aren’t thinking about it because they don’t know what it will look like and they have problems today to deal with.
Getting this policy response right to narrow AI does have a large impact. Large scale unemployment could destabilize countries, causing economic woes and potentially war.
So perhaps people interested in general AI policy should get involved with narrow AI policy, but make it clear that this is the first battle in a war, not the whole thing. This would place them well and they could build up reputations etc. They could be be in contact with the disentanglers so that when the general AI picture is clearer, they can make policy recommendations.
I’d love it if the narrow-general AI split was reflected in all types of AI work.
It’s been 3 years now. Is it possible to do a retake evaluating the current situation of the Disentanglement and if there has been growth in possibilities to work in AI strategy & policy (be in implementation or research)?
Thanks for your thoughts on this. I found this really helpful and I think 80′000 hours could maybe consider linking to it on the AI policy guide.
Disentanglement research feels like a valid concept, and it’s great to see it exposed here. But given how much weight pivots on the idea and how much uncertainty surrounds identifying these skills, it seems like disentanglement research is a subject that is itself asking for further disentanglement! Perhaps it could be a trial question for any prospective disentanglers out there.
You’ve given examples of some entangled and under-defined questions in AI policy and provided the example of Bostrom as exhibiting strong disentanglement skills; Ben has detailed an example of an AI strategy question that seems to require some sort of “detangling” skill; Jade has given an illuminative abstract picture. These are each very helpful. But so far, the examples are either exclusively AI strategy related or entirely abstract. The process of identifying the general attributes of good disentanglers and disentanglement research might be assisted by providing a broader range of examples to include instances of disentanglement research outside of the field of AI strategy. Both answered and unanswered research questions of this sort might be useful. (I admit to being unable to think of any good examples right now)
Moving away from disentanglement, I’ve been interested for some time by your fourth, tentative suggestion for existing policy-type recommendations to
fund joint intergovernmental research projects located in relatively geopolitically neutral countries with open membership and a strong commitment to a common good principle.
This is a subject that I haven’t been able to find much written material on—if you’re aware of any I’d be very interested to know about it. It isn’t completely clear whether or how to push for an idea like this. Additionally, based on the lack of literature, it feels like this hasn’t received as much thought as it should, even in an exploratory sense (but being outside of a strategy research cluster, I could be wrong on this). You mention that race dynamics are easier to start than stop, meanwhile early intergovernmental initiatives are one of the few tools that can plausibly prevent/slow/stop international races of this sort. These lead me to believe that this ‘recommendation’ is actually more of a high priority research area. Exploring this area appears robustly positive in expectation. I’d be interested to hear other perspectives on this subject and to know whether any groups or individuals are currently working/thinking about it, and if not, how research on it might best be started, if indeed it should be.
For five years, my favorite subject to read about was talent. Unlike developmental psychologists, I did not spend most of my learning time on learning disabilities. I also did a lot of intuition calibration which helps me detect various neurological differences in people. Thus, I have a rare area of knowledge and an unusual skill which may be useful for assisting with figuring out what types of people have a particular kind of potential, what they’re like, what’s correlated with their talent(s), what they might need, and how to find and identify them. If any fellow EAs can put this to use, feel free to message me.
I would broadly agree. I think this is an important post and I agree with most of the ways to prepare. I think we are not there yet for large scale AI policy/strategy.
There are few things that I would highlight as additions.
1) We need to cultivate the skills of disentanglement. Different people might be differently suited, but like all skills it is one that works better with practice and people to practice with. Lesswrong is trying to place itself as that kind of place. It is having a little resurgence with the new website www.lesserwrong.com. For example there has been lots of interesting discussion on the problems of Goodheart’s law, which will be necessary to at least somewhat solve if we are to get AISafety groups that actually do AISafety research and don’t just optimise some research output metric to get funding.
I am not sure if lesswrong is the correct place, but we do need places for disentanglers to grow.
2) I would also like to highlight the fact that we don’t understand intelligence and that there have been lots of people studying it for a long time (psychologists etc) that I don’t think we do enough to bring into discussing artificial versions of the thing they have studied.
Lots of work on policy side of AI safety models it as utility maximimising agent in the economic style. I am pretty skeptical that is a good model of humans or of the AIs we will create. Figuring out what better models might be, is on the top of my personal priority list.
Edited to add 3) It seems like a sensible policy is to fund a competition in the style of at super forecasting aimed at AI and related technologies. This should give you some idea of the accuracy of peoples view on technology development/forecasting.
I would caution that we are also in the space of wicked problems so it may be there is never a complete certainty of the way we should move.
If you’re an EA who wants to work on AI policy/strategy (including in support roles), you should absolutely get in touch with 80,000 Hours about coaching. Often, we’ve been able to help people interested in the area clarify how they can contribute, made introductions etc.
We need disentanglement research examples. I tried using Google to search intelligence.org and www.fhi.ox.ac.uk for the term “disentanglement” and received zero results for both. What I need to determine whether I should pursue this path is three examples of good disentanglement research. Before reading the study or book for the examples, I will need a very quick gist—a sentence or three that summarizes what each example is about. An oversimplification is okay as long as this is mentioned and we’re given a link to a paper or something so we can understand it correctly if we choose to look into it further. Additionally, I need to be shown a list of open questions.
If I am the only person who asked for this, then your article has not been very effective at getting new people to try out disentanglement research. The obstacle of not even knowing what, specifically, disentanglement research looks like would very effectively prevent a new person from getting into it. I think it would be a really good idea to write a follow-up article that contains the three examples of disentanglement research, the quick gists of what’s contained in each example, and the list of open questions. That information has a chance to get someone new involved.
AI policy is important, but we don’t really know where to begin at the object level
You can potentially do 1 of 3 things, ATM:
A. “disentanglement” research:
B. operational support for (e.g.) FHI
C. get in position to influence policy, and wait for policy objectives to be cleared up
Get in touch / Apply to FHI!
I think this is broadly correct, but have a lot of questions and quibbles.
I found “disentanglement” unclear. [14] gave the clearest idea of what this might look like. A simple toy example would help a lot.
Can you give some idea of what an operations role looks like? I find it difficult to visualize, and I think uncertainty makes it less appealling.
Do you have any thoughts on why operations roles aren’t being filled?
One more policy that seems worth starting on: programs that build international connections between researchers (especially around policy-relevant issues of AI (i.e. ethics/safety)).
The timelines for effective interventions in some policy areas may be short (e.g. 1-5 years), and it may not be possible to wait for disentanglement to be “finished”.
Is it reasonable to expect the “disentanglement bottleneck” to be cleared at all? Would disentanglement actually make policy goals clear enough? Trying to anticipate all the potential pitfalls of policies is a bit like trying to anticipate all the potential pitfalls of a particular AI design or reward specification… fortunately, there is a bit of a disanalogy in that we are more likely to have a chance to correct mistakes with policy (although that still could be very hard/impossible). It seems plausible that “start iterating and create feedback loops” is a better alternative to the “wait until things are clearer” strategy.
That’s the TLDR that I took away from the article too.
I agree that “disentanglement” is unclear. The skillset that I previously thought was needed for this was something like IQ + practical groundedness + general knowledge + conceptual clarity, and that feels mostly to be confirmed by the present article.
It seems plausible that “start iterating and create feedback loops” is a better alternative to the “wait until things are clearer” strategy.
I have some lingering doubts here as well. I would flesh out an objection to the ‘disentanglement’-focus as follows: AI strategy depends critically on government, some academic communities and some companies, that are complex organizations. (Suppose that) complex organizations are best understood by an empirical/bottom-up approach, rather than by top-down theorizing. Consider the medical establishment that I have experience with. If I got ten smart effective altruists to generate mutually exclusive collectively exhaustive (MECE) hypotheses about it, as the article proposes doing for AI strategy, they would, roughly speaking, hallucinate some nonsense, that could be invalidated in minutes by someone with years of experience in the domain. So if AI strategy depends in critical components on the nature of complex institutions, then what we need for this research may be, rather than conceptual disentanglement, something more like high-level operational experience of these domains. Since it’s hard to find such people, we may want to spend the intervening time interacting with these institutions or working within them on less important issues. Compared to this article, this perspective would de-emphasize the importance of disentanglement, while maintaining the emphasis on entering these institutions, and increasing the emphasis on interacting with and making connections within these institutions.
Given your background, I will take as given your suggestion that disentanglement research is both very important and a very rare skill. With that said, I feel like there’s a reasonable meta-solution here, one that’s at least worth investigating. Since you’ve identified at least one good disentaglement researcher (eg, Nick Bolstrom), have you considered asking them to design a test to assess possible researchers?
The suggestion may sound a bit silly, so I’ll elaborate. I read your article and found it compelling. I may or may not be a good disentanglement researcher, but per your article, I probably am not. So, your article has simultaneously raised my awareness of the issue and dissuaded me from possibly helping with it. The initial pessimism, followed by your suggestion to “Read around in the area, find something sticky you think you might be able to disentangle, and take a run at it”, all but guarantees low followthrough from your audience.
Ben Garfinkel’s suggestion in your footnote is a step in the right direction, but it doesn’t go far enough. If your assessment that the skill is easily assessed is accurate, then this is fertile ground for designing an actual filter. I’m imagining a well-defined scope (for example, a classic 1-2 page “essay test”) posing an under-defined DR question. There are plenty of EA-minded folk out there who would happily spend an afternoon thinking about, and writing up their response to, an interesting question for its own sake (cf. the entire internet), and even more who’d do it in exchange for receiving a “disentanglement grade”. Realistically, most submissions will be clear F/D scores within a paragraph or two (perhaps modulo initial beta testing and calibration of the writing prompt), and the responses requiring a more careful reading will be worth your while for precisely the reason that makes this exercise interesting in the first place.
TLDR: don’t define a problem and immediately discourage everyone from helping you solve it.
I wrote this in a google doc and copy-pasted, without intending the numbers to be links to anything. I’m not really sure why it made them highlight like a hyperlink.
I run an independent rationality group on Facebook, Evidence and Reasoning Enthusiasts. This is targeted toward people with at least some knowledge of rationality or science and halfway decent social skills. As such, I can help “build up this community and its capacity” and would like to know what specifically to do.
I agree “disentanglement research” is unclear. To me transdisciplinarity is easier to understand. My argument is transdisciplinary research is key to developing effective AI Policy and Strategy for Africa while disentanglement could be for the west.This primary because the transdisciplinarity approach is strongly woven into the fabric of the continent. However in Africa, solutions have to be broken down to ontology specificity and then the transdisciplinarity applied. I agree with klevanoff with his notion of wicked problem. I do not know how much sense it would make if we replaced the word disentanglement with transdiscplinarity.
To further support my argument on why AI needs to take a transdicplinarity research perspective in Africa. I notice that there existing institutions like politics,sciences and businesses that need to work together for development to occur [1]. The writer refers to them as object interfaces that need to be integrated together for a greater purpose “interrelatedness with in interests and values” [2]. He further argues that these objects must be weakly structured and easily malleable. In the context of Africa, some of the emerging technologies like AI have no existing policies and the policy implementing institutions are developing at a slower rate than the technology is. The technology on the other hand is developed to solve challenges in the western world and introduced to Africa for adoption. Such are examples of how loose the existing structures are.Secondly, the need for these structures to be malleable is quite evident because the technology advancements like AI that have strong impact on the general public must be regulated both in deployment and development. But how does one regulate what one does not understand.The risk in this approach is that one may enforce strict restrictions which may stifle the technology innovation. I think the complexity comes from integrating epistemology principles/
Quick question: Is your term “disentanglement research” similar to the discipline of “systems thinking” and what are the differences?
(Trying to get to grips with what you mean by “disentanglement research” )
(https://en.wikipedia.org/wiki/Systems_theory)
In Tetlock’s book Superforecasting, he distinguishes between two skills related to forecasting: generating questions, and answering them. This “disentanglement research” business sounds more like the first sort of work. Unfortunately, Tetlock’s book focuses on the second skill, but I do believe he talks some about the first skill (e.g. giving examples of people who are good at it).
I would imagine that for generating questions, curiosity and creativity are useful. Unfortunately, the Effective Altruism movement seems to be bad at creativity.
John Cleese gave this great talk about creativity in which he distinguishes between two mental modes, “open mode” and “closed mode”. Open mode is good for generating ideas, whereas closed mode is good for accomplishing well-defined tasks. It seems to me that for a lot of different reasons, the topic of AI strategy might put a person in closed mode:
Ethical obligation—Effective altruism is often framed as an ethical obligation. If I recall correctly, a Facebook poll indicated that around half of the EA community sees EA as more of an obligation than an opportunity. Obligations don’t typically create a feeling of playfulness.
Size of the problem—Paul Graham writes: “Big problems are terrifying. There’s an almost physical pain in facing them.” AI safety strategy is almost the biggest problem imaginable.
Big names—People like Nick Bostrom, Eliezer Yudkowsky, and Eric Drexler have a very high level of prestige within the EA community. (The status difference between them and your average EA is greater than what I’ve observed between the students & the professor in any college class I remember taking.) Eliezer in particular can get very grumpy with you if you disagree with him. I’ve noticed that I’m much more apt to generate ideas if I see myself as being at the top of the status hierarchy, and if there is no penalty for coming up with a “bad” idea (even a bad idea can be a good starting point). One idea for solving the EA community’s creativity problem is to encourage more EAs to develop Richard Feynman-level indifference to our local status norms.
Urgency—As you state in this post, every second counts! Unfortunately urgency typically has the effect of triggering closed mode.
Difficulty—As you state in this post, many brilliant people have tried & failed. For some people, this fact is likely to create a sense of intimidation which precludes creativity.
For curiosity, one useful exercise I’ve found is Anna Salamon’s practice of setting a 7-minute timer and trying to think of as many questions as possible within that period. The common pattern here seems to be “quantity over quality”. If you’re in a mental state where you feel a small amount of reinforcement for a bad idea, and a large amount of reinforcement for a good idea, don’t be surprised if a torrent of ideas follows (some of which are good).
Another practice I’ve found useful is keeping a notebook. Harnessing “ambient thought” and recording ideas as they come to me, in the appropriate notebook page, seems to be much more efficient on a per-minute basis than dedicated brainstorming.
If I was attacking this problem, my overall strategic approach would differ a little from what you are describing here.
I would place less emphasis on intellectual centralization and more emphasis on encouraging people to develop idiosyncratic perspectives/form their own ontologies. Rationale: if many separately developed idiosyncratic perspectives all predict that a particular action X is desirable, that is good evidence that we should do X. There’s an analogy to stock trading here. (Relatedly, the finance/venture capital industry might be the segment of society that has the most domain expertise related to predicting the future, modulo principle-agent problems that come with investing other peoples’ money. Please let me know if you can think of other candidates… perhaps the intelligence community?)
Discipline could be useful for reading books & passing classes which expand one’s library of concepts, but once you get to the original reasoning part, discipline gets less useful. Centralization could be useful for making sure that the space of ideas relevant to AI strategy gets thoroughly covered through our collective study, and for helping people find intellectual collaborators. But I would go for beers, whiteboards, and wikis with long lists of crowdsourced pros and cons, structured to maximize the probability that usefully related ideas will at one point or another be co-located in someone’s working memory, before any kind of standard curriculum. I suspect it’s better to see AI strategy as a fundamentally interdisciplinary endeavor. (It might be useful to look at successful interdisciplinary research groups such as the Santa Fe Institute for ideas.) And forget all that astronomical waste nonsense for a moment. We are in a simulation. We score 1 point if we get a positive singularity, 0 points otherwise. Where is the loophole in the game’s rules that the designers didn’t plan for?
[Disclaimer: I haven’t made a serious effort to survey the literature or systematically understand the recommendations of experts on either creativity or curiosity, and everything in this comment is just made up of bits and pieces I picked up here and there. If you agree with my hunch that creativity/curiosity are a core part of the problem, it might be worth doing a serious lit review/systematically reading authors who write about this stuff such as Thomas Kuhn, plus reading innovators in various fields who have written about their creative process.]
Another thought: Given the nature of this problem, I wonder why the focus is on enabling EAs to discover AI strategy vs trying to gather ideas from experts who are outside the community. Most college professors have office hours you can go to and ask questions. Existing experts aren’t suffering from any of the issues that might put EAs in closed mode, and they already have the deep expertise it would take years for us to accumulate. I could imagine an event like the Asilomar AI conference, but for AI safety strategy, where you invite leading experts in every field that seems relevant, do the beer and whiteboards thing, and see what people come up with. (A gathering size much smaller than the Asilomar conference might be optimal for idea generation. I think it’d be interesting to sponsor independent teams where each team consists of one deep learning expert, one AI venture capitalist, one game theory person, one policy person, one historian, one EA/rationalist, etc. and then see if the teams end up agreeing about anything.)
Are there any best practices for getting academics interested in problems?
I run a group for creatives on Facebook called Altruistic Ideas. In it, I have worked to foster a creative culture. I’ve also written about the differences between the EA and rationality cultures vs. the culture creatives need. If this might be useful for anyone’s EA goals, please feel free to message me.
I agree that creativity is key.
I’d would point out that you may need discipline to do experiments based upon your creative thoughts (if the information you need is not available). If you can’t check your original reasoning against the world, you are adrift in a sea of possibilities.
Yeah, that sounds about right. Research and idea generation are synergistic processes. I’m not completely sure what the best way to balance them is.
I strongly agree that independent thinking seems undervalued (in general and in EA/LW). There is also an analogy with ensembling in machine learning (https://en.wikipedia.org/wiki/Ensemble_learning).
By “independent” I mean “thinking about something without considering others’ thoughts on it” or something to that effect… it seems easy for people’s thoughts to converge too much if they aren’t allowed to develop in isolation.
Thinking about it now, though, I wonder if there isn’t some even better middle ground; in my experience, group brainstorming can be much more productive than independent thought as I’ve described it.
There is a very high-level analogy with evolution: I imagine sexual reproduction might create more diversity in a population than horizontal gene transfer, since in the latter case, an idea(=gene) which seems good could rapidly become universal, and thus “local optima” might be more of a problem for the population (I have no idea if that’s actually how this works biologically… in fact, it seems like it might not be, since at least some viruses/bacteria seem to do a great job of rapidly mutating to become resistant to defences/treatments.)
Carrick, this is an excellent post. I agree with most of the points that you make. I would, however, like to call attention to the wide consensus that exists in relation to acting prematurely.
As you observe, there are often path dependencies at play in AI strategy. Ill-conceived early actions can amplify the difficulty of taking corrective action at a later date. Under ideal circumstances, we would act under as close to certainty as possible. Achieving this ideal, however, is impractical for several interrelated reasons:
AI strategy is replete with wicked problems. The confidence that we can have in many (most?) of our policy recommendations must necessarily be relatively low. If the marginal costs of further research are high, then undertaking that research may not be worthwhile.
Delaying policy recommendations can sometimes be as harmful as or more harmful than making sub-par policy recommendations. There are several reasons for this. First, there are direct costs (e.g., lives lost prior to implementing sanitary standards). Second, delays allow other actors—most of whom are less concerned with rigor and welfare—to make relative gains in implementing their favored policies. If outcomes are path dependent, then inaction from AI strategists can lead to worse effects than missteps. Third, other actors are likely to gain influence if AI strategists delay. Opaque incentive structures and informal networks litter the path from ideation to policymaking. Even if there are not path dependencies baked into the policies themselves, there are sociopolitical path dependencies in the policymaking process. Gaining clout at an early stage tends to increase later influence. If AI strategists are unwilling to recommend policies, others will do so and reap the reputational gains entailed. Inversely, increased visibility may confer legitimacy to AI strategy as a discipline.
Policy communities in multiple countries are becoming more aware of AI, and policymaking activity is poised to increase. China’s national AI strategy, released several months ago, is a long-range plan, the implementation of which is being carried out by top officials. For the CCP, AI is not a marginal issue. Westerners will look to Chinese policies to inform their own decisions. In Washington, think tanks are increasingly recognizing the importance of AI. The Center for a New American Security, for example, now has a dedicated AI program (https://www.cnas.org/research/technology-and-national-security/artificial-intelligence) and is actively hiring. Other influential organizations are following suit. While DC policymakers paid little attention to AlphaGo, they definitely noticed Putin’s comments on AI’s strategic importance earlier this month. As someone with an inside vantage point, I can say with a high degree of confidence that AI will not remain neglected for long. Inaction on the part of AI strategists will not mean an absence of policy; it will mean the implementation of less considered policy.
As policy discussions in relation to AI become more commonplace and more ideologically motivated, EAs will likely have less ability to influence outcomes, ceteris paribus (hence Carrick’s call for individuals to build career capital). Even if we are uncertain about specific recommendations—uncertainty that may be intractable—we will need to claim a seat at the table or risk being sidelined.
There are also many advantages to starting early. To offer a few:
If AI strategists are early movers, they can wield disproportionate influence in framing the discourse. Since anchoring effects can be large, introducing policymakers to AI through the lens of safety rather than, say, national military advantage is probably quite positive in expectation.
Making policy recommendations can be useful in outsourcing cognitive labor. Once an idea becomes public, others can begin working on it. Research rarely becomes policy overnight. In the interim period, proponents and critics alike can refine thinking and increase the analytical power brought to bear on a topic. This enables greater scrutiny for longer-range thought that has no realistic path to near-term implementation, and may result in fewer unidentified considerations.
Taking reversible harmful actions at an early stage allows us to learn from our mistakes. If these mistakes are difficult to avoid ex ante, and we wait until later to make them, the consequences are likely to be more severe. Of course, we may not know which actions are reversible. This indicates to me that researching path dependence in policymaking would be valuable.
This is not a call for immediate action, and it is not to suggest that we should be irresponsible in making recommendations. I do, however, think that we should increasingly question the consensus around inaction and begin to consider more seriously how much uncertainty we are willing to accept, as well as when and how to take a more proactive approach to implementation.
I think it is important to note that in the political world there is the vision of two phases of AI development, narrow AI and general AI.
Narrow AI is happening now. The 30+% job loss predictions in the next 20 years, all narrow AI. This is what people in the political sphere are preparing for, from my exposure to it.
General AI is conveniently predicted more that 20 years away, so people aren’t thinking about it because they don’t know what it will look like and they have problems today to deal with.
Getting this policy response right to narrow AI does have a large impact. Large scale unemployment could destabilize countries, causing economic woes and potentially war.
So perhaps people interested in general AI policy should get involved with narrow AI policy, but make it clear that this is the first battle in a war, not the whole thing. This would place them well and they could build up reputations etc. They could be be in contact with the disentanglers so that when the general AI picture is clearer, they can make policy recommendations.
I’d love it if the narrow-general AI split was reflected in all types of AI work.
@Carrickflynn,
It’s been 3 years now. Is it possible to do a retake evaluating the current situation of the Disentanglement and if there has been growth in possibilities to work in AI strategy & policy (be in implementation or research)?
Thanks.
Hi Carrick,
Thanks for your thoughts on this. I found this really helpful and I think 80′000 hours could maybe consider linking to it on the AI policy guide.
Disentanglement research feels like a valid concept, and it’s great to see it exposed here. But given how much weight pivots on the idea and how much uncertainty surrounds identifying these skills, it seems like disentanglement research is a subject that is itself asking for further disentanglement! Perhaps it could be a trial question for any prospective disentanglers out there.
You’ve given examples of some entangled and under-defined questions in AI policy and provided the example of Bostrom as exhibiting strong disentanglement skills; Ben has detailed an example of an AI strategy question that seems to require some sort of “detangling” skill; Jade has given an illuminative abstract picture. These are each very helpful. But so far, the examples are either exclusively AI strategy related or entirely abstract. The process of identifying the general attributes of good disentanglers and disentanglement research might be assisted by providing a broader range of examples to include instances of disentanglement research outside of the field of AI strategy. Both answered and unanswered research questions of this sort might be useful. (I admit to being unable to think of any good examples right now)
Moving away from disentanglement, I’ve been interested for some time by your fourth, tentative suggestion for existing policy-type recommendations to
This is a subject that I haven’t been able to find much written material on—if you’re aware of any I’d be very interested to know about it. It isn’t completely clear whether or how to push for an idea like this. Additionally, based on the lack of literature, it feels like this hasn’t received as much thought as it should, even in an exploratory sense (but being outside of a strategy research cluster, I could be wrong on this). You mention that race dynamics are easier to start than stop, meanwhile early intergovernmental initiatives are one of the few tools that can plausibly prevent/slow/stop international races of this sort. These lead me to believe that this ‘recommendation’ is actually more of a high priority research area. Exploring this area appears robustly positive in expectation. I’d be interested to hear other perspectives on this subject and to know whether any groups or individuals are currently working/thinking about it, and if not, how research on it might best be started, if indeed it should be.
For five years, my favorite subject to read about was talent. Unlike developmental psychologists, I did not spend most of my learning time on learning disabilities. I also did a lot of intuition calibration which helps me detect various neurological differences in people. Thus, I have a rare area of knowledge and an unusual skill which may be useful for assisting with figuring out what types of people have a particular kind of potential, what they’re like, what’s correlated with their talent(s), what they might need, and how to find and identify them. If any fellow EAs can put this to use, feel free to message me.
I would broadly agree. I think this is an important post and I agree with most of the ways to prepare. I think we are not there yet for large scale AI policy/strategy.
There are few things that I would highlight as additions. 1) We need to cultivate the skills of disentanglement. Different people might be differently suited, but like all skills it is one that works better with practice and people to practice with. Lesswrong is trying to place itself as that kind of place. It is having a little resurgence with the new website www.lesserwrong.com. For example there has been lots of interesting discussion on the problems of Goodheart’s law, which will be necessary to at least somewhat solve if we are to get AISafety groups that actually do AISafety research and don’t just optimise some research output metric to get funding.
I am not sure if lesswrong is the correct place, but we do need places for disentanglers to grow.
2) I would also like to highlight the fact that we don’t understand intelligence and that there have been lots of people studying it for a long time (psychologists etc) that I don’t think we do enough to bring into discussing artificial versions of the thing they have studied. Lots of work on policy side of AI safety models it as utility maximimising agent in the economic style. I am pretty skeptical that is a good model of humans or of the AIs we will create. Figuring out what better models might be, is on the top of my personal priority list.
Edited to add 3) It seems like a sensible policy is to fund a competition in the style of at super forecasting aimed at AI and related technologies. This should give you some idea of the accuracy of peoples view on technology development/forecasting.
I would caution that we are also in the space of wicked problems so it may be there is never a complete certainty of the way we should move.
Great article, thanks Carrick!
If you’re an EA who wants to work on AI policy/strategy (including in support roles), you should absolutely get in touch with 80,000 Hours about coaching. Often, we’ve been able to help people interested in the area clarify how they can contribute, made introductions etc.
Apply for coaching here.
All of the endnote links are broken.
We need disentanglement research examples. I tried using Google to search intelligence.org and www.fhi.ox.ac.uk for the term “disentanglement” and received zero results for both. What I need to determine whether I should pursue this path is three examples of good disentanglement research. Before reading the study or book for the examples, I will need a very quick gist—a sentence or three that summarizes what each example is about. An oversimplification is okay as long as this is mentioned and we’re given a link to a paper or something so we can understand it correctly if we choose to look into it further. Additionally, I need to be shown a list of open questions.
If I am the only person who asked for this, then your article has not been very effective at getting new people to try out disentanglement research. The obstacle of not even knowing what, specifically, disentanglement research looks like would very effectively prevent a new person from getting into it. I think it would be a really good idea to write a follow-up article that contains the three examples of disentanglement research, the quick gists of what’s contained in each example, and the list of open questions. That information has a chance to get someone new involved.
Thanks for writing this. My TL;DR is:
AI policy is important, but we don’t really know where to begin at the object level
You can potentially do 1 of 3 things, ATM: A. “disentanglement” research: B. operational support for (e.g.) FHI C. get in position to influence policy, and wait for policy objectives to be cleared up
Get in touch / Apply to FHI!
I think this is broadly correct, but have a lot of questions and quibbles.
I found “disentanglement” unclear. [14] gave the clearest idea of what this might look like. A simple toy example would help a lot.
Can you give some idea of what an operations role looks like? I find it difficult to visualize, and I think uncertainty makes it less appealling.
Do you have any thoughts on why operations roles aren’t being filled?
One more policy that seems worth starting on: programs that build international connections between researchers (especially around policy-relevant issues of AI (i.e. ethics/safety)).
The timelines for effective interventions in some policy areas may be short (e.g. 1-5 years), and it may not be possible to wait for disentanglement to be “finished”.
Is it reasonable to expect the “disentanglement bottleneck” to be cleared at all? Would disentanglement actually make policy goals clear enough? Trying to anticipate all the potential pitfalls of policies is a bit like trying to anticipate all the potential pitfalls of a particular AI design or reward specification… fortunately, there is a bit of a disanalogy in that we are more likely to have a chance to correct mistakes with policy (although that still could be very hard/impossible). It seems plausible that “start iterating and create feedback loops” is a better alternative to the “wait until things are clearer” strategy.
That’s the TLDR that I took away from the article too.
I agree that “disentanglement” is unclear. The skillset that I previously thought was needed for this was something like IQ + practical groundedness + general knowledge + conceptual clarity, and that feels mostly to be confirmed by the present article.
I have some lingering doubts here as well. I would flesh out an objection to the ‘disentanglement’-focus as follows: AI strategy depends critically on government, some academic communities and some companies, that are complex organizations. (Suppose that) complex organizations are best understood by an empirical/bottom-up approach, rather than by top-down theorizing. Consider the medical establishment that I have experience with. If I got ten smart effective altruists to generate mutually exclusive collectively exhaustive (MECE) hypotheses about it, as the article proposes doing for AI strategy, they would, roughly speaking, hallucinate some nonsense, that could be invalidated in minutes by someone with years of experience in the domain. So if AI strategy depends in critical components on the nature of complex institutions, then what we need for this research may be, rather than conceptual disentanglement, something more like high-level operational experience of these domains. Since it’s hard to find such people, we may want to spend the intervening time interacting with these institutions or working within them on less important issues. Compared to this article, this perspective would de-emphasize the importance of disentanglement, while maintaining the emphasis on entering these institutions, and increasing the emphasis on interacting with and making connections within these institutions.
Carrick,
Given your background, I will take as given your suggestion that disentanglement research is both very important and a very rare skill. With that said, I feel like there’s a reasonable meta-solution here, one that’s at least worth investigating. Since you’ve identified at least one good disentaglement researcher (eg, Nick Bolstrom), have you considered asking them to design a test to assess possible researchers?
The suggestion may sound a bit silly, so I’ll elaborate. I read your article and found it compelling. I may or may not be a good disentanglement researcher, but per your article, I probably am not. So, your article has simultaneously raised my awareness of the issue and dissuaded me from possibly helping with it. The initial pessimism, followed by your suggestion to “Read around in the area, find something sticky you think you might be able to disentangle, and take a run at it”, all but guarantees low followthrough from your audience.
Ben Garfinkel’s suggestion in your footnote is a step in the right direction, but it doesn’t go far enough. If your assessment that the skill is easily assessed is accurate, then this is fertile ground for designing an actual filter. I’m imagining a well-defined scope (for example, a classic 1-2 page “essay test”) posing an under-defined DR question. There are plenty of EA-minded folk out there who would happily spend an afternoon thinking about, and writing up their response to, an interesting question for its own sake (cf. the entire internet), and even more who’d do it in exchange for receiving a “disentanglement grade”. Realistically, most submissions will be clear F/D scores within a paragraph or two (perhaps modulo initial beta testing and calibration of the writing prompt), and the responses requiring a more careful reading will be worth your while for precisely the reason that makes this exercise interesting in the first place.
TLDR: don’t define a problem and immediately discourage everyone from helping you solve it.
Your link [2] points to a .docx file in a folder on a computer. It isn’t a usable download link. Was that the purpose?
I wrote this in a google doc and copy-pasted, without intending the numbers to be links to anything. I’m not really sure why it made them highlight like a hyperlink.
I run an independent rationality group on Facebook, Evidence and Reasoning Enthusiasts. This is targeted toward people with at least some knowledge of rationality or science and halfway decent social skills. As such, I can help “build up this community and its capacity” and would like to know what specifically to do.
Can you elaborate more on this?
I agree “disentanglement research” is unclear. To me transdisciplinarity is easier to understand. My argument is transdisciplinary research is key to developing effective AI Policy and Strategy for Africa while disentanglement could be for the west.This primary because the transdisciplinarity approach is strongly woven into the fabric of the continent. However in Africa, solutions have to be broken down to ontology specificity and then the transdisciplinarity applied. I agree with klevanoff with his notion of wicked problem. I do not know how much sense it would make if we replaced the word disentanglement with transdiscplinarity.
I think an important thing for Ai strategy is to figure out ishow to fund empirical studies into questions that impinge on crucial considerations.
For example funding studies into the nature of IQ. I’ll post an article on that later but wanted to flag it here as well.
To further support my argument on why AI needs to take a transdicplinarity research perspective in Africa. I notice that there existing institutions like politics,sciences and businesses that need to work together for development to occur [1]. The writer refers to them as object interfaces that need to be integrated together for a greater purpose “interrelatedness with in interests and values” [2]. He further argues that these objects must be weakly structured and easily malleable. In the context of Africa, some of the emerging technologies like AI have no existing policies and the policy implementing institutions are developing at a slower rate than the technology is. The technology on the other hand is developed to solve challenges in the western world and introduced to Africa for adoption. Such are examples of how loose the existing structures are.Secondly, the need for these structures to be malleable is quite evident because the technology advancements like AI that have strong impact on the general public must be regulated both in deployment and development. But how does one regulate what one does not understand.The risk in this approach is that one may enforce strict restrictions which may stifle the technology innovation. I think the complexity comes from integrating epistemology principles/
[1]https://i2insights.org/2017/05/09/transdisciplinary-integration-methods/
[2]https://complexitycontrol.org/methods-of-transdisciplinary-research/
Quick question: Is your term “disentanglement research” similar to the discipline of “systems thinking” and what are the differences? (Trying to get to grips with what you mean by “disentanglement research” ) (https://en.wikipedia.org/wiki/Systems_theory)
In fact a more general version of the above question is:
What are the existing research / consultancy / etc disciplines that are most similar to the kind of work you are looking for?
If you can identify that it could help people in local communities direct people to this kind of work.
The closest seems to be really well done analytic philosophy. I recommend Nick Bostrom’s work as a great example of this.
I also think that is seems similar to what mathematicians, theoretical physicists, and model builders frequently do.
Other good examples would probably be Thomas Schelling in IR and nuclear security. Coase in economics. Maybe Feynman?