In this 2017 talk, IARPA director Jason Matheny talks about how effective altruists can have an impact through entering government. He draws widely from his own experience in the US government.
The below transcript is lightly edited for readability.
The Talk
I don’t have any slides just because they’re much harder to get through government review, which will be a big part of my talk today on what to do and not to do in government. It’s great to see so many young people who are thinking about career options. I feel like I’m close to dying now, so there’s only so much good remaining I can do, unless I’m cryonically preserved and resuscitated later. You all have so many decades left to do an enormous amount of good with your lives, so thank you for spending at least 50 minutes of that life to think about options in government.
I’m going to describe a few different paths to doing good within government that don’t rely on spending an entire career in government. It might be a two-year stint, or a five-year stint, but still allowing you to accomplish a huge amount of good.
I’m going to talk about three things. The first is how I came into government. The second is the roles for EAs working within government. The third is some practical advice on picking jobs within government, whether short-term or longer-term.
My own route was sort of circuitous in that I started working in college, planning to become an architect. Shortly after I graduated, I found an orphan copy of the 1993 World Bank World Development Report, which both dates me and dates a lot of my perspective on problems of altruism. The report was really focused on how you can do cost-effectiveness analysis on health and development. For me, this was the first time I was ever exposed to an argument about how cheaply you can save lives and significantly reduce suffering. This was one of the first reports that looked at the cost per DALY averted for a range of different health interventions. I thought this application of cost-effectiveness analysis to health and development was pretty eye-opening. For one, it showed that you could save millions of live for less than about $1,000 per life. But it also showed that there were tremendous differences between the most and the least cost-effective interventions, so that our decisions about what to invest in have a huge impact.
I ended up deciding not to become an architect, or else I’d be giving a talk on effective altruism in architecture. I decided to instead go into global health, and worked for several years on global infectious diseases—especially malaria, tuberculosis, and HIV in South Asia, but also in East Asia. It was in 2002 that I made another career shift. This was the same year that the first virus was synthesized from scratch, de novo. It turned biology into something more closely approximating computer science, or an engineering discipline. You could now take the chemical constituents of DNA and create a new genome.
At that time, the technology was fairly primitive. The longest virus that you could assemble was polio. That wasn’t too much to worry about. We already had vaccines for polio. We knew how to control polio. But I and most of the people who I had worked with had worried that somebody would apply this technology to recreate smallpox—to recreate the 1918 influenza that killed over 50 million people in one year. Or, they could make something much worse than any naturally occurring virus, since there are limits to nature’s ingenuity that might be outpaced by human ingenuity.
So I moved from working on naturally occurring global infectious diseases to working on defences against engineered threats. That work then led me to places like the Future of Humanity Institute, thinking about how we wrestle with risks from emerging technologies. I thought about ways that I could have an impact on this area. One way seemed to be doing research myself, but I didn’t think I was especially smart in doing that research. It seemed like there were people who were smarter than me, like Nick Bostrom, who could be doing that research. I thought: how could I deliver more funding to people who were smarter than me? That’s why I joined government.
I came to IARPA to put a multiplier effect on my effort and the effort of other researchers. My goal was to set up a budget in which I could fund important research in a range of areas including risk assessment, technical forecasting, work on biosecurity, nuclear security, cyber security, and assessments of future risks from things like autonomous weapons or AI accidents. The work that I’ve done at IARPA has convinced me that there’s a lot of low-hanging fruit within government positions that we should be picking as effective altruists. There are many different roles that effective altruists can have within government organizations.
I’m going to tell you about a few of them. I’ll limit it to the three areas that I have some background in, which are global health, animal welfare and catastrophic risks. But I think that government is in general a place that has leverage over a variety of different pressing societal issues. Thus if I’m leaving out some topics that’s not because I think government doesn’t have influence on those topics. It’s just because I’m ignorant.
In global health, government funding has an enormous impact on the development of new vaccines, antivirals, and antibiotics, as well as other therapies. Much of that work is conducted at the National Institute for Allergies and Infectious Diseases, that funds basic and applied research in infectious diseases, as well as the development of new therapies. The FDA has an important role determining which kinds of drugs and vaccines are ultimately introduced. Despite being a regulatory agency, they make really interesting innovations, incentive systems, and reward systems, including things like priority review vouchers that can accelerate the introduction of new vaccines for important diseases. And there are places like the Fogarty Center at the National Institutes of Health, which does things like economic analysis of international health interventions. Then there are the more operational arms of the US government, like the Centers for Disease Control and the US Agency for International Development, that have direct impacts on health and development overseas.
In animal welfare, the USDA has an important role to play in establishing policies that govern some treatments of animals. They also have a small research budget, some of which is used—and a larger amount of which could be used—to develop better, healthier alternatives. State legislatures also have a big role to play in animal welfare as they can pass laws surrounding animal handling and slaughter.
Then on catastrophic risks where I’ve spent most of my time, governments have a very significant role. A lot of those risks are influenced by government decisions, both positively and negatively. Multiple organizations work on preventing nuclear war, biological warfare, or accidents, as well as cyber warfare, and the misuse of various emerging technologies. I’ll go through a few of the most important of those organizations. First you have the National Security Council, a part of the White House that informs national security decision making, including decisions about war. Another is the White House Office of Science and Technology Policy, which looks at emerging technologies and risks associated with them. That office has groups that examine a range of important technologies, including AI, bio-technology, and neuroscience among other topics.
Within the Department of Defense, there is the Office of Net Assessment, which in my view is one of the most unusual organizations within government, as well as one of the most important. It looks at long-range security issues that could be decades in the making. For example, what changes in future weapon systems are likely to disrupt deterrents? What would be the consequences of strategic miscalculation with nuclear weapons?
The Defense Threat Reduction Agency within the Department of Defense is the lead agency responsible for countering chemical, biological, radiological, and nuclear weapons. The Federal Emergency Management Agency is responsible for considering worst-case scenarios that could affect the US and developing mitigations against them. Then there are many others that deal with very specific threats such as the Defense Department’s Strategic Command, which is responsible for the United States’s nuclear weapons and their safety. BARDA, which is a part of Health and Human Services, is responsible for developing medical countermeasures against bioterrorism. The intelligence agencies like CIA and DIA, assess how advanced a particular group’s biological weapons program is, or their ability to access disruptive technologies, or the likelihood of industrial accidents, for example in foreign biology labs.
Across all three of the EA topics that I mentioned, in global health, in animal welfare, and in catastrophic risks, one cost-effective route to having an impact is to affect the funding of new technologies that could in some ways obviate the need for certain kinds of harmful technologies or reduce the risks of technologies by making sure they’re sufficiently protected through safety engineering. For funding scientific and technological research, there are a few important organizations within government. There’s the White House Office of Management and Budget, which helps to set the White House budget requests. Often we’ll find even fairly junior folks who are putting their weight on multi-billion dollar decisions. It really is extraordinary that even fairly junior positions can have incredible influence. If you think of this just in terms of an expected value calculation, even a 10% probability of affecting a $10 billion decision means a billion dollars in expectation, and can be hugely consequential on topics such as nuclear safety, biological safety, future of autonomous weapons and so forth.
There are the Congressional Appropriations Committees (link to senate page) that approve the budgets from the White House. Here, too, you find even fairly junior staffers that have an incredible impact. Then there are the organizations that take the budgets that they’ve been given and freely decide how to allocate them. Those include places like the National Institutes of Health, the National Science Foundation, as well as the ARPAs—the intelligence IARPA where I work, DARPA, HSARPA and ARPA-E. At those organizations the program managers, who are typically in their 30s say, have come out of graduate school programs and a science or engineering discipline. They’ve spent a few years working in a lab, sometimes within academia or within industry. Then they spend a term-limited time in the government, usually not exceeding five years. They’re given an extraordinary amount of latitude. They’re given a budget of several tens of millions of dollars with the expectation, the trust, that they will invest that money as cost-effectively as possible in solving a particular technical problem. For IARPA, those problems are often associated with reducing the risks of emerging technologies.
As one example, we have a brilliant biologist at IARPA, John Julias, who runs a program called Fun GCAT, which is focused on developing new systems for screening the sequences that go into DNA synthesizers. Can you determine whether this is a safe sequence or a dangerous sequence? That’s the kind of work that we really need program managers to do, and we’ve entrusted John with a +$50 million budget which he uses to fund work here at Harvard, at MIT, and at many other universities and companies in order to advance this goal of reducing risks from synthetic biology. It’s much more money I think than at least I could’ve expected to earn in my lifetime, but we give it, we entrust it, to a program manager to spend as wisely as possible—with the rigor of spending a quarter of that money on testing and evaluation to figure out whether the investments that we’re making are actually making a difference, and whether we can accurately assess the risks from, say, a novel sequence. Those are program managers.
Agency directors can further direct hundreds of millions or even billions of dollars to key projects. Again, even if those are only, say, 10% as effective as funding that would be given outside of a bureaucracy, the expected value of those investments is quite large and can have a dramatic impact. Within government, there are also other levers that one can pull. At IARPA, we’ve not only been able to erect a large budget on reducing catastrophic risks, but we’ve also been able to engage in policy discussions. We’ve led groups within the White House on the long-term impacts of AI and of biotechnology. We co-led the White House AI R&D strategy, and we’ve advised the National Security Council on other emerging technologies. There are some decisions that are made only by governments, and some of those decisions are highly consequential. They include decisions like going to war, or what weapon systems will be fielded, or how technologies will be embedded within larger critical systems. It makes sense to engage more effective altruists within these positions where they can influence those decisions.
One can also have influence on the outside, working as a contractor within a government agency. Most of the people who work at IARPA are contractors rather than government employees. The amount of expertise that we have to draw on is too vast to hire them all ourselves directly, especially with short-term positions. So we hire computer scientists, and biologists, and chemists, and physicists, and neuroscientists, and sociologists, political scientists, and cognitive psychologists, because we need them all. We also need lawyers, and we even need philosophers. We have a program on applied philosophy called CREATE, which is a program to develop new systems for argumentation and informal reasoning that can lead intelligence analysts to make better judgements.
So we need lots of help. We need them from lots of places including contractors, but also think-tanks. There are a range of think-tanks that inform the policy-making process that sometimes have a quite deep influence on administration. For instance, the Heritage Foundation has a substantial influence on the current administration, while past administrations have been influenced by other think tanks such as Brookings, the Center for a New American Security, the Center for Strategic and International Studies, and Harvard’s own Belfer Center. There are many others that help shape the decision-making of government leaders. Hence, that’s another way you can have an influence on government.
I’m going to close in my final few minutes just by providing some general advice if you’re interested in pursuing a job, whether short-term or long-term, within government. I would recommend really thinking about opportunities to move across and among the different sectors of society, government, industry, academia, and NGOs, because there’s a need for horizontal transfer of knowledge and best practices. If all of our folks within government have come from government straight out of school, that will prevent us from being able to adopt best practices from industry or from academia. So there really is a need I think for continuous cycling throughout a career, bouncing around between the different sectors in order to bring knowledge across them.
My first suggestion is to reach out to 80,000 Hours, which I think has been pulling together some advice about government jobs (see below). I think one of the pieces of advice is at least to consider it as an option, because we are nowhere near the saturation point of effective altruists going into government positions. There are fairly junior positions across government that have a high potential impact that we have trouble recruiting for.
My second suggestion is to get to know the people who work within the organizations where you’d like to work. You can learn a lot about those organizations, their structure, and their staff just from online websites as well as Wikipedia. You can find the biographies of some of the people whose careers you might want to mimic. One strategy is just to reverse engineer their biography. Figure out what the steps are that seem critical in getting to the positions that you would like to have in the future. On that point, many of these people are lawyers, but just as many of them are scientists and engineers, and we do need more philosophers in government as well. But I think you’ll find the diversity of talent that we need is ever growing. There’s a particular intersection between policy and technology that is extremely difficult to recruit for. So for people who are still picking their major or their concentration for a thesis, if you look at the science policy of blank, pretty much any of the topics that are critical on our list have not been saturated with attention. There’s still lots of low-hanging fruit to pick.
My last suggestion is to reach out to me, especially if you’re interested in pursuing a job, short-term or long-term, in national security or reducing global catastrophic risks. Mostly because I really need the help. That’s it.
Jason Matheny: Effective altruism in government
Link post
In this 2017 talk, IARPA director Jason Matheny talks about how effective altruists can have an impact through entering government. He draws widely from his own experience in the US government.
The below transcript is lightly edited for readability.
The Talk
I don’t have any slides just because they’re much harder to get through government review, which will be a big part of my talk today on what to do and not to do in government. It’s great to see so many young people who are thinking about career options. I feel like I’m close to dying now, so there’s only so much good remaining I can do, unless I’m cryonically preserved and resuscitated later. You all have so many decades left to do an enormous amount of good with your lives, so thank you for spending at least 50 minutes of that life to think about options in government.
I’m going to describe a few different paths to doing good within government that don’t rely on spending an entire career in government. It might be a two-year stint, or a five-year stint, but still allowing you to accomplish a huge amount of good.
I’m going to talk about three things. The first is how I came into government. The second is the roles for EAs working within government. The third is some practical advice on picking jobs within government, whether short-term or longer-term.
My own route was sort of circuitous in that I started working in college, planning to become an architect. Shortly after I graduated, I found an orphan copy of the 1993 World Bank World Development Report, which both dates me and dates a lot of my perspective on problems of altruism. The report was really focused on how you can do cost-effectiveness analysis on health and development. For me, this was the first time I was ever exposed to an argument about how cheaply you can save lives and significantly reduce suffering. This was one of the first reports that looked at the cost per DALY averted for a range of different health interventions. I thought this application of cost-effectiveness analysis to health and development was pretty eye-opening. For one, it showed that you could save millions of live for less than about $1,000 per life. But it also showed that there were tremendous differences between the most and the least cost-effective interventions, so that our decisions about what to invest in have a huge impact.
I ended up deciding not to become an architect, or else I’d be giving a talk on effective altruism in architecture. I decided to instead go into global health, and worked for several years on global infectious diseases—especially malaria, tuberculosis, and HIV in South Asia, but also in East Asia. It was in 2002 that I made another career shift. This was the same year that the first virus was synthesized from scratch, de novo. It turned biology into something more closely approximating computer science, or an engineering discipline. You could now take the chemical constituents of DNA and create a new genome.
At that time, the technology was fairly primitive. The longest virus that you could assemble was polio. That wasn’t too much to worry about. We already had vaccines for polio. We knew how to control polio. But I and most of the people who I had worked with had worried that somebody would apply this technology to recreate smallpox—to recreate the 1918 influenza that killed over 50 million people in one year. Or, they could make something much worse than any naturally occurring virus, since there are limits to nature’s ingenuity that might be outpaced by human ingenuity.
So I moved from working on naturally occurring global infectious diseases to working on defences against engineered threats. That work then led me to places like the Future of Humanity Institute, thinking about how we wrestle with risks from emerging technologies. I thought about ways that I could have an impact on this area. One way seemed to be doing research myself, but I didn’t think I was especially smart in doing that research. It seemed like there were people who were smarter than me, like Nick Bostrom, who could be doing that research. I thought: how could I deliver more funding to people who were smarter than me? That’s why I joined government.
I came to IARPA to put a multiplier effect on my effort and the effort of other researchers. My goal was to set up a budget in which I could fund important research in a range of areas including risk assessment, technical forecasting, work on biosecurity, nuclear security, cyber security, and assessments of future risks from things like autonomous weapons or AI accidents. The work that I’ve done at IARPA has convinced me that there’s a lot of low-hanging fruit within government positions that we should be picking as effective altruists. There are many different roles that effective altruists can have within government organizations.
I’m going to tell you about a few of them. I’ll limit it to the three areas that I have some background in, which are global health, animal welfare and catastrophic risks. But I think that government is in general a place that has leverage over a variety of different pressing societal issues. Thus if I’m leaving out some topics that’s not because I think government doesn’t have influence on those topics. It’s just because I’m ignorant.
In global health, government funding has an enormous impact on the development of new vaccines, antivirals, and antibiotics, as well as other therapies. Much of that work is conducted at the National Institute for Allergies and Infectious Diseases, that funds basic and applied research in infectious diseases, as well as the development of new therapies. The FDA has an important role determining which kinds of drugs and vaccines are ultimately introduced. Despite being a regulatory agency, they make really interesting innovations, incentive systems, and reward systems, including things like priority review vouchers that can accelerate the introduction of new vaccines for important diseases. And there are places like the Fogarty Center at the National Institutes of Health, which does things like economic analysis of international health interventions. Then there are the more operational arms of the US government, like the Centers for Disease Control and the US Agency for International Development, that have direct impacts on health and development overseas.
In animal welfare, the USDA has an important role to play in establishing policies that govern some treatments of animals. They also have a small research budget, some of which is used—and a larger amount of which could be used—to develop better, healthier alternatives. State legislatures also have a big role to play in animal welfare as they can pass laws surrounding animal handling and slaughter.
Then on catastrophic risks where I’ve spent most of my time, governments have a very significant role. A lot of those risks are influenced by government decisions, both positively and negatively. Multiple organizations work on preventing nuclear war, biological warfare, or accidents, as well as cyber warfare, and the misuse of various emerging technologies. I’ll go through a few of the most important of those organizations. First you have the National Security Council, a part of the White House that informs national security decision making, including decisions about war. Another is the White House Office of Science and Technology Policy, which looks at emerging technologies and risks associated with them. That office has groups that examine a range of important technologies, including AI, bio-technology, and neuroscience among other topics.
Within the Department of Defense, there is the Office of Net Assessment, which in my view is one of the most unusual organizations within government, as well as one of the most important. It looks at long-range security issues that could be decades in the making. For example, what changes in future weapon systems are likely to disrupt deterrents? What would be the consequences of strategic miscalculation with nuclear weapons?
The Defense Threat Reduction Agency within the Department of Defense is the lead agency responsible for countering chemical, biological, radiological, and nuclear weapons. The Federal Emergency Management Agency is responsible for considering worst-case scenarios that could affect the US and developing mitigations against them. Then there are many others that deal with very specific threats such as the Defense Department’s Strategic Command, which is responsible for the United States’s nuclear weapons and their safety. BARDA, which is a part of Health and Human Services, is responsible for developing medical countermeasures against bioterrorism. The intelligence agencies like CIA and DIA, assess how advanced a particular group’s biological weapons program is, or their ability to access disruptive technologies, or the likelihood of industrial accidents, for example in foreign biology labs.
Across all three of the EA topics that I mentioned, in global health, in animal welfare, and in catastrophic risks, one cost-effective route to having an impact is to affect the funding of new technologies that could in some ways obviate the need for certain kinds of harmful technologies or reduce the risks of technologies by making sure they’re sufficiently protected through safety engineering. For funding scientific and technological research, there are a few important organizations within government. There’s the White House Office of Management and Budget, which helps to set the White House budget requests. Often we’ll find even fairly junior folks who are putting their weight on multi-billion dollar decisions. It really is extraordinary that even fairly junior positions can have incredible influence. If you think of this just in terms of an expected value calculation, even a 10% probability of affecting a $10 billion decision means a billion dollars in expectation, and can be hugely consequential on topics such as nuclear safety, biological safety, future of autonomous weapons and so forth.
There are the Congressional Appropriations Committees (link to senate page) that approve the budgets from the White House. Here, too, you find even fairly junior staffers that have an incredible impact. Then there are the organizations that take the budgets that they’ve been given and freely decide how to allocate them. Those include places like the National Institutes of Health, the National Science Foundation, as well as the ARPAs—the intelligence IARPA where I work, DARPA, HSARPA and ARPA-E. At those organizations the program managers, who are typically in their 30s say, have come out of graduate school programs and a science or engineering discipline. They’ve spent a few years working in a lab, sometimes within academia or within industry. Then they spend a term-limited time in the government, usually not exceeding five years. They’re given an extraordinary amount of latitude. They’re given a budget of several tens of millions of dollars with the expectation, the trust, that they will invest that money as cost-effectively as possible in solving a particular technical problem. For IARPA, those problems are often associated with reducing the risks of emerging technologies.
As one example, we have a brilliant biologist at IARPA, John Julias, who runs a program called Fun GCAT, which is focused on developing new systems for screening the sequences that go into DNA synthesizers. Can you determine whether this is a safe sequence or a dangerous sequence? That’s the kind of work that we really need program managers to do, and we’ve entrusted John with a +$50 million budget which he uses to fund work here at Harvard, at MIT, and at many other universities and companies in order to advance this goal of reducing risks from synthetic biology. It’s much more money I think than at least I could’ve expected to earn in my lifetime, but we give it, we entrust it, to a program manager to spend as wisely as possible—with the rigor of spending a quarter of that money on testing and evaluation to figure out whether the investments that we’re making are actually making a difference, and whether we can accurately assess the risks from, say, a novel sequence. Those are program managers.
Agency directors can further direct hundreds of millions or even billions of dollars to key projects. Again, even if those are only, say, 10% as effective as funding that would be given outside of a bureaucracy, the expected value of those investments is quite large and can have a dramatic impact. Within government, there are also other levers that one can pull. At IARPA, we’ve not only been able to erect a large budget on reducing catastrophic risks, but we’ve also been able to engage in policy discussions. We’ve led groups within the White House on the long-term impacts of AI and of biotechnology. We co-led the White House AI R&D strategy, and we’ve advised the National Security Council on other emerging technologies. There are some decisions that are made only by governments, and some of those decisions are highly consequential. They include decisions like going to war, or what weapon systems will be fielded, or how technologies will be embedded within larger critical systems. It makes sense to engage more effective altruists within these positions where they can influence those decisions.
One can also have influence on the outside, working as a contractor within a government agency. Most of the people who work at IARPA are contractors rather than government employees. The amount of expertise that we have to draw on is too vast to hire them all ourselves directly, especially with short-term positions. So we hire computer scientists, and biologists, and chemists, and physicists, and neuroscientists, and sociologists, political scientists, and cognitive psychologists, because we need them all. We also need lawyers, and we even need philosophers. We have a program on applied philosophy called CREATE, which is a program to develop new systems for argumentation and informal reasoning that can lead intelligence analysts to make better judgements.
So we need lots of help. We need them from lots of places including contractors, but also think-tanks. There are a range of think-tanks that inform the policy-making process that sometimes have a quite deep influence on administration. For instance, the Heritage Foundation has a substantial influence on the current administration, while past administrations have been influenced by other think tanks such as Brookings, the Center for a New American Security, the Center for Strategic and International Studies, and Harvard’s own Belfer Center. There are many others that help shape the decision-making of government leaders. Hence, that’s another way you can have an influence on government.
I’m going to close in my final few minutes just by providing some general advice if you’re interested in pursuing a job, whether short-term or long-term, within government. I would recommend really thinking about opportunities to move across and among the different sectors of society, government, industry, academia, and NGOs, because there’s a need for horizontal transfer of knowledge and best practices. If all of our folks within government have come from government straight out of school, that will prevent us from being able to adopt best practices from industry or from academia. So there really is a need I think for continuous cycling throughout a career, bouncing around between the different sectors in order to bring knowledge across them.
My first suggestion is to reach out to 80,000 Hours, which I think has been pulling together some advice about government jobs (see below). I think one of the pieces of advice is at least to consider it as an option, because we are nowhere near the saturation point of effective altruists going into government positions. There are fairly junior positions across government that have a high potential impact that we have trouble recruiting for.
My second suggestion is to get to know the people who work within the organizations where you’d like to work. You can learn a lot about those organizations, their structure, and their staff just from online websites as well as Wikipedia. You can find the biographies of some of the people whose careers you might want to mimic. One strategy is just to reverse engineer their biography. Figure out what the steps are that seem critical in getting to the positions that you would like to have in the future. On that point, many of these people are lawyers, but just as many of them are scientists and engineers, and we do need more philosophers in government as well. But I think you’ll find the diversity of talent that we need is ever growing. There’s a particular intersection between policy and technology that is extremely difficult to recruit for. So for people who are still picking their major or their concentration for a thesis, if you look at the science policy of blank, pretty much any of the topics that are critical on our list have not been saturated with attention. There’s still lots of low-hanging fruit to pick.
My last suggestion is to reach out to me, especially if you’re interested in pursuing a job, short-term or long-term, in national security or reducing global catastrophic risks. Mostly because I really need the help. That’s it.