The obvious answer is âby working on important things at orgs which need software engineersâ. To name specific examples that are somewhat biased towards the orgs I know well:
MIRI needs software engineers who can learn functional programming and some math
I think that if youâre an engineer who likes functional programming, it might be worth your time to take a Haskell job and gamble on MIRI wanting to hire you one day when youâre really good at it. One person who currently works at MIRI is an EA who worked in Haskell for a few years; his professional experience is really helpful for him. If youâre interested in doing this, feel free to email me asking about whether I think itâs a good idea for you.
OpenAIâs safety team needs software engineers who can work as research engineers (which might only be a few months of training if youâre coming from a software engineering background; MIRI has a retraining program for software engineers who want to try this; if youâre interested in that, you should email me.)
I have two main thoughts on how talented software engineers should try to do good.
Strategy 1: become a great software engineer
I think that itâs worth considering a path where you try to become an extremely good software engineer/âcomputer scientist. (Iâm going to lump those two disciplines together in the rest of this answer.)
Here are some properties which really good engineers tend to have. Iâm going to give examples which are true of a friend of mine who I think is an exceptional engineer.
Extremely broad knowledge of computer science (and other quantitative fields). Eg knowledge of ML, raytracing-based rendering, SMT solvers, formal verification, cryptography, physics and biology and math. This means that when heâs in one discipline (like game programming) he can notice that the problem heâs solving could be done more effectively using something from a totally different discipline. Edward Kmett is another extremely knowledgeable programmer who uses his breadth of knowledge to spot connections and write great programs.
Experience with a wide variety of programming settingsâweb programming, data science, distributed systems, GPU programming.
Experience solving problems in a wide variety of computer science disciplinesâdesigning data structures, doing automata theory reductions, designing distributed systems
Experience getting an ill-defined computer science problem, and then searching around for a crisp understanding of whatâs happening, and then turning that into code and figuring out what fundamental foolish mistakes youâd made in your attempt at a crisp understanding.
I am not as good as this friend of mine, but Iâm a lot better at my job because I am able to solve problems like my data structure search problem, and I got much better at solving problems like that from trying to solve many problems like that.
How do I think you should try to be a great programmer? I donât really know, but here are some ideas:
Try to write many different types of programs from scratch. The other day I spent a couple hours trying to write the core of an engine for a real-time strategy game from scratch; I think I learned something useful from this experience. One problem with working as a professional programmer is that you relatively rarely have to build things from scratch; I think itâs worth doing that in your spare time. (Of course, thereâs very useful experience that you get from building large projects, too; your work can be a good place to get that experience.)
Learn many different disciplines. I think I got a lot out of learning to program web frontends.
Have coworkers who are good programmers.
Try to learn about subjects and then investigate questions about them by programming. For example, Iâve been learning about population genetics recently, and Iâve been thinking about trying to write a library which does calculations about coalescent theory; I think that this will involve an interesting set of design questions as well as involving designing and using some interesting algorithms, and itâs good practice for finding the key facts in a subject that Iâm learning.
Itâs hard to know what the useful advice to provide here is. I guess I want to say that (especially early in your career, eg when youâre an undergrad) it might be worth following your passions and interests within computer science, and I think you should plausibly do the kinds of programming youâre most excited by, instead of doing the kinds of programming that feel most directly relevant to EA.
Strategy 2: becoming really useful
Hereâs part of my more general theory of how to do good as a software engineer, a lot of which generalizes to other skillsets:
I think itâs helpful to think about the question âwhy canât EA orgs just hire non-EAs to do software engineering work for themâ. Some sample answers:
Non-EAs are often unwilling to join weird small orgs, because theyâre very risk-averse and donât want to join a project that might fold after four months.
Non-EAs arenât as willing to do random generalist tasks, or scrappy product-focused work like building the 80K career quiz, analytics dashboards, or web apps to increase efficiency of various internal tasks.
Itâs easier to trust EAs than non-EAs to do a task when itâs hard to supervise them, because the EAs might be more intrinsically motivated by the task, and they might also have more context on what the org is trying to do.
Non-EAs arenât as willing to do risky things like spending a few months learning some topic (eg ML, or a particular type of computer science, or how to build frontend apps on top of Airtable) which might not translate into a job.
Non-EAs can be disruptive to various aspects of the EA culture that the org wants to preserve. For example, in my experience EA orgs often have a culture that involves people being pretty transparent with their managers about the weaknesses in the work theyâve done, and hiring people who have less of that attitude can screw up the whole culture.
I think EA software engineers should try to translate those into ways that they can be better at doing EA work. For example, I think EAs should do the following (these pieces of advice are ranked roughly most to least important.):
Try to maintain flexibility in your work situation, so that you can quickly take opportunities which arise for which youâd be a good fit. In order to do this, itâs good to have some runway and general life flexibility.
Be willing to take jobs that arenât entirely software engineering, or which involve scrappy product-focused work. Consider taking non-EA jobs which are going to help you learn these generalist or product-focused-engineering skills. (For example, working for Triplebyte was great for me because it involved a lot of non-engineering tasks and a lot of scrappy, product-focused engineering.)
Practice doing work in a setting where you have independent control over your work and where you need to get context on a particular industry. For example, it might be good to take a job at a startup where youâre going to have relatively large amounts of freedom to work on projects that you think will help the company become more efficient, and where the startup involves dealing with a specific domain like spoon manufacturing and so you have to learn about this specific domain in order to be maximally productive.
Be willing to take time off to learn skills that might be useful. (In particular, you should be relatively enthusiastic to do this in cases where some EA or EA org is willing to fund you to do it.) Also, compared to most software engineers you should be more inclined to take jobs that will teach you more varied things but which are worse in other ways.
Practice working in an environment which rewards transparency and collaborative truth-seeking. I am very unconfident about the following point, but: perhaps you should be wary of working in companies where thereâs a lot of office politics or where you have to make up a lot of bullshit, because perhaps that trains you in unhealthy epistemic practices.
I think the point about flexibility is extremely important. I think that if you set your life up so that most of the time you can leave your current non-EA job and move to an EA job within two months, youâre much more likely to get jobs which are very high impact.
A point thatâs related to flexibility but distinct: Sometimes I talk to EAs about their careers and they seem to have concrete plans that we can talk about directly, and theyâre able to talk about the advantages and disadvantages of various paths they could take, and it overall feels like weâre working together to help them figure out what the best thing for them to do is. When conversations go like this, itâs much easier to do things like figure out what theyâd have to change their minds about in order to think they should drop out of their PhD. I think that when people have a mindset like this, itâs much easier for them to be persuaded of opportunities which are actually worth them inconveniencing themselves to access. In contrast, some people seem to treat direct work as something youâre âsupposedâ to consider, so they put a token effort into it, but their heart isnât in it and they arenât putting real cognitive effort into thinking about different possibilities, ways to overcome initial obstacles, etc.
I think these two points are really important; I think that when I meet someone who is flexible in those ways, my forecast of their impact is about twice as high as it would have been if they werenât.
As an appendix to the above, some of my best learning experiences as a programmer were the following (starting from when I started programming properly as a freshman in 2012). (Many of these arenât that objectively hard (and would fit in well as projects in a CS undergrad course); they were much harder for me because I didnât have the structure of a university course to tell me what design decisions were reasonable and when I was going down blind alleys. I think that this difficulty created some great learning experiences for me.)
I translated the proof of equivalence between regular expressions and finite state machines from âIntroduction to Automata Theory, Languages, and Computationâ into Haskell.
I wrote a program which would take a graph describing a circuit built from resistors and batteries and then solve for the currents and potential drops.
I went to App Academy and learned to write full stack web applications.
I wrote a compiler from C to assembly in Scala. It took a long time for me to figure out that I should eg separate out the compiling to an intermediate output that didnât have registers allocated.
I wrote the first version of my data structure searcher. (Not linking because Iâm embarrassed by how much worse it was than my second attempt.)
I wrote the second version of my data structure searcher, which involved putting a lot of effort into deconfusing myself about what data structures are and how they connect to each other.
One example of something I mean by deconfusion here: when you have a composite data structure (eg a binary search tree and a heap representing the same data, with pointers into each other), when youâre figuring out how quickly your reads happen, you take the union of all the things you can do with all your structuresâeg you can read in any of the structures. But when you want to do a write, you need to do it in all of the structures, so you take the maximum write time. This feels obvious when I write it now, but wasnât obvious until Iâd figured out exactly what question was interesting to me. And itâs somewhat more complicatedâfor example, we actually want to take the least upper bound rather than the maximum.
At Triplebyte, I wrote a variety of web app features. The one I learned the most from was building a UI for composing structured emails quicklyâthe UI concept was original to me, and it was great practice at designing front end web widgets and building things to solve business problems. My original design kind of sucked so I had to rewrite it; this was also very helpful. I learned the most from trying to design complicated frontend UI components for business logic, because that involves more design work than backend Rails programming does.
I then shifted my programming efforts to MIRI work; my learning here has mostly been a result of learning more Haskell and trying to design good abstractions for some of the things weâre doing; Iâve also recently had to think about the performance of my code, which has been interesting.
I learned basically nothing useful from my year at PayPal.
I have opinions about how to choose jobs in order to maximize how much programming you learn and I might write them up at some point.
The motivation behind strategy 2 seems pretty clear; are you emphasising strategy 1 (become a great engineer) for its instrumental benefit to strategy 2 (become useful to EA orgs), or for some other reason, like EtG?
Strongly agree that some of the best engineers I come across have had very broad, multi-domain knowledge (and have been able to apply it cross-domain to whatever problem theyâre working on).
One factor here is that a lot of AI safety research seems to need ML expertise, which is one of my least favorite types of CS/âengineering.
Another is that compared to many EAs I think I have a comparative advantage at roles which require technical knowledge but not doing technical research day-to-day.
Iâm emphasizing strategy 1 because I think that there are EA jobs for software engineers where the skill ceiling is extremely high, so if youâre really good itâs still worth it for you to try to become much better. For example, AI safety research needs really great engineers at AI safety research orgs.
How should talented EA software engineers best put their skills to use?
The obvious answer is âby working on important things at orgs which need software engineersâ. To name specific examples that are somewhat biased towards the orgs I know well:
MIRI needs software engineers who can learn functional programming and some math
I think that if youâre an engineer who likes functional programming, it might be worth your time to take a Haskell job and gamble on MIRI wanting to hire you one day when youâre really good at it. One person who currently works at MIRI is an EA who worked in Haskell for a few years; his professional experience is really helpful for him. If youâre interested in doing this, feel free to email me asking about whether I think itâs a good idea for you.
OpenAIâs safety team needs software engineers who can work as research engineers (which might only be a few months of training if youâre coming from a software engineering background; MIRI has a retraining program for software engineers who want to try this; if youâre interested in that, you should email me.)
Ought needs an engineering lead
The 80000 Hours job board lists positions
I have two main thoughts on how talented software engineers should try to do good.
Strategy 1: become a great software engineer
I think that itâs worth considering a path where you try to become an extremely good software engineer/âcomputer scientist. (Iâm going to lump those two disciplines together in the rest of this answer.)
Here are some properties which really good engineers tend to have. Iâm going to give examples which are true of a friend of mine who I think is an exceptional engineer.
Extremely broad knowledge of computer science (and other quantitative fields). Eg knowledge of ML, raytracing-based rendering, SMT solvers, formal verification, cryptography, physics and biology and math. This means that when heâs in one discipline (like game programming) he can notice that the problem heâs solving could be done more effectively using something from a totally different discipline. Edward Kmett is another extremely knowledgeable programmer who uses his breadth of knowledge to spot connections and write great programs.
Experience with a wide variety of programming settingsâweb programming, data science, distributed systems, GPU programming.
Experience solving problems in a wide variety of computer science disciplinesâdesigning data structures, doing automata theory reductions, designing distributed systems
Experience getting an ill-defined computer science problem, and then searching around for a crisp understanding of whatâs happening, and then turning that into code and figuring out what fundamental foolish mistakes youâd made in your attempt at a crisp understanding.
I am not as good as this friend of mine, but Iâm a lot better at my job because I am able to solve problems like my data structure search problem, and I got much better at solving problems like that from trying to solve many problems like that.
How do I think you should try to be a great programmer? I donât really know, but here are some ideas:
Try to write many different types of programs from scratch. The other day I spent a couple hours trying to write the core of an engine for a real-time strategy game from scratch; I think I learned something useful from this experience. One problem with working as a professional programmer is that you relatively rarely have to build things from scratch; I think itâs worth doing that in your spare time. (Of course, thereâs very useful experience that you get from building large projects, too; your work can be a good place to get that experience.)
Learn many different disciplines. I think I got a lot out of learning to program web frontends.
Have coworkers who are good programmers.
Try to learn about subjects and then investigate questions about them by programming. For example, Iâve been learning about population genetics recently, and Iâve been thinking about trying to write a library which does calculations about coalescent theory; I think that this will involve an interesting set of design questions as well as involving designing and using some interesting algorithms, and itâs good practice for finding the key facts in a subject that Iâm learning.
Itâs hard to know what the useful advice to provide here is. I guess I want to say that (especially early in your career, eg when youâre an undergrad) it might be worth following your passions and interests within computer science, and I think you should plausibly do the kinds of programming youâre most excited by, instead of doing the kinds of programming that feel most directly relevant to EA.
Strategy 2: becoming really useful
Hereâs part of my more general theory of how to do good as a software engineer, a lot of which generalizes to other skillsets:
I think itâs helpful to think about the question âwhy canât EA orgs just hire non-EAs to do software engineering work for themâ. Some sample answers:
Non-EAs are often unwilling to join weird small orgs, because theyâre very risk-averse and donât want to join a project that might fold after four months.
Non-EAs arenât as willing to do random generalist tasks, or scrappy product-focused work like building the 80K career quiz, analytics dashboards, or web apps to increase efficiency of various internal tasks.
Itâs easier to trust EAs than non-EAs to do a task when itâs hard to supervise them, because the EAs might be more intrinsically motivated by the task, and they might also have more context on what the org is trying to do.
Non-EAs arenât as willing to do risky things like spending a few months learning some topic (eg ML, or a particular type of computer science, or how to build frontend apps on top of Airtable) which might not translate into a job.
Non-EAs can be disruptive to various aspects of the EA culture that the org wants to preserve. For example, in my experience EA orgs often have a culture that involves people being pretty transparent with their managers about the weaknesses in the work theyâve done, and hiring people who have less of that attitude can screw up the whole culture.
I think EA software engineers should try to translate those into ways that they can be better at doing EA work. For example, I think EAs should do the following (these pieces of advice are ranked roughly most to least important.):
Try to maintain flexibility in your work situation, so that you can quickly take opportunities which arise for which youâd be a good fit. In order to do this, itâs good to have some runway and general life flexibility.
Be willing to take jobs that arenât entirely software engineering, or which involve scrappy product-focused work. Consider taking non-EA jobs which are going to help you learn these generalist or product-focused-engineering skills. (For example, working for Triplebyte was great for me because it involved a lot of non-engineering tasks and a lot of scrappy, product-focused engineering.)
Practice doing work in a setting where you have independent control over your work and where you need to get context on a particular industry. For example, it might be good to take a job at a startup where youâre going to have relatively large amounts of freedom to work on projects that you think will help the company become more efficient, and where the startup involves dealing with a specific domain like spoon manufacturing and so you have to learn about this specific domain in order to be maximally productive.
Be willing to take time off to learn skills that might be useful. (In particular, you should be relatively enthusiastic to do this in cases where some EA or EA org is willing to fund you to do it.) Also, compared to most software engineers you should be more inclined to take jobs that will teach you more varied things but which are worse in other ways.
Practice working in an environment which rewards transparency and collaborative truth-seeking. I am very unconfident about the following point, but: perhaps you should be wary of working in companies where thereâs a lot of office politics or where you have to make up a lot of bullshit, because perhaps that trains you in unhealthy epistemic practices.
I think the point about flexibility is extremely important. I think that if you set your life up so that most of the time you can leave your current non-EA job and move to an EA job within two months, youâre much more likely to get jobs which are very high impact.
A point thatâs related to flexibility but distinct: Sometimes I talk to EAs about their careers and they seem to have concrete plans that we can talk about directly, and theyâre able to talk about the advantages and disadvantages of various paths they could take, and it overall feels like weâre working together to help them figure out what the best thing for them to do is. When conversations go like this, itâs much easier to do things like figure out what theyâd have to change their minds about in order to think they should drop out of their PhD. I think that when people have a mindset like this, itâs much easier for them to be persuaded of opportunities which are actually worth them inconveniencing themselves to access. In contrast, some people seem to treat direct work as something youâre âsupposedâ to consider, so they put a token effort into it, but their heart isnât in it and they arenât putting real cognitive effort into thinking about different possibilities, ways to overcome initial obstacles, etc.
I think these two points are really important; I think that when I meet someone who is flexible in those ways, my forecast of their impact is about twice as high as it would have been if they werenât.
As an appendix to the above, some of my best learning experiences as a programmer were the following (starting from when I started programming properly as a freshman in 2012). (Many of these arenât that objectively hard (and would fit in well as projects in a CS undergrad course); they were much harder for me because I didnât have the structure of a university course to tell me what design decisions were reasonable and when I was going down blind alleys. I think that this difficulty created some great learning experiences for me.)
I translated the proof of equivalence between regular expressions and finite state machines from âIntroduction to Automata Theory, Languages, and Computationâ into Haskell.
I wrote a program which would take a graph describing a circuit built from resistors and batteries and then solve for the currents and potential drops.
I wrote a GUI for a certain subset of physics problems; this involved a lot of deconfusion-style thinking as well as learning how to write GUIs.
I went to App Academy and learned to write full stack web applications.
I wrote a compiler from C to assembly in Scala. It took a long time for me to figure out that I should eg separate out the compiling to an intermediate output that didnât have registers allocated.
I wrote the first version of my data structure searcher. (Not linking because Iâm embarrassed by how much worse it was than my second attempt.)
I wrote the second version of my data structure searcher, which involved putting a lot of effort into deconfusing myself about what data structures are and how they connect to each other.
One example of something I mean by deconfusion here: when you have a composite data structure (eg a binary search tree and a heap representing the same data, with pointers into each other), when youâre figuring out how quickly your reads happen, you take the union of all the things you can do with all your structuresâeg you can read in any of the structures. But when you want to do a write, you need to do it in all of the structures, so you take the maximum write time. This feels obvious when I write it now, but wasnât obvious until Iâd figured out exactly what question was interesting to me. And itâs somewhat more complicatedâfor example, we actually want to take the least upper bound rather than the maximum.
At Triplebyte, I wrote a variety of web app features. The one I learned the most from was building a UI for composing structured emails quicklyâthe UI concept was original to me, and it was great practice at designing front end web widgets and building things to solve business problems. My original design kind of sucked so I had to rewrite it; this was also very helpful. I learned the most from trying to design complicated frontend UI components for business logic, because that involves more design work than backend Rails programming does.
I wrote another version of my physics problem GUI, which taught me about designing UIs in front of complicated functional backends.
I then shifted my programming efforts to MIRI work; my learning here has mostly been a result of learning more Haskell and trying to design good abstractions for some of the things weâre doing; Iâve also recently had to think about the performance of my code, which has been interesting.
I learned basically nothing useful from my year at PayPal.
I have opinions about how to choose jobs in order to maximize how much programming you learn and I might write them up at some point.
This is an awesome answer; thanks Buck!
The motivation behind strategy 2 seems pretty clear; are you emphasising strategy 1 (become a great engineer) for its instrumental benefit to strategy 2 (become useful to EA orgs), or for some other reason, like EtG?
Strongly agree that some of the best engineers I come across have had very broad, multi-domain knowledge (and have been able to apply it cross-domain to whatever problem theyâre working on).
(Notably, the other things you might work on if you werenât at MIRI seem largely to be non-software-related)
I hadnât actually noticed that.
One factor here is that a lot of AI safety research seems to need ML expertise, which is one of my least favorite types of CS/âengineering.
Another is that compared to many EAs I think I have a comparative advantage at roles which require technical knowledge but not doing technical research day-to-day.
Iâm emphasizing strategy 1 because I think that there are EA jobs for software engineers where the skill ceiling is extremely high, so if youâre really good itâs still worth it for you to try to become much better. For example, AI safety research needs really great engineers at AI safety research orgs.