Hiring engineers and researchers to help align GPT-3

My team at OpenAI, which works on al­ign­ing GPT-3, is hiring ML en­g­ineers and re­searchers. Ap­ply here for the ML en­g­ineer role and here for the ML re­searcher role.

GPT-3 is similar enough to “pro­saic” AGI that we can work on key al­ign­ment prob­lems with­out rely­ing on con­jec­ture or spec­u­la­tive analo­gies. And be­cause GPT-3 is already be­ing de­ployed in the OpenAI API, its mis­al­ign­ment mat­ters to OpenAI’s bot­tom line — it would be much bet­ter if we had an API that was try­ing to help the user in­stead of try­ing to pre­dict the next word of text from the in­ter­net.

I think this puts our team in a great place to have an im­pact:

  • If our re­search suc­ceeds I think it will di­rectly re­duce ex­is­ten­tial risk from AI. This is not meant to be a warm-up prob­lem, I think it’s the real thing.

  • We are work­ing with state of the art sys­tems that could pose an ex­is­ten­tial risk if scaled up, and our team’s suc­cess ac­tu­ally mat­ters to the peo­ple de­ploy­ing those sys­tems.

  • We are work­ing on the whole pipeline from “in­ter­est­ing idea” to “pro­duc­tion-ready sys­tem,” build­ing crit­i­cal skills and get­ting em­piri­cal feed­back on whether our ideas ac­tu­ally work.

We have the real-world prob­lems to mo­ti­vate al­ign­ment re­search, the fi­nan­cial sup­port to hire more peo­ple, and a re­search vi­sion to ex­e­cute on. We are bot­tle­necked by ex­cel­lent re­searchers and en­g­ineers who are ex­cited to work on al­ign­ment.

What the team does

In the past Reflec­tion fo­cused on fine-tun­ing GPT-3 us­ing a re­ward func­tion learned from hu­man feed­back. Our most re­cent re­sults are here, and had the un­usual virtue of si­mul­ta­neously be­ing ex­cit­ing enough to ML re­searchers to be ac­cepted at NeurIPS while be­ing de­scribed by Eliezer as “di­rectly, straight-up rele­vant to real al­ign­ment prob­lems.”

We’re cur­rently work­ing on three things:

  • [20%] Ap­ply­ing ba­sic al­ign­ment ap­proaches to the API, aiming to close the gap be­tween the­ory and prac­tice.

  • [60%] Ex­tend­ing ex­ist­ing ap­proaches to tasks that are too hard for hu­mans to eval­u­ate; in par­tic­u­lar, we are train­ing mod­els that sum­ma­rize more text than hu­man train­ers have time to read. Our ap­proach is to use weaker ML sys­tems op­er­at­ing over shorter con­texts to help over­see stronger ones over longer con­texts. This is con­cep­tu­ally straight­for­ward but still poses sig­nifi­cant en­g­ineer­ing and ML challenges.

  • [20%] Con­cep­tual re­search on do­mains that no one knows how to over­see and em­piri­cal work on de­bates be­tween hu­mans (see our 2019 writeup). I think the biggest open prob­lem is figur­ing out how and if hu­man over­seers can lev­er­age “knowl­edge” the model ac­quired dur­ing train­ing (see an ex­am­ple here).

If suc­cess­ful, ideas will even­tu­ally move up this list, from the con­cep­tual stage to ML pro­to­types to real de­ploy­ments. We’re view­ing this as prac­tice for in­te­grat­ing al­ign­ment into trans­for­ma­tive AI de­ployed by OpenAI or an­other or­ga­ni­za­tion.

What you’d do

Most peo­ple on the team do a sub­set of these core tasks:

  • De­sign+build+main­tain code for ex­per­i­ment­ing with novel train­ing strate­gies for large lan­guage mod­els. This in­fras­truc­ture needs to sup­port a di­ver­sity of ex­per­i­men­tal changes that are hard to an­ti­ci­pate in ad­vance, work as a solid base to build on for 6-12 months, and han­dle the com­plex­ity of work­ing with large lan­guage mod­els. Most of our code is main­tained by 1-3 peo­ple and con­sumed by 2-4 peo­ple (all on the team).

  • Oversee ML train­ing. Eval­u­ate how well mod­els are learn­ing, figure out why they are learn­ing badly, and iden­tify+pri­ori­tize+im­ple­ment changes to make them learn bet­ter. Tune hy­per­pa­ram­e­ters and man­age com­put­ing re­sources. Pro­cess datasets for ma­chine con­sump­tion; un­der­stand datasets and how they af­fect the model’s be­hav­ior.

  • De­sign and con­duct ex­per­i­ments to an­swer ques­tions about our mod­els or our train­ing strate­gies.

  • De­sign+build+main­tain code for del­e­gat­ing work to ~70 peo­ple who provide in­put to train­ing. We au­to­mate work­flows like sam­pling text from books, get­ting mul­ti­ple work­ers’ an­swers to ques­tions about that text, run­ning a lan­guage model on those an­swers, then show­ing the re­sults to some­one else for eval­u­a­tion. It also in­volves mon­i­tor­ing worker through­put and qual­ity, au­tomat­ing de­ci­sions about what tasks to del­e­gate to whom, and mak­ing it easy to add new work or change what peo­ple are work­ing on.

  • Par­ti­ci­pate in high-level dis­cus­sion about what the team should be work­ing on, and help brain­storm and pri­ori­tize pro­jects and ap­proaches. Com­pli­cated pro­jects seem to go more smoothly if ev­ery­one un­der­stands why they are do­ing what they are do­ing, is on the look­out for things that might slip through the cracks, is think­ing about the big pic­ture and helping pri­ori­tize, and cares about the suc­cess of the whole pro­ject.

If you are ex­cited about this work, ap­ply here for the ML en­g­ineer role and here for the ML re­searcher role.


The value
is not of type