Notes on hiring a copyeditor for CEA

Over the last year, there’s been a lot of dis­cus­sion about the “EA job mar­ket” and how to build an effec­tive ca­reer in an EA field.

A few months ago, I went through my first hiring pro­cess from the em­ployer side when I found a part-time copy­ed­i­tor to work on CEA’s so­cial me­dia posts (and a few other tasks). I thought my pro­cess might be in­ter­est­ing to peo­ple who’ve been fol­low­ing the afore­men­tioned dis­cus­sion, so I’m writ­ing up my notes in this post.

Meta: It can be legally tricky to write ap­pli­ca­tions, con­duct job in­ter­views, and provide feed­back to ap­pli­cants. I recom­mend con­sult­ing your lo­cal HR ex­pert be­fore at­tempt­ing to hire.

Statis­tics on the hiring process

The ini­tial ap­pli­ca­tion was meant to give me in­for­ma­tion about ap­pli­cants’ edit­ing skills and ex­pe­rience, as well as their fa­mil­iar­ity with EA (which I felt would be helpful for the role, given the ma­te­rial they’d be edit­ing and some­times writ­ing).

Ap­pli­cants were asked to edit half of a tran­script of an EA Global talk gen­er­ated by a tran­scrip­tion ser­vice, which con­tained many er­rors. They were given a two-hour limit to make it as clean and read­able as they could. Only a few ap­pli­cants didn’t finish the full edit, some be­cause they went over the time limit and oth­ers be­cause they ap­plied very soon be­fore the dead­line. It’s pos­si­ble that some peo­ple ig­nored the limit; I spot-checked the edit his­tory of some of the best tran­scripts and didn’t see this, but I didn’t check all tran­scripts in this way.

Ap­pli­cants were also asked to provide some in­for­ma­tion about them­selves; see the next sec­tion for a link to the full job de­scrip­tion.

Num­ber of ap­pli­cants who com­pleted the ini­tial work trial: 183

Num­ber of ap­pli­cants who scored at least “1” on each of two 1-3 scales (one for EA/​edit­ing ex­pe­rience, one for edit­ing skill): 147

(I de­scribe my sys­tem in more de­tail be­low, but you can think of this as “the num­ber of peo­ple who fol­lowed the in­struc­tions, seemed to be fluent in English, and in­di­cated a gen­uine in­ter­est in the po­si­tion.”)

Num­ber of ap­pli­cants who reached the in­ter­view stage: 21

Num­ber of ap­pli­cants who reached the “fi­nal work trial” stage: 8

The job description

Here’s the de­scrip­tion I used to ad­ver­tise the po­si­tion. I shared it through the EA Newslet­ter, the 80,000 Hours job board, the EA Job Post­ings Face­book group, and EA Work Club. I didn’t try to track how many can­di­dates came through each source. The po­si­tion was posted in the first week of June, and was open un­til the last day of June.

Some thoughts on how I han­dled this pro­cess, and what I wish I’d changed:

I didn’t have a good sense for how many peo­ple would ap­ply, so I erred on the side of hav­ing a more “open” de­scrip­tion: I de­scribed an “ideal” can­di­date, but didn’t set out many strict re­quire­ments.

On the one hand, this led to many more peo­ple ap­ply­ing than I had ex­pected, and quite a lot of ap­pli­cant time be­ing in­vested in a work test for minor (if any) benefit to ap­pli­cants. This makes me wish I’d done more re­search be­fore­hand to bet­ter un­der­stand how many peo­ple tend to ap­ply for these po­si­tions — for ex­am­ple, by ask­ing GiveWell about their past ex­pe­rience hiring peo­ple to write up con­ver­sa­tion notes (a po­si­tion with similarly loose qual­ifi­ca­tions).

On the other hand, had I screened for pro­fes­sional edit­ing ex­pe­rience or pro­fes­sional EA ex­pe­rience, I might have missed out on some of my best can­di­dates,per­haps in­clud­ing the per­son I even­tu­ally hired.

I could have saved even more ap­pli­cant time by ask­ing con­tacts in the EA com­mu­nity for refer­ences; sev­eral can­di­dates I chose to in­ter­view were peo­ple I ex­pect I’d have found through this method. How­ever, since I didn’t think the po­si­tion would re­quire a strong EA back­ground to be done well, I wanted to make the pro­cess more open and give chances to peo­ple look­ing for their first EA-al­igned work ex­pe­rience.

The ini­tial work test

The most im­por­tant skill for the po­si­tion was ba­sic copy­edit­ing: Even if some­one had a strong re­sume/​back­ground and a lot of pas­sion for EA, I still needed to be able to trust their ed­its and min­i­mize the time I spent re­view­ing their work.

Things I think went well with this test:

  • I was able to iden­tify a set of “tricky ed­its” that did a good job of sep­a­rat­ing the best ed­i­tors from the rest of the ap­pli­cants. This let me fo­cus mostly on those spots and save time read­ing through 150+ ap­pli­ca­tions.

Things I wish I’d re­con­sid­ered:

  • Had I re­al­ized so many peo­ple would ap­ply, I’d have aimed at a shorter test, per­haps with some spe­cial mod­ifi­ca­tions so that a shorter pas­sage would still have enough “tricky ed­its” to let me gauge an ap­pli­cant’s skill. While I’m guess­ing that the av­er­age time spent by ap­pli­cants was well un­der the two-hour limit, I still might have been able to save 50-100 hours of ap­pli­cant time with a cou­ple of hours of thought on my part. When hiring, you have con­trol over a lot of ap­pli­cants’ col­lec­tive time — don’t squan­der it!

Scor­ing rubric:

  • 0 = I couldn’t open the Google Doc, or the ap­pli­cant sent a doc­u­ment with no visi­ble ed­its (e.g. they shared it with the wrong per­mis­sions, leav­ing their ed­its in­visi­ble, and didn’t change those per­mis­sions af­ter I wrote to them let­ting them know what had hap­pened)

    • This was sur­pris­ingly com­mon. Upon re­flec­tion, I wish I’d writ­ten “and en­able com­ments” as “and share your doc with the set­ting “any­one with the link can com­ment” — it was the last line of the in­struc­tions, and a lot of peo­ple seem to have missed or mi­s­un­der­stood it.

  • 1= Quite a few mis­takes; based on this test, I’d feel un­com­fortable with this can­di­date pub­lish­ing on CEA’s be­half, and I think I’d have to watch them closely

  • 2 = Few mis­takes; copy al­most as clean as what I think I’d have done; hits all or al­most all of the tricky spots correctly

  • 3 = Al­most zero mis­takes; copy as clean or cleaner than what I think I’d have done; in ad­di­tion to get­ting the tricky spots right, may also show some cre­ative flair (e.g. us­ing bul­let points to break up a very long sen­tence with many ex­am­ples of some­thing)

The differ­ence be­tween, say, a 2 and a 2.5 was highly sub­jec­tive. It’s likely that I gave slightly higher scores to can­di­dates whose nat­u­ral “flow” in edit­ing sen­tences was similar to mine, even if other can­di­dates’ work was perfectly gram­mat­i­cal/​smooth. This was sub­con­scious, but I think I en­dorse it; if I’m go­ing to look over a lot of some­one’s work, it helps if we have a similar sense for how sen­tences should sound.

Score dis­tri­bu­tion:

  • Score <1: 34

  • 1 ⇐ score < 2: 110

  • 2 ⇐ score < 2.5: 26

  • 2.5 ⇐ score: 13 (two perfect “3” scores)

Feed­back from the per­son I even­tu­ally hired:

“One of the rea­sons I ap­plied was that your ini­tial screen/​first step was a test — not an in­ter­view. It made me re­spect CEA’s at­tempts to avoid hiring bias (plus it backed up the org’s claim that you are ev­i­dence-based).”

Grad­ing the applications

In ad­di­tion to the writ­ing task, I asked ap­pli­cants to send some in­for­ma­tion about them­selves. I graded this on an 0-3 scale, as well. I’m satis­fied with the in­for­ma­tion I asked for, and I can’t eas­ily think of any ques­tions I wish I’d in­cluded in hind­sight.

Scor­ing rubric:

  • 0 = the ap­pli­cant didn’t send any in­for­ma­tion about them­selves, or it was clear that they were to­tally un­fa­mil­iar with EA (or firmly op­posed to it, etc.)

  • 1 = the ap­pli­cant seemed fa­mil­iar with the ba­sic ideas of EA but not much be­yond that; al­ter­na­tively, they had some edit­ing ex­pe­rience but no fa­mil­iar­ity with EA

  • 2 = the ap­pli­cant showed a lot of fa­mil­iar­ity with EA and/​or a mod­er­ately good sense of what CEA does; they have some ex­pe­rience with writ­ing and edit­ing, though maybe not in a role spe­cific to copyediting

  • 3 = the ap­pli­cant was a long­time mem­ber of the EA com­mu­nity and/​or had pre­vi­ous EA work ex­pe­rience, or paired at least mod­er­ate EA knowl­edge with ex­pe­rience as a copyeditor

I cared more about perfor­mance on the work trial than on an ap­pli­cant’s EA back­ground; I think it takes less time and effort to be­come fa­mil­iar with EA (as­sum­ing at least a ba­sic in­cli­na­tion to­ward its ideas) than to be­come a very pol­ished copy­writer/​ed­i­tor from a baseline of hav­ing only mod­er­ate skill.

Within the ap­pli­ca­tion score, know­ing about some­one’s edit­ing back­ground was helpful, but I put less weight on that than on their EA back­ground, be­cause I had ac­cess to their edit­ing task already. (For what it’s worth, edit­ing ex­pe­rience did cor­re­late quite pos­i­tively with perfor­mance on the task.)

Score dis­tri­bu­tion:

  • Score <1: 14

  • 1 ⇐ score < 2: 94

  • 2 ⇐ score < 2.5: 52

  • 2.5 ⇐ score: 23 (four perfect “3” scores)

Se­cond-round decisions

I calcu­lated a “to­tal” score by mul­ti­ply­ing the 0-3 edit­ing score by two, then adding the 0-3 ap­pli­ca­tion score. Nine can­di­dates scored at 7 or above, with many more be­tween 6 and 7. I se­lected some but not all can­di­dates from the 6-7 range, in part by us­ing the fol­low­ing fac­tors:

  • Ex­clud­ing a cou­ple of peo­ple who re­quired sub­stan­tially higher salaries than the $25/​hour we stated in the ad; we might have paid such rates for an out­stand­ing can­di­date, but not for one whose scores were similar to those of many other candidates

  • Ad­ding a small bonus (0.5 points) for can­di­dates within a short dis­tance of Berkeley or Oxford, since it is mildly use­ful when our con­trac­tors can visit one of CEA’s offices from time to time

  • Ad­ding a small bonus (0.5 points) for can­di­dates who in­cluded strong and rele­vant per­sonal recom­men­da­tions (e.g. a pro­fes­sor who praised a can­di­date’s work as an ed­i­tor for a book they had pub­lished)

  • Ad­ding a small bonus (0.5 points) for peo­ple who sub­mit­ted es­pe­cially well-writ­ten ap­pli­ca­tions. This doesn’t mean beau­tiful flow­ing prose (some peo­ple who got this bonus used bul­let points), but it did mean go­ing be­yond the stan­dard “list of ac­com­plish­ments” in a way that helped me un­der­stand:

    • A can­di­date’s writ­ing style (im­por­tant for a job where some tasks in­volve origi­nal writ­ing), or:

    • What they might be like to work with (e.g. can­di­dates who in­cluded a brief sum­mary of their most rele­vant ex­pe­rience, but also a link to a more com­plete sum­mary in case I found it use­ful to read fur­ther — or who did var­i­ous other things to make the eval­u­a­tion pro­cess smoother on my end)

This isn’t an ex­haus­tive list of fac­tors that mat­tered (there may have been a dozen el­e­ments in any given ap­pli­ca­tion that made me more or less in­clined to move a can­di­date to the next round), but it cov­ers all the most im­por­tant points.

Fol­low-up and feedback

Of the can­di­dates I did not in­ter­view, most got the fol­low­ing email:

Thank you for ap­ply­ing to the free­lance copy­ed­i­tor po­si­tion at CEA. I ap­pre­ci­ate your tak­ing the time to com­plete the ini­tial work test, and your in­ter­est in our mis­sion.

After care­ful con­sid­er­a­tion, I’ve de­cided not to move for­ward with your ap­pli­ca­tion.

Please don’t take this as a sign that you aren’t a ca­pa­ble ed­i­tor, or that you shouldn’t ap­ply for po­si­tions with other or­ga­ni­za­tions con­nected to effec­tive al­tru­ism. More than 180 peo­ple ap­plied for the po­si­tion, and many peo­ple with strong qual­ifi­ca­tions didn’t pass the first stage.

If you have any fur­ther ques­tions, please let me know; I’m open to pro­vid­ing in­di­vi­d­ual feed­back, though it may be brief. And I wish you the best of luck with any other edit­ing jobs for which you may ap­ply!

How­ever, 28 non-in­ter­viewed can­di­dates demon­strated strong edit­ing skills and had work trial scores com­pet­i­tive with some of the can­di­dates I in­ter­viewed. I ex­pected these can­di­dates to be com­pet­i­tive ap­pli­cants for other writ­ing/​edit­ing jobs at EA orgs that might open up in the fu­ture. They got the fol­low­ing email:

Thank you for ap­ply­ing to the copy­ed­i­tor po­si­tion at CEA. I ap­pre­ci­ate your tak­ing the time to com­plete the work test, and your in­ter­est in our mis­sion.

More than 180 peo­ple ap­plied for the po­si­tion, and many peo­ple with strong qual­ifi­ca­tions didn’t pass the first stage. After care­ful con­sid­er­a­tion, I’ve de­cided not to move for­ward with your ap­pli­ca­tion.

How­ever, you were one of a small num­ber of ap­pli­cants who, de­spite not pass­ing to the sec­ond round, made un­usu­ally strong ed­its. Based on your work test, I think you might be a good can­di­date for other jobs that in­volve writ­ing or edit­ing for or­ga­ni­za­tions con­nected to effec­tive al­tru­ism — most of which don’t have nearly as many ap­pli­cants. I en­courage you to ap­ply to those po­si­tions in the fu­ture (if they in­ter­est you).

If you have any fur­ther ques­tions, please let me know; I wish you the best of luck with your fu­ture ap­pli­ca­tions!

(Note that I didn’t in­clude the “feed­back” note here: In ret­ro­spect, this was a mis­take. While high-scor­ing ap­pli­cants may not have needed edit­ing ad­vice, many peo­ple also asked for my thoughts on their re­sumes/​in­tro emails, and this seemed valuable to provide; I wish I’d got­ten more re­quests.)

Can­di­dates I did in­ter­view got an email which in­cluded the fol­low­ing lan­guage, meant to help them un­der­stand the pro­cess and de­cide whether they wanted to keep in­vest­ing time in the po­si­tion. (For ex­am­ple, some­one might have been will­ing to stay in the run­ning:for a free­lance gig if they were com­pet­ing with two other peo­ple, but not with twenty.)

For con­text, this is what I have planned for the rest of the ap­pli­ca­tion pro­cess:

  1. In­ter­views with can­di­dates who passed the first round (21 out of 183 ap­pli­cants)

  2. A sec­ond work test for ap­prox­i­mately 10 of those can­di­dates, based on their in­ter­views and a closer ex­am­i­na­tion of their origi­nal work tests.

  3. One per­son cho­sen to take the po­si­tion — though it’s pos­si­ble that work might be split among mul­ti­ple peo­ple, or that other ap­pli­cants might be asked to take the po­si­tion if the first per­son hired be­comes un­able to con­tinue the work.

I strongly ex­pect to make my fi­nal de­ci­sion by the end of Au­gust.

Please let me know if you’d re­quire an ear­lier de­ci­sion in or­der to be able to take the job.

Nearly two dozen ap­pli­cants (of those I didn’t in­ter­view) asked for feed­back. I re­sponded to them with spe­cific notes on their par­tic­u­lar ap­pli­ca­tions, in­clud­ing edit­ing mis­takes and ar­eas where I felt un­cer­tain about their ex­pe­rience, as well as pos­i­tive feed­back (many of these can­di­dates did make strong ed­its, or wrote ex­cel­lent ap­pli­ca­tion emails).

Feed­back on my feed­back (when I re­ceived it) was highly pos­i­tive; it seems as though peo­ple re­ally ap­pre­ci­ated hear­ing back from a “hiring man­ager”. Send­ing those notes took a fair amount of time, but I’m glad I did it; it seems to have been helpful to some of the ap­pli­cants, and I hope that it also made them feel more pos­i­tive about CEA, and about EA in gen­eral. I’d cau­tiously recom­mend that other or­ga­ni­za­tions do the same if they can spare the time and trou­ble (again, le­gal trick­i­ness).

Interviews

Every can­di­date offered an in­ter­view chose to sched­ule one. The in­ter­views had less struc­ture than I’d have liked; while I asked each can­di­date the same set of ini­tial ques­tions, alongside spe­cific ques­tions about their ap­pli­ca­tion, I didn’t have a scor­ing rubric in mind.

I wound up giv­ing each in­ter­view a score “out of 10” (ac­tual scores ranged from 6 to 9) af­ter I finished, which made it hard to di­rectly com­pare can­di­dates later on. How­ever, the can­di­date with the strongest in­ter­view, who I wound up hiring, also had among the strongest trial tasks in both rounds, so I wound up not need­ing to think too hard about these com­par­i­sons.

How I se­lected can­di­dates for the sec­ond work trial (fac­tors or­dered from most to least im­por­tant):

  1. The strength of their ini­tial ap­pli­ca­tion (still a ma­jor fac­tor, and weighted more heav­ily than the in­ter­view)

  2. How cer­tain I was that they’d be available for the po­si­tion for the right num­ber of hours, and for a long time to come, de­spite its part-time na­ture (dis­cussing this was a part of the in­ter­view)

  3. How en­gaged and cu­ri­ous they were dur­ing the in­ter­view? Did they ask ques­tions that showed they were se­ri­ously think­ing about how the po­si­tion would work out for them? Did they seem to be think­ing care­fully about my ques­tions be­fore they an­swered?

The eight strongest can­di­dates re­ceived this task as the fi­nal stage of the ap­pli­ca­tion.

I’m happy with the first two tasks (I got a great sense for how the can­di­dates thought about so­cial me­dia, plus a lot of use­ful sug­ges­tions for im­prove­ments to the EA Newslet­ter). But I don’t think the third task wound up mat­ter­ing much; it’s pos­si­ble that I should have skipped it to save the can­di­dates’ time.

The most im­por­tant fac­tors in my eval­u­a­tion of this test (in no par­tic­u­lar or­der):

  • Were the sam­ple so­cial me­dia posts writ­ten such that I’d have been happy to see them ap­pear on our feeds with­out fur­ther changes? If not, did the posts’ prob­lems seem like they’d be easy to pre­vent with the right ad­vice to the au­thor?

  • What frac­tion of known ty­pos and lay­out odd­i­ties (e.g. mis­matched quo­ta­tion marks) did the can­di­date catch? I’m not too con­cerned about mis­matched quo­ta­tion marks, but this felt like a good mea­sure of at­ten­tion to de­tail.

  • Was the Newslet­ter ad­vice writ­ten such that I could eas­ily act on it, or at least run an ex­per­i­ment to test its im­pact?

  • When giv­ing ad­vice, did the can­di­date ac­knowl­edge/​ac­count for their un­cer­tainty about the Newslet­ter’s pur­pose, au­di­ence, and his­tory? Did they know what they didn’t know?

Note: The set of “known” ty­pos/​odd­i­ties con­sisted of all the differ­ent is­sues that can­di­dates found; I didn’t re-copy­edit my own newslet­ter for this task.

Three of the eight can­di­dates had es­pe­cially strong tests (par­tic­u­larly their Newslet­ter ad­vice). I in­formed the top can­di­date that I wanted to offer her the po­si­tion, and I in­formed the other two know that I was strongly con­sid­er­ing them if the top can­di­date did not ac­cept.

After think­ing about the ini­tial offer and ne­go­ti­at­ing briefly for a higher rate, she did ac­cept the po­si­tion, and is cur­rently work­ing on sev­eral CEA pro­jects. (Her new rate was still much lower than those re­quested by can­di­dates I ex­cluded for their high rate re­quire­ments.)

She re­quested that I not use her name, but gave me per­mis­sion to talk a bit about her back­ground and ap­pli­ca­tion:

  • She had less back­ground EA knowl­edge than most of the other can­di­dates, but more pro­fes­sional ex­pe­rience.

  • I’m al­most cer­tain we’d have hired some­one else had I made the job ad more nar­row, or re­lied en­tirely on refer­ences from my con­tacts.

  • Her ed­its on the first trial and her posts on the sec­ond trial were nearly perfect. It was hard to spot any changes I’d have made as her ed­i­tor in ei­ther case. Since one of my cen­tral goals in mak­ing this hire was to save time, I was happy to find some­one whose work didn’t seem like it would need much dou­ble-check­ing. (Sev­eral other ap­pli­cants fit this de­scrip­tion, too.)

Keep­ing in touch with can­di­dates (want to hire some­one?)

Hiring for this po­si­tion took dozens of hours of my time, and hun­dreds of hours of can­di­dates’ time. I want to squeeze as much value as I can from the pro­cess.

So, in ad­di­tion to hiring a can­di­date, I’ve also kept a record of the other ap­pli­cants who most im­pressed me, so that I can let them know if I hear about promis­ing op­por­tu­ni­ties. I’ve already referred a few can­di­dates for differ­ent part-time roles at other EA orgs, and I an­ti­ci­pate more chances to come.

(If you’re look­ing to hire some­one for writ­ing and/​or edit­ing, let me know!)

Any ques­tions?

I’d be happy to re­spond to ques­tions about the hiring pro­cess or any­thing else I’ve men­tioned in this post. Please leave a com­ment or send me an email.