Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
This isn’t completely directly related, but this is a comment given in the EA subreddit to this post. This comment was well upvoted.
This person appears to be a staff engineer and hiring manager at Google, and has worked over 10 years there.
There is a lot going on in this comment. I think the content is important. It talks about institutional competence, culture fit with founders, operations (which might be underrated and this comment one reason—talent can smell org competence) and economic security, that seem to apply to rare talent.
Counterintuitively, I see the underlying issues as tending to favoring EA. I don’t want to write a giant manifesto about it unless there is demand.
It’s worth noting that many of these restrictions (especially the first and third) would apply not only to working at EA nonprofits but also, e.g., tech startups or a political campaign as well.
This problem seems much more doable. I imagine many early-stage nonprofit CEOs would be willing to spend 5 hours chatting with top people who they made an offer to, though probably not early on in the process.
In general there’s a “vibe” of the comment that I somewhat disagree with, something in the general vein of “morality ought to be really convenient, and other people should figure that out.”
Hi Linch, we met and talked for awhile at the SF Picnic last year. This was my Reddit comment. I’ll reply here even though this is my first time interacting on this forum. I have lurked here for a long time but felt like the conversations were too time intensive to get involved. So, I’ll try to keep this brief.
Yes, this is true. While I didn’t say it in this comment, I do believe EA orgs have a competitive edge of also offering more-meaningfully-certain employment, which is new, and Big Tech doesn’t really have a way to counter that. Among the set of {cause, startup, politics}, FAANG compensation is effective at keeping us from leaving for these riskier things. Sometimes an exec will counter a senior engineer who is thinking of leaving with an offer to work on a project that is more values-aligned but, generally, Big Tech has a strategy of paying top-of-market and, if that fails, making it easy to come back if the outside gig fails[1].
Agree. It seems like there could be an EA-aligned startup that is cultivating talent connections and sharing that pool and facilitating those conversations among all EA orgs. Not just for software engineers but for all roles; HR is a big problem facing new companies—talent there is hard to find, too.
As a starting point, there could be some folks offering presentations to spread the expertise around. As an example, I did an hour-long training on Forming and training a distributed InfoSec team during the pandemic in November to the Infosec in EA Facebook group. There at topics like these that every EA org could benefit from.
Well, to be starkly transparent about my own biases and mental framing: the message that EA sends is that there are effective charities and, if one only gives away enough money as EtG, one can live a moral life. This is intoxicating and it’s a sanguine trap because of that: when I wrote my first annual check, the feels were real. And the feels keep coming, year after year.
An EA startup has to overcome the feels that EtG offers to attract top talent. (I say this to make the observation that this is a market reality; not to virtue signal.)
To close, allow me to state a straw-person risk analysis for a top-of-market tech employee (based on my own intuitions, not reproducible data):
Stay at FAANG
40-60% chance that a top performer will be able to EtG >$1M/yr during the final 10 years of their career.
>95% chance that the person will be able to EtG $>100k/yr during the final 10 years of their career.
~70% chance of retiring at 50. Then, the person can direct their attention to EA orgs for the last 10-20 working years of their career for free/little comp.
Join an early EA startup
~20% chance that the startup survives its first 5 years of existence and is effective, depending on cause area. People-problems are the reasons that startups fail, not bad ideas.
Depending on how early and in what capacity one contributes, it’s hard to guess at the possible impact one might have in the final moral calculus of this organization’s altruistic contributions, if successful, when measured against something like EtG donations to AMF. The wide range of outcomes could far outpace EtG, yes. And, importantly, enable other EtG-ers to find new places to sink altruistic capital. What are the odds? Unknown.
The leave-and return-in-2-years-on-failure path has lifetime earnings opportunity cost of ~$2M via lost equity and career trajectory.
Hi Jason,
Do you mind humoring a few follow up questions?
Firstly, it seems like the skills of a staff engineer at your current role might be different from what many new EA or AI safety orgs need. For example, you probably handle a lot of design and dependencies currently. Writing, communication is probably a lot of your value. In contrast, in a small, new organization, you probably need to knock out a lot of smaller systems quickly. Your skills and the edge you have might be different between these orgs.
Does the above sound accurate to you, or is it misleading?
Secondly, retirement security was an important point to you. I didn’t fully understand this. My guess is that in your personal case, the security you could provide for yourself and your partner might be large compared to what an EA org (or really most jobs) could easily provide. So my read was that you were talking about more junior engineers and SWE outside of FAANG.
Is this guess wrong? For example, in the US / Cali, maybe I’m really ignorant and you need a large 7 figure nest egg to be safe.
Finally, in terms of financial security, would income stability, such as guaranteeing transitional income if an organization needed to shutdown, substitute for very large income? The idea here is that everyone trusts you, and you were funded to move to successful organizations, so you didn’t have to stay on or bail out zombie organizations.
Could you write a little more here to make this more legible? Like, is there a book or blog post you can share?
To give context, it’s not clear to me what you mean by HR needs. Do you mean basic operational tasks involved in HR? I know there’s a recruiting org that might be pretty sophisticated, maybe that is what you mean? Maybe I’m really ignorant, but in many tech companies, modulo of the recruiters, it seemed that both talent attraction and team functionality is entirely up to management (e.g. the manager or skip of your “two pizza team”). HR was involved in only pretty fundamental processes, like scheduling interviews or paying out checks (and many of these were contracted out). I knew a few director/VP at a FAANG who said they didn’t understand HR and literally said they provided documentation for terminations.
To be clear, the above could be really dysfunctional/disrespectful and betray my ignorance.
Context/Subtext for my questions:
I thought your comment was more selfless than it seemed, because I didn’t think you were talking about yourself. I think you were talking about the choices other software engineers face.
Another point is that the skill match might suggest EtG, at least for your particular case.
Yes, I agree but with a huge caveat: every person will progress through various stages of competency during their career. While many early-stage folks could contribute just as well at an early stage EA startup (and should consider it), in the context of the 80k Hours article that I was replying to, we need to be transparent with folks about what a typical career path looks like and what tradeoffs there are to consider down each of the startup vs EtG paths. Here’s the typical career progression for software engineers (though it’s general enough to map onto other fields).
New grad/early stage: needing direction from others on what to work on, executing that work.
Leading self: proposing work in the context of larger goals and then executing that work
Leading a small team: proposing work and technical direction for a small team of engineers. Major design, some direct contributions, work-stream shepherding, mentorship.
Leading a large, ambiguous area/leading multiple teams: proposing strategic direction shifts, aligning team leaders, building consensus without authority, major design work, little direct contributions, mentorship + cultivation at scale.
Leading the entire technical direction for a business function: everything of the previous role, except heavily influencing all of the non-tech functions in the organization.
A business executive[1]
Regarding which level of progression that individuals might achieve by the end of their career, there’s a bell curve distribution around the 3rd step. Only a handful will ever reach the 6th step of being an executive[2]. FAANG pays somewhere between $750k-$1.5M for step 5, though, and—while still rarefied—it’s attainable for top talent, so a possible EtG goal to plan for.
All of this is a long-winded way of saying that CS folks who are about to graduate shouldn’t throw away a job offer from FAANG for an EA startup, out-of-hand, if they think that they have career luck in their favor. It would be a hard call. If I were 22 and about to graduate today[3], I would give an EA startup 3-5 years to be successful before I switched tactics and tried for a FAANG or other top-of-market option.
Right: speaking about general SWE population considering private-sector versus non-profits which tend to pay less and also tend to provide less benefits like retirement account funding.
In US/Cali Bay Area (where some EA startups are based), the median house price is $1.3M. So, for someone looking to put down roots in the Bay Area and retire within the same friend network close by, a nest egg of $2M isn’t an unreasonable guess; $3-4m if their spouse isn’t working and they start a family. If we expect EA-ers to come work for a startup in the Bay Area and then move to a lower cost of living place later, we should be transparent about that. (Or, we should be encouraging EA orgs to go remote-first to unlock paying top-of-market rates in rural areas.)
Yes, that would make the job offers of EAs more attractive to new-career and mid-career folks. It’s probably also applicable to all other roles that an EA would hire for.
It’s common it tech to hear the sentiment of your social network that HR provides no value so I’m not surprised to see this. In Silicon Valley, there’s similar discounting of the value provided by folks in operations, support, logistics and finance.
A note on horizontal organization roles: there are types of roles that apply horizontal, cultural influence. For example, a wise person once said, “If you want to understand why an organization behaves the way that it does, look at the incentives of the people in that organization.”
I point to HR, specifically, because it’s an area where I’ve seen the most struggles in small-stage startups precisely because it is a horizontal force multiplier. Here’s some values that a functioning HR organization provides to a small stage startup:
A meritocratic system of promotion/career advancement that’s seen as fair by the employees. This includes transparency of the expected roles and responsibilities at each stage of career progression. Of course, this is includes some objective criteria for deciding to fire someone, and all of the legal implications thereof (as mentioned above), but that’s not the most important part. Retention is partially a function of aligning the hedonic treadmill with real career progress possibilities.
Setting norms on how individuals interact and, theoretically, backstopping those norms with enforcement. For example, an org might say that aggressive behavior in meetings and emails is not tolerated. This is just a theoretical rule unless org leaders actually back up those words with actions through the promotion process and, in extreme cases, HR-backed disciplinary action. It’s also the function of HR to repeat the company’s behavioral expectations, periodically.
Ensuring fair hiring practices is non-trivial. It’s common in startups to hand-waive over this problem. But actually objectively evaluating candidates and ensuring that bias doesn’t creep in and that pay is equal among all similar roles and levels is hard. Radical transparency can help here but it doesn’t just magically fix the problem.
Setting organizational goals against which the org is measured is sometimes seen as operations or Product Management but there’s an HR role there too: leaders of those sub-orgs that set those goals need to be held accountable and any exec/leadership compensation should be tied to business outcomes in a way that lower-level employees do not face. And all of those company benchmarks and the feedback cycle need to be done in front of the whole employee population, quarterly.
Assessing employee satisfaction and collecting feedback anonymously on an ongoing basis. This can be as simple as an anonymous Google Form that’s open for two weeks once a year. But, actually collating the data, slicing it by org, trending over time, and proposing cultural changes to address employee feed back is hard.
Benefits benefits benefits. This is a constantly evolving space. To some extent, this can be outsourced, but there should be someone on staff continually evaluating the changing landscape of offerings and competitor offerings and continually updating the employees about those changes and acting as a partner to fix problems when they come up.
I could go on but these are the ones that came to mind while I was writing this, and I think that I’ve exceeded the amount of time that I intended to spend on this. 😉
I acknowledge that this assumes a fully meritocratic progression; there are indeed many reasons that individuals might be given these roles without being qualified.
I recognize that founders of startups are not necessarily destined to lead 1,000 employee organizations but that they do need some mix of all of the skills in these stacks. And this is often why startups fail.
Full disclosure: I do not have a degree and am an anomaly. So, I can’t really speak with authenticity on this hypothetical.
Thanks so much! I thought this was incredibly informative and in-depth. There are some really valuable insights here.
Like you, I also think it’s not-obvious that someone of Jason’s skillsets should be doing something other than earning-to-give. As a practical matter, most EA employers aren’t willing to pay >1M/year, except maybe in a few niche situations in ML/AI Safety or very successful EA fintech startups.
(I do vaguely think that if you just naively crunch the numbers about how great SWEs can contribute now vs. Open Phil’s last dollar, there probably exists opportunities somewhat above this bar however, including outside of AI Safety and weird crypto stuff).
There’s also a strong argument that someone like Jason has a clear comparative advantage relative to the rest of the movement in doing E2G stuff in BigTech, and maybe as a community this coordination problem is better solved with less career shuffles.
Ozzie’s post on opportunity costs of technical talent is related.
Hi Jason,
Thank you so much for your detailed engagement with my somewhat blunt + rude comment, and for the density of your honest comments! I will try to make a more substantive comment later, but I want to say I really appreciate your comments and for your hard work earning-to-give.
I do remember meeting you last summer and I thought our conversation was quite good.
A moral saint might suffer arbitrary inconveniences to have an impact. But most real people won’t.
A better framing is that Jason is a “customer” of the EA talent pipeline, and before telling him that his desires are a “bug”, we should try really hard to give him more of what he wants!
Thanks, this is a good reframing. :)
Two other paths that might be unusually impactful for EA software engineers:
Joining a for-profit startup with a) EA founders, b) a small engineering team, and c) where engineering appears critical to the organization’s success.
The “EA founder” part of this isn’t strictly necessary, but this makes it much more likely that a great software engineer can create a lot of value through generating future donations (if we expect much of the equity to be captured by founders and investors).
Note that because EA isn’t very funding-constrained, your bar for this might be very high, like ~$millions/year in expectation per new engineer.
Personally, I think FTX, Alameda Research, and Lantern Ventures are over this bar.
(COI disclaimer: I have multiple personal financial ties here, including via the FTX EA fellowship)
I would not be surprised if several others exist in the space that I’m not personally aware of.
Ladder climbing within AI labs at large tech companies.
I think it’s plausibly really good to have EA engineers in positions that’d be close to the AI leadership at top tech companies in 5-15 years.
Note that I think most software folks who are in a position to do so should not do this, partially for reasons outlined here. The people who are best at it are a) comparatively good at climbing prestige ladders and b) agenty enough to spot great opportunities for impact and c) act upon them.
This also has obvious downside risks (e.g. via speeding up AI timelines slightly).
And also I’m more than a bit worried about motivated cognition causing people to overrate the value (and/or be wrong about the sign) of the direct impact of working technically interesting and cushy BigTech jobs.
Still, right now this appears to be a very underrated path, and the case for it at least naively seems reasonable.
Two things I’d like to note from my corner of EA:
I would add just how underrepresented frontend / full-stack still-sets are in EA relative to what I was used to in Silicon Valley. If you have those skill sets, you should realize how valuable that is.
CEA has an expression of interest open for a full stack engineer.
TL;DR: Even though 80k talk about personal fit: In practice EA software developers neglect their personal fit within the domain of software, which is a concept I recommend adding.
Example very common things that happen:
People pick a domain (for example, Software Engineering vs ML) based ONLY on the domain’s expected impact or on what this person already has experience with, with no consideration of how fun/exciting this domain is for them. I think this is bad.
Someone doesn’t enjoy their job, but this is due to “having a bad boss” or “having nobody to learn from”, and not due to having a bad personal fit to software engineering in general
You said “gain a really deep understanding of the basics”—I wouldn’t put this in an EA Software Career guide without serious disclaimers
TL;DR: I think EAs spend too much time “learning the basics” rather than “doing something productive and scrappy”, and it is a bad idea to push them more towards the “learning the basics” side.
I have a ton to say about this and I’ve deleted several too-long-drafts already, but feel free to ask/disagree of course.
A similar example: I wouldn’t tell the EA community that they’ve got to write even longer documents. ;)
Just as I’m trying to tell myself to not make this comment even longer. This is really hard. Ok sending!
I found this a bit hard to follow, especially given the focus in the previous paragraphs on safety work specifically. It reads to me like it’s making the counterintuitive claim that “safety” work is actually where much of the danger lies. Is that intended?
That’s not the intention, thanks for pointing this out!
To clarify, by “route”, I mean gaining experience in this space through working on engineering roles directly related to AI. Where those roles are not specifically working on safety, it’s important to try to consider any downside risk that could result from advancing general AI capabilities (this in general will vary a lot across roles and can be very difficult to estimate).
Is “safe” here meant in the sense of “not accelerating risks from AI,” or in the sense of “difficult to steal” (i.e. secure)?
A bit of both—but you’re right, I primarily meant “secure” (as I expect this is where engineers have something specific to contribute).
I can think of a few other areas of direct impact which could particularly benefit from talented software engineers:
Improving climate models is a potential route for high impact on climate change, there are computational modelling initiatives such as the Climate Modeling Alliance and startups such as Cervest. It would also be valuable to contribute to open source computational tools such as the Julia programming language and certain Python libraries etc.
There is also the area of computer simulations for organisational / government decision making, such as Improbable Defence (disclosure: I am a former employee and current shareholder), Simudyne and Hash.ai. I’ve heard anecdotally that a few employees of Hash.ai are sympathetic to EA, but I don’t have first hand evidence of this.
More broadly there are many areas of academic research, not just AI safety, which could benefit from more research software engineers. The Society of Research Software Engineering aims to provide a community for research engineers and to make this a more established career path. This type of work in academia tends to pay significantly lower than private sector software salaries, so this is worse for ETG, but on the flip side this is an argument for it being a relatively neglected opportunity.