Thanks for doing this coaching and sharing the update! Some answers from myself; I’m not sure how much my coworkers at CEA agree with me, so these are intended as a personal opinion rather than an official stance:
A LOT of people are asking themselves if they should work for you or earn to give.
I spent a few minutes trying to think of engineers that I personally know for whom it’s so obvious they should earn to give that they shouldn’t even apply to EA organizations. I was only able to think of a handful, and they are all on the founding teams of $100M+ companies.
I advise myself (CTO/cofounder of a reasonably successful company) and engineers at top quant trading firms to strongly consider direct work. I can think of only one person who had an engineering offer from an EA organization I respected but I thought they should do earning to give instead.
A more common scenario is “earning to learn” – EA org’s don’t always have junior positions or provide mentorship, and many (probably most) junior EA engineers should be skilling up at for-profit organizations. Again here though, I think these people should at least try to apply for positions at EA organizations, and if they get rejected ask the hiring manager where they should skill up, then try to gain those skills elsewhere.
(I acknowledge that applying to a bunch of EA organizations and getting rejected from all of them really sucks; the above advice is what I think an impact-maximizing robot would do, but there might be psychological reasons why the people you are advising should not do it. To the extent that you can coach people to feel okay with rejection though, that seems helpful.)
If there are specific ways people can prepare for an interview with you, it would be useful to publicly say so
For CEA:
The initial stage of the interview is leetcode-style algorithms questions. Typescript/JavaScript preferred, Python second choice. I haven’t given this a lot of thought, but the next time we hire for an engineer I might just outsource to Triplebyte.
1.1. I’m having difficulty calibrating exactly how hard our questions are. I think they are harder than “hard” leetcode questions. I can try to think of a better calibration point, if this would be useful to you.
The last stage is the person actually working with us for a day or two. We have an open source code base, so you can see verbatim what this looks like; e.g. this is what the most recent engineer we hired did during her interview.
Regarding whether people should work somewhere for some amount of time before applying to CEA: the email I send to people with limited experience is basically “to be respectful of your time, I’ll tell you that I think you probably won’t be a fit here, but it’s really hard to judge someone from their resume so if you want to try our trial task you should go for it.” Most people bail after that email without trying the task, and no one I’ve sent that email to has successfully completed it, but I’m sure there are people with zero work experience who would be capable of doing it.
Or perhaps you could make up an example question that is has a similar difficulty to whatever you ask?
Another example: I really like how Anthropic said “If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today”
Yes, I’m fairly sure this is what Ben meant. I’ve looked over some of the earlier versions of questions Ben was interested in, and I can confirm that the questions are harder than the average “Hard” questions on leetcode, which in turn is harder than typical interview questions at large tech companies. However, for people intimidated by this, I think there are ~3 things that claw back the difficulty somewhat:
The typical expected time to complete a BigTech interview question is about 30 minutes in a 45 minute interview (with ~15 minutes extra time to explain your solutions, ask questions of the interviewer, etc), iirc CEA gives you ~2-3h to solve their questions.
Most BigTech interviews are closed book on a whiteboard, iiuc CEA’s interviews are open book and you can use your preferred IDE.
(Only relevant for people familiar with leetcode questions). Leetcode questions have, for want of a better word, “idioms,” ie, common patterns that crop up fairly frequently across leetcode problems, including Leetcode Hard problems (e.g., sliding window, square root decomposition) but almost never real programming jobs. This makes Leetcode Hard problems easier for people who’ve already solved other LC problems, in ways that don’t generalize as well to solving algorithm puzzles that don’t come from the same generator.
One reason I asked about the difficulty of Ben’s problems, is that I want to calibrate myself to the EA tech talent pool when giving comments.
It would be good to know if many people here, with little practice, can solve unseen leetcode hard problems in a BigTech interview format (where you have to simultaneously create a verbal narrative throughout the interview).
If this is true, I will stop giving “advice” on any tech related post as this is out of my reference class.
Charles, TL;DR: I would not stop giving tech advice just because you can’t solve hard leetcode puzzels
Longer:
If this [solveing unseen leetcode hard problems in a BigTech interview format] is true, I will stop giving “advice” on any tech related post as this is out of my reference class.
If it influences your opinion:
I think I’m very good at leetcode puzzels
I give tech advice
I think those “skills” are almost totally unrelated.
It is very rare for me to actually teach someone how to solve leetcode
It is more common for me to just point out “cracking the coding interview” as a legit way to prepare for big-tech interviews, but knowing this is unrelated to my ability to solving the questions myself. (I know it because a Google recruiter recommended that book for me. I can elaborate)
I guess my intuition is that the correlation between being good at giving career advice for techies and being able to solve algorithms puzzles is only moderate, and very little of that is causal.
It would be good to know if many people here, with little practice, can solve unseen leetcode hard problems in a BigTech interview format (where you have to simultaneously create a verbal narrative throughout the interview).
Fwiw I suspect this is pretty uncommon among EAs, including EAs in tech. I was unusually good at leetcode-style problems (both in general and especially relative to my ability to actually do useful work in programming), and I suspect my hit rate for novel LC Hard problems in 30 minutes was less than 50%*.
*I’ve regressed a lot in my algorithms puzzles ability, I’m not even confident I can consistently do LC Easys now.
As Linch says, we don’t use the interview format you describe. It’s all done asynchronously, and with less time pressure. I’m not well calibrated on how well the average EA does under the whiteboard format, but I would say that in our format:
~50% of applicants pass our 30 minute screen
5-10% pass our 2.5 hour in-depth task
~5% get an offer
Note that the people who apply for our jobs are probably not representative of who you would speak to on the Forum, so I’m not sure how helpful these statistics actually are.
Lorenzo was correct, I was referring to the classification of “hard” on leetcode.
A rough equivalent to our trial task is: implement a piece table in 2.5 hours from a scaffold with some basic tests, using Wikipedia/the Internet but you aren’t allowed to verbatim copy code. It’s done as a take-home, so you don’t have to think out loud or write on a whiteboard or whatever.
I spent a few minutes trying to think of an analog to the Anthropic statement; I think if you are able to write a substantial pull request to a popular npm package (or similar web-based framework) I’m probably interested in talking. (Though I want to reemphasize that people should value my time at ~0, and therefore apply even if they think they are unlikely to get an offer.)
Thanks for doing this coaching and sharing the update! Some answers from myself; I’m not sure how much my coworkers at CEA agree with me, so these are intended as a personal opinion rather than an official stance:
I spent a few minutes trying to think of engineers that I personally know for whom it’s so obvious they should earn to give that they shouldn’t even apply to EA organizations. I was only able to think of a handful, and they are all on the founding teams of $100M+ companies.
I advise myself (CTO/cofounder of a reasonably successful company) and engineers at top quant trading firms to strongly consider direct work. I can think of only one person who had an engineering offer from an EA organization I respected but I thought they should do earning to give instead.
A more common scenario is “earning to learn” – EA org’s don’t always have junior positions or provide mentorship, and many (probably most) junior EA engineers should be skilling up at for-profit organizations. Again here though, I think these people should at least try to apply for positions at EA organizations, and if they get rejected ask the hiring manager where they should skill up, then try to gain those skills elsewhere.
(I acknowledge that applying to a bunch of EA organizations and getting rejected from all of them really sucks; the above advice is what I think an impact-maximizing robot would do, but there might be psychological reasons why the people you are advising should not do it. To the extent that you can coach people to feel okay with rejection though, that seems helpful.)
For CEA:
The initial stage of the interview is leetcode-style algorithms questions. Typescript/JavaScript preferred, Python second choice. I haven’t given this a lot of thought, but the next time we hire for an engineer I might just outsource to Triplebyte. 1.1. I’m having difficulty calibrating exactly how hard our questions are. I think they are harder than “hard” leetcode questions. I can try to think of a better calibration point, if this would be useful to you.
The last stage is the person actually working with us for a day or two. We have an open source code base, so you can see verbatim what this looks like; e.g. this is what the most recent engineer we hired did during her interview.
Regarding whether people should work somewhere for some amount of time before applying to CEA: the email I send to people with limited experience is basically “to be respectful of your time, I’ll tell you that I think you probably won’t be a fit here, but it’s really hard to judge someone from their resume so if you want to try our trial task you should go for it.” Most people bail after that email without trying the task, and no one I’ve sent that email to has successfully completed it, but I’m sure there are people with zero work experience who would be capable of doing it.
Can...can you share one that you think is that hard?
+1 to this question.
Do you mean “hard” in a specific website?
Or perhaps you could make up an example question that is has a similar difficulty to whatever you ask?
Another example: I really like how Anthropic said “If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today”
Leetcode has a “hard” tag on questions, I assume Ben meant these https://leetcode.com/problemset/all/?difficulty=HARD
I agree there is a huge variance, I have solved 19 last year and remember some being much much harder than others
Yes, I’m fairly sure this is what Ben meant. I’ve looked over some of the earlier versions of questions Ben was interested in, and I can confirm that the questions are harder than the average “Hard” questions on leetcode, which in turn is harder than typical interview questions at large tech companies. However, for people intimidated by this, I think there are ~3 things that claw back the difficulty somewhat:
The typical expected time to complete a BigTech interview question is about 30 minutes in a 45 minute interview (with ~15 minutes extra time to explain your solutions, ask questions of the interviewer, etc), iirc CEA gives you ~2-3h to solve their questions.
Most BigTech interviews are closed book on a whiteboard, iiuc CEA’s interviews are open book and you can use your preferred IDE.
(Only relevant for people familiar with leetcode questions). Leetcode questions have, for want of a better word, “idioms,” ie, common patterns that crop up fairly frequently across leetcode problems, including Leetcode Hard problems (e.g., sliding window, square root decomposition) but almost never real programming jobs. This makes Leetcode Hard problems easier for people who’ve already solved other LC problems, in ways that don’t generalize as well to solving algorithm puzzles that don’t come from the same generator.
One reason I asked about the difficulty of Ben’s problems, is that I want to calibrate myself to the EA tech talent pool when giving comments.
It would be good to know if many people here, with little practice, can solve unseen leetcode hard problems in a BigTech interview format (where you have to simultaneously create a verbal narrative throughout the interview).
If this is true, I will stop giving “advice” on any tech related post as this is out of my reference class.
Charles,
TL;DR: I would not stop giving tech advice just because you can’t solve hard leetcode puzzels
Longer:
If it influences your opinion:
I think I’m very good at leetcode puzzels
I give tech advice
I think those “skills” are almost totally unrelated.
It is very rare for me to actually teach someone how to solve leetcode
It is more common for me to just point out “cracking the coding interview” as a legit way to prepare for big-tech interviews, but knowing this is unrelated to my ability to solving the questions myself. (I know it because a Google recruiter recommended that book for me. I can elaborate)
Thank you for the thoughtful reply Yonatan!
I guess my intuition is that the correlation between being good at giving career advice for techies and being able to solve algorithms puzzles is only moderate, and very little of that is causal.
Fwiw I suspect this is pretty uncommon among EAs, including EAs in tech. I was unusually good at leetcode-style problems (both in general and especially relative to my ability to actually do useful work in programming), and I suspect my hit rate for novel LC Hard problems in 30 minutes was less than 50%*.
*I’ve regressed a lot in my algorithms puzzles ability, I’m not even confident I can consistently do LC Easys now.
As Linch says, we don’t use the interview format you describe. It’s all done asynchronously, and with less time pressure. I’m not well calibrated on how well the average EA does under the whiteboard format, but I would say that in our format:
~50% of applicants pass our 30 minute screen
5-10% pass our 2.5 hour in-depth task
~5% get an offer
Note that the people who apply for our jobs are probably not representative of who you would speak to on the Forum, so I’m not sure how helpful these statistics actually are.
Lorenzo was correct, I was referring to the classification of “hard” on leetcode.
A rough equivalent to our trial task is: implement a piece table in 2.5 hours from a scaffold with some basic tests, using Wikipedia/the Internet but you aren’t allowed to verbatim copy code. It’s done as a take-home, so you don’t have to think out loud or write on a whiteboard or whatever.
I spent a few minutes trying to think of an analog to the Anthropic statement; I think if you are able to write a substantial pull request to a popular npm package (or similar web-based framework) I’m probably interested in talking. (Though I want to reemphasize that people should value my time at ~0, and therefore apply even if they think they are unlikely to get an offer.)