Or perhaps you could make up an example question that is has a similar difficulty to whatever you ask?
Another example: I really like how Anthropic said “If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today”
Yes, I’m fairly sure this is what Ben meant. I’ve looked over some of the earlier versions of questions Ben was interested in, and I can confirm that the questions are harder than the average “Hard” questions on leetcode, which in turn is harder than typical interview questions at large tech companies. However, for people intimidated by this, I think there are ~3 things that claw back the difficulty somewhat:
The typical expected time to complete a BigTech interview question is about 30 minutes in a 45 minute interview (with ~15 minutes extra time to explain your solutions, ask questions of the interviewer, etc), iirc CEA gives you ~2-3h to solve their questions.
Most BigTech interviews are closed book on a whiteboard, iiuc CEA’s interviews are open book and you can use your preferred IDE.
(Only relevant for people familiar with leetcode questions). Leetcode questions have, for want of a better word, “idioms,” ie, common patterns that crop up fairly frequently across leetcode problems, including Leetcode Hard problems (e.g., sliding window, square root decomposition) but almost never real programming jobs. This makes Leetcode Hard problems easier for people who’ve already solved other LC problems, in ways that don’t generalize as well to solving algorithm puzzles that don’t come from the same generator.
One reason I asked about the difficulty of Ben’s problems, is that I want to calibrate myself to the EA tech talent pool when giving comments.
It would be good to know if many people here, with little practice, can solve unseen leetcode hard problems in a BigTech interview format (where you have to simultaneously create a verbal narrative throughout the interview).
If this is true, I will stop giving “advice” on any tech related post as this is out of my reference class.
Charles, TL;DR: I would not stop giving tech advice just because you can’t solve hard leetcode puzzels
Longer:
If this [solveing unseen leetcode hard problems in a BigTech interview format] is true, I will stop giving “advice” on any tech related post as this is out of my reference class.
If it influences your opinion:
I think I’m very good at leetcode puzzels
I give tech advice
I think those “skills” are almost totally unrelated.
It is very rare for me to actually teach someone how to solve leetcode
It is more common for me to just point out “cracking the coding interview” as a legit way to prepare for big-tech interviews, but knowing this is unrelated to my ability to solving the questions myself. (I know it because a Google recruiter recommended that book for me. I can elaborate)
I guess my intuition is that the correlation between being good at giving career advice for techies and being able to solve algorithms puzzles is only moderate, and very little of that is causal.
It would be good to know if many people here, with little practice, can solve unseen leetcode hard problems in a BigTech interview format (where you have to simultaneously create a verbal narrative throughout the interview).
Fwiw I suspect this is pretty uncommon among EAs, including EAs in tech. I was unusually good at leetcode-style problems (both in general and especially relative to my ability to actually do useful work in programming), and I suspect my hit rate for novel LC Hard problems in 30 minutes was less than 50%*.
*I’ve regressed a lot in my algorithms puzzles ability, I’m not even confident I can consistently do LC Easys now.
As Linch says, we don’t use the interview format you describe. It’s all done asynchronously, and with less time pressure. I’m not well calibrated on how well the average EA does under the whiteboard format, but I would say that in our format:
~50% of applicants pass our 30 minute screen
5-10% pass our 2.5 hour in-depth task
~5% get an offer
Note that the people who apply for our jobs are probably not representative of who you would speak to on the Forum, so I’m not sure how helpful these statistics actually are.
Lorenzo was correct, I was referring to the classification of “hard” on leetcode.
A rough equivalent to our trial task is: implement a piece table in 2.5 hours from a scaffold with some basic tests, using Wikipedia/the Internet but you aren’t allowed to verbatim copy code. It’s done as a take-home, so you don’t have to think out loud or write on a whiteboard or whatever.
I spent a few minutes trying to think of an analog to the Anthropic statement; I think if you are able to write a substantial pull request to a popular npm package (or similar web-based framework) I’m probably interested in talking. (Though I want to reemphasize that people should value my time at ~0, and therefore apply even if they think they are unlikely to get an offer.)
Can...can you share one that you think is that hard?
+1 to this question.
Do you mean “hard” in a specific website?
Or perhaps you could make up an example question that is has a similar difficulty to whatever you ask?
Another example: I really like how Anthropic said “If you think you could write a substantial pull request for a major machine learning library, then major AI safety labs want to interview you today”
Leetcode has a “hard” tag on questions, I assume Ben meant these https://leetcode.com/problemset/all/?difficulty=HARD
I agree there is a huge variance, I have solved 19 last year and remember some being much much harder than others
Yes, I’m fairly sure this is what Ben meant. I’ve looked over some of the earlier versions of questions Ben was interested in, and I can confirm that the questions are harder than the average “Hard” questions on leetcode, which in turn is harder than typical interview questions at large tech companies. However, for people intimidated by this, I think there are ~3 things that claw back the difficulty somewhat:
The typical expected time to complete a BigTech interview question is about 30 minutes in a 45 minute interview (with ~15 minutes extra time to explain your solutions, ask questions of the interviewer, etc), iirc CEA gives you ~2-3h to solve their questions.
Most BigTech interviews are closed book on a whiteboard, iiuc CEA’s interviews are open book and you can use your preferred IDE.
(Only relevant for people familiar with leetcode questions). Leetcode questions have, for want of a better word, “idioms,” ie, common patterns that crop up fairly frequently across leetcode problems, including Leetcode Hard problems (e.g., sliding window, square root decomposition) but almost never real programming jobs. This makes Leetcode Hard problems easier for people who’ve already solved other LC problems, in ways that don’t generalize as well to solving algorithm puzzles that don’t come from the same generator.
One reason I asked about the difficulty of Ben’s problems, is that I want to calibrate myself to the EA tech talent pool when giving comments.
It would be good to know if many people here, with little practice, can solve unseen leetcode hard problems in a BigTech interview format (where you have to simultaneously create a verbal narrative throughout the interview).
If this is true, I will stop giving “advice” on any tech related post as this is out of my reference class.
Charles,
TL;DR: I would not stop giving tech advice just because you can’t solve hard leetcode puzzels
Longer:
If it influences your opinion:
I think I’m very good at leetcode puzzels
I give tech advice
I think those “skills” are almost totally unrelated.
It is very rare for me to actually teach someone how to solve leetcode
It is more common for me to just point out “cracking the coding interview” as a legit way to prepare for big-tech interviews, but knowing this is unrelated to my ability to solving the questions myself. (I know it because a Google recruiter recommended that book for me. I can elaborate)
Thank you for the thoughtful reply Yonatan!
I guess my intuition is that the correlation between being good at giving career advice for techies and being able to solve algorithms puzzles is only moderate, and very little of that is causal.
Fwiw I suspect this is pretty uncommon among EAs, including EAs in tech. I was unusually good at leetcode-style problems (both in general and especially relative to my ability to actually do useful work in programming), and I suspect my hit rate for novel LC Hard problems in 30 minutes was less than 50%*.
*I’ve regressed a lot in my algorithms puzzles ability, I’m not even confident I can consistently do LC Easys now.
As Linch says, we don’t use the interview format you describe. It’s all done asynchronously, and with less time pressure. I’m not well calibrated on how well the average EA does under the whiteboard format, but I would say that in our format:
~50% of applicants pass our 30 minute screen
5-10% pass our 2.5 hour in-depth task
~5% get an offer
Note that the people who apply for our jobs are probably not representative of who you would speak to on the Forum, so I’m not sure how helpful these statistics actually are.
Lorenzo was correct, I was referring to the classification of “hard” on leetcode.
A rough equivalent to our trial task is: implement a piece table in 2.5 hours from a scaffold with some basic tests, using Wikipedia/the Internet but you aren’t allowed to verbatim copy code. It’s done as a take-home, so you don’t have to think out loud or write on a whiteboard or whatever.
I spent a few minutes trying to think of an analog to the Anthropic statement; I think if you are able to write a substantial pull request to a popular npm package (or similar web-based framework) I’m probably interested in talking. (Though I want to reemphasize that people should value my time at ~0, and therefore apply even if they think they are unlikely to get an offer.)