This is from my blog Said Twice where I write advice that I’ve said twice.
Do you want to find something to spend your professional development time on or improve your chances of getting a job? Look for cheap tests that tell you what it’s like doing the skill you’re practicing and expensive signals that demonstrate you have that skill.
A cheap test is something that doesn’t cost too much in time or money and provides information about what it’s like doing something and whether you enjoy doing it.
An expensive signal is a piece of strong evidence to your manager or a future employer that you have the skill or experience that they’re looking for. A cheap signal is saying “I’m good at X”, while an expensive signal is saying “Here’s a concrete example of me doing X and here’s enough information for you to validate whether I’m good at it”.
Cheap tests and expensive signals is a type of heuristic I use to help me identify areas to improve or focus on and methods to do so. Here are the three steps I use:
Step 1: Decide on an area or skill to focus on
Here are some prompts for identifying potential candidates for areas to focus on:
What feedback have you received recently? What did your last performance review say?
If you don’t have this already, consider asking your colleagues, manager, or friends for ideas of areas to improve on.
What is something you think you’re good at, but haven’t been able to demonstrate to others that you can do?
What is a recurring skill in the job openings you’re interested in?
Which of your current areas of responsibilities do you have the most potential to grow in?
What type of work do you want to be able to do more of in your current role or in a future role?
What’s a type of task you struggled with in the last work test you did as part of a hiring process?
Step 2: Generate cheap tests
Here are some examples:
One-off project for an organisation
One-off project at your organisation
Personal one-off project
Online course that’s practical or has a lot of exercises you can go through
Applying to a job and getting to the work test stage
Internship or volunteer role
Conversations with people in the positions you’re curious about
Step 3: Generate expensive signals
Here are some examples:
Good stories in a cover letter or interview
Responses to work tests that show you’ve done the type of work before
Connections to and, better yet, endorsements from relevant people in the field
Strong experiences on your CV
Concrete products that you can easily share that demonstrate your skill
Examples of cheap tests and expensive signals
Let organisations know that you’re available to help with one-off projects, listing the type of areas you can help with.
Worst case, the organisation knows you’re after this type of work, and may mention your name if some other organisation needs help.
Best case, you get some work experience, connections and ideally endorsements from people in the field, and a good story to tell in an interview.
Personal example: I had additional time at Forethought and was able to take on a project of 50% for a couple months. I let a few organisations know that I could potentially have time to help out with operations, recruiting, and project management. This led to me supporting 80,000 Hours’ CEO search in late 2023. It was a 2-3 month project where I got experience with executive hiring (which I had little experience in) and hiring in general (which I have a fair amount of experience in).
Spend your professional development time producing an end product.
You want to be able to show the fact that you have a skill in a way that your manager or future employer can validate the quality of themselves.
This should be something you could provide as a work sample in a hiring process or something you can link to in a performance review when listing the type of work you have done and can do.
Personal example: I wanted to improve my ability to provide advice on external communications. To do so, I asked a few communications professionals for tips and read up on best practices. Based on that, I drafted a list of organisation-wide heuristics for external communications, which I asked our executive to review. Once we agreed on a set of heuristics, I turned that into a communications checklist we could use when considering whether to e.g. respond to a request or take advantage of an opportunity. I’ve later shared the list of heuristics and checklist as a work sample when being assessed in a hiring process.
Take a course or read a book in the area you want to improve on, and then practice those skills.
A cheap signal is having read a book on something, an expensive signal is deliberately practicing what you’ve learned.
Personal example: When I was Executive Director of EA Norway I wanted to improve my management and coaching skills. It was my first time having employees and we also wanted to start providing career coaching for our members. My Assistant Director and I read the books Quiet Leadership and Messages, both of which have concrete exercises we could do. We did about one exercise per week, which often included practicing a technique in e.g. a career coaching session. This was hugely useful, and still influences my management and coaching methods today. I’ve both used this as an example of my management experience in a job interview, and I’m confident it’s improved my performance on related work tests.
I like this advice a lot but want to add a quick asterisk when transitioning to a new field.
It’s really really hard to tell what an expensive signal is without feedback. If you’re experienced in a field or you hang out with folks who work in a field, then you’ve probably internalized what counts as an “impressive project” to some degree.
In policy land, this cashes out as advice to take a job you don’t want in the organization you do want. Because that’s how you’ll learn what’s valuable and what’s not. Or taking low paid internships and skilled volunteering roles. Or dropping a lot of money to attend a conference for your target field.
It’s also really hard to know the steps to executing the “impressive project” (which is why the signal is so expensive!). With internships and skilled volunteering, you’ll get supervision. And even a light touch can prevent you from investing a ton of time in something that doesn’t matter. Or get reassurance that task X really does take everyone a long time so don’t feel bad about the time sink.
But with grants or independent work, you’ll have to seek out the feedback, brief them on project and hope you’ve given enough context for useful feedback, and also hope you picked someone who knows your area well enough. (I haven’t had success here and I’m not sure how realistic it is for most people.)
Work tests are awesome here since they’re mini-projects. But feedback is often noisy and hard to interpret since there aren’t good incentives for orgs to specialize in concrete feedback. I’ve interpreted this feedback wrong in both directions (first being too optimistic about a generic but lengthy “there were many strong candidates” and then too pessimistic about the terse but personalized rejections encouraging me to still consider research as a career)
The point I’m trying to make is that the idea of “cheap tests, expensive signal” is probably a lot easier for mid-career folks to apply independently. But for people without any experience, the advice depends on whether you can get supervision from an organization. Without that, it may be better to just “get your foot in the door” in any way possible. Maybe a “good enough cool sounding project” helps to demonstrate interest, but it’s tough for people to perform at 1-year of experience level before they have that 1 year of experience.
I imagine you can get a lot of the value here more cheaply by reaching out to people in the field and asking them a bunch of questions about what signals do and do not impress them?
Doing internships etc is valuable to get the supervision to DO the impressive projects, of course.
EDIT: Speaking as someone who does hiring of interpretability researchers, I think there’s a bunch of signals I look for and ones I don’t care about, and sometimes people new to the field have very inaccurate guesses here
Ooh good idea. I should do more of that.
I do think this can run into Goodhart’s Law problems. For example, in the early 2010s, back when being a self-taught software engineer was much more doable, and it was a strong sign when someone had a GitHub profile with some side projects with a few months of work behind each of them. GitHub profile correlated with a lot of other desirable things. But then everyone got that advice (including me) and that signal got rapidly discounted.
So I guess I’d qualify that with: press really hard on why the signal is impressive and also ask people explicitly if they agree with the signals you heard from others (ex. I heard from people in field that signal X is good / bad, do you agree with that?)
I think this is a valid long term concern but takes at least a few months to properly propagate—if someone qualified tells you that when hiring they look at a github profile, that’s probably pretty good for the duration of your job search
This advice sounds right to me if you already have the signal in hand and deciding whether to job search.
But if you’re don’t have the signal, then you need to spend time getting it. And then I think the advice hinges on how long it takes to get the signal. Short time-capped projects are great (like OP’s support on 80,000 hours CEO search). But for learning and then demonstrating new skills on your own, it’s not always clear how long you’ll need.