Hi! Congratulations on the growth. A few questions from a fairly interested outside observer:
1. How do you think about counterfactual impact? It seems quite difficult to separate outcomes that would likely have happened anyway from outcomes meaningfully attributable to your interventions, particularly in a tough market. I don’t expect that’s fully solvable, but I’m curious about what methods or heuristics you use.
2. What do you see as the main differences between Probably Good and other career advisory organisations/services such as 80,000 Hours or High Impact Professionals? Where do you think your comparative advantage is emerging?
3. You mention an extensive search before selecting an internal hire for the ED role. Roughly how many applicants were involved, and how much staff time did the process consume overall? I’m generally sympathetic to internal hires/closed rounds in most cases because of the potential time/cost savings, but I’d be interested to better understand the level of those potential savings in practice.
Thanks for sharing, I’d definitely be interested in reading a longer retrospective at some point.
We try to assess this type of counterfactual causal attribution through several approaches, including asking people how likely they were to make career changes without our services, looking at other information they provide (like qualitative notes they share), and looking at other information we have on them (like which of our services they used). This is very much a work in progress though.
Mainly, being broader in who we can help and with what, in terms of causes we touch on, audiences we engage, and services we provide. As one example, this sometimes involves doing 1:1 advising with people who are interested in career impact and want a broad perspective across a range of causes, including ones that are underrepresented by other orgs (though we might then refer them to a specialist org). As another example, this involves aiming to offer a broader range of impactful roles on our job board than other organizations might.
(Answering this question from a generalizable hiring perspective, which seems more helpful and appropriate here, though note there’s significant variances across orgs and roles.) While internal rounds sometimes make sense, that’s not the case for a tiny org with only 1-2 potential hires. Closed rounds can also make sense sometimes, but only when you’re confident you’ll get the right candidate pool from this, given how important and hard it is to get hiring right (particularly for a small org, and even more so for a leadership role). The time savings aren’t necessarily great either, since many hiring costs are fixed (e.g., the time it takes to create a job description), and many of the seemingly marginal costs are actually fixed too (e.g., the time it takes to do interviews, if you have a cap for the number of interviews you can do).
Hi! Congratulations on the growth. A few questions from a fairly interested outside observer:
1. How do you think about counterfactual impact? It seems quite difficult to separate outcomes that would likely have happened anyway from outcomes meaningfully attributable to your interventions, particularly in a tough market. I don’t expect that’s fully solvable, but I’m curious about what methods or heuristics you use.
2. What do you see as the main differences between Probably Good and other career advisory organisations/services such as 80,000 Hours or High Impact Professionals? Where do you think your comparative advantage is emerging?
3. You mention an extensive search before selecting an internal hire for the ED role. Roughly how many applicants were involved, and how much staff time did the process consume overall? I’m generally sympathetic to internal hires/closed rounds in most cases because of the potential time/cost savings, but I’d be interested to better understand the level of those potential savings in practice.
Thanks for sharing, I’d definitely be interested in reading a longer retrospective at some point.
Thanks, Siobhan!
We try to assess this type of counterfactual causal attribution through several approaches, including asking people how likely they were to make career changes without our services, looking at other information they provide (like qualitative notes they share), and looking at other information we have on them (like which of our services they used). This is very much a work in progress though.
Mainly, being broader in who we can help and with what, in terms of causes we touch on, audiences we engage, and services we provide. As one example, this sometimes involves doing 1:1 advising with people who are interested in career impact and want a broad perspective across a range of causes, including ones that are underrepresented by other orgs (though we might then refer them to a specialist org). As another example, this involves aiming to offer a broader range of impactful roles on our job board than other organizations might.
(Answering this question from a generalizable hiring perspective, which seems more helpful and appropriate here, though note there’s significant variances across orgs and roles.) While internal rounds sometimes make sense, that’s not the case for a tiny org with only 1-2 potential hires. Closed rounds can also make sense sometimes, but only when you’re confident you’ll get the right candidate pool from this, given how important and hard it is to get hiring right (particularly for a small org, and even more so for a leadership role). The time savings aren’t necessarily great either, since many hiring costs are fixed (e.g., the time it takes to create a job description), and many of the seemingly marginal costs are actually fixed too (e.g., the time it takes to do interviews, if you have a cap for the number of interviews you can do).