Notes on quantifying the impact of hiring/​funding EAs

[brb243]

Summary

This post is inspired by question 5 (source). Multiplicative (of the organization’s impact) and additive impact of candidates at different organizations should be recorded in two matrices. The impact without any new hires of each organization considered by the candidates should be estimated (as a vector). Then, an FTE fraction matrix that maximizes the total impact premium due to hiring the candidates should be found. This is mathematically possible but I am only walking through a simple case where the FTE matrix is given. The practical challenges include finding the two candidate impact matrices and the organizations’ impact vector. I am wondering which direction this research should go. The post is based on this discussion.

Definitions and assumptions

The impact of n new candidates hired by m organizations (EA-related and -unlabeled) is the sum of the impacts of the organizations with the candidates minus the sum of the original (without any new candidates) impacts of the organizations.

$impactofhiring={\sum }^{}impactwithhires-{\sum }^{}impactwithouthires$

The original impact of m organizations can be expressed as an m-dimensional vector where each element represents the impact of one organization. The sum of the elements is the total impact.

Organizations’ original impact vector (o):

${\sum }^{}impactwithouthires:sumofelements\left[\begin{array}{c}impactorg1\\ impactorg2\\ ...\\ impactorgm\end{array}\right]$

Let’s assume that if any of the n candidates works full-time for any of the m organizations, they multiply the organization’s impact by a factor from the following matrix.

Candidates’ ‘multiplicative impact’ (multiplication of an org’s impact) matrix (M):

$Candidate{s}^{\prime }multiplicativeimpact:\left[\begin{array}{cccc}cand1prodorg1& cand2prodorg1& \dots & candnprodorg1\\ cand1prodorg2& \dots & ...& ...\\ \dots & \dots & \dots & ...\\ cand1prodorgm& \dots & ...& ...\end{array}\right]$

Rows share organizations and columns candidates.

Let’s further assume that the candidates’ impact on the org’s impact increases proportionally with the time (fraction of FTE) they work there. For example, if a candidate with an FTE impact factor 10 works 0.2 FTEs at that org, their multiplicative impact factor is 2. Candidates are hired for the fraction of FTE specified by the following matrix. The sum of elements in columns is between 0 and 1 (in total, a candidate can work from 0 to 1 FTE in total for any of the m organizations).

Candidates’ FTE fraction matrix (F):

$Candidate{s}^{\prime }FTEmatrix:\left[\begin{array}{cccc}cand1FTEorg1& cand2FTEorg1& \dots & candnFTEorg1\\ cand1FTEorg2& \dots & ...& ...\\ \dots & \dots & \dots & ...\\ cand1FTEorgm& \dots & ...& ...\end{array}\right]$

Rows share organizations and columns candidates.

Let’s also assume that candidates’ multiplicative productivity at an org is not influenced by others’ FTEs at that org. For example, if an organization of original impact 10 hires two candidates who would alone multiply its impact by 1.1 and 3, respectively, the new impact of that org is 33.

Calculation in words

(You can also skip to the example below.)

Upon this assumption, the impact of the organizations with the new hires is calculated by

1. First, subtracting 1 from positive elements in M and adding 1 to negative elements (matrix A). This is the premium productivity of the candidate. For example, if a candidate would multiply the impact by 1.1 then the premium is 1.1-1=0.1 (they increase the productivity of the org by 10%).

2. Then, multiplying each element of the A matrix with the element in the same position in the F matrix (the result is matrix B).

3. Further, adding 1 to all elements in B and taking the absolute value of the resulting elements (matrix C).

4. After, multiplying the elements in each row of C by each other (but excluding zeros) (vector d).

5. Subsequently, multiplying the elements of the same indices (on the same row) of d and o (vector e).

6. The sum (f) of the elements of vector d is the premium impact caused by hiring n candidates for the FTEs specified by the F matrix.

7. To compare this impact to that of the original situation (without new hires), the sum (g) of elements in o should be subtracted from f (difference in impact, h) or e should be divided by g (percent increase, i).

Example calculation

I am unsure about the notation, so I use an example of 2 candidates and 2 organizations.

Let:

$o=\left[\begin{array}{c}10\\ 5\end{array}\right]$

Organization 1 without any new hires has an impact of 10 units while organization 2 has originally an impact of 5 units.

$M=\left[\begin{array}{cc}1.1& 3\\ 0.9& -2\end{array}\right]$

If candidate 1 works for an FTE for organization 1, they multiply its impact by 1.1. If this person works for organization 2, they multiply their impact by 0.9 (so the resulting impact will be lower than that of organization 2 without hiring candidate 1 but still of the same sign). Candidate 2 multiplies the impact of organization 1 by 3 and they would double and change the sign of impact of organization 2.

$F=\left[\begin{array}{cc}0.5& 0.2\\ 0.1& 0\end{array}\right]$

In this scenario, candidate 1 works exactly half-FTE for organization 1 and 0.1 FTE at organization 2. Candidate 2 is hired for 0.2 FTEs by organization 1 and does not work for organization 2.

Then,

Subtracting 1 from positive elements in M and adding 1 to negative elements:

$A=\left[\begin{array}{cc}0.1& 2\\ -0.1& -1\end{array}\right]$

Multiplying each element in A and F:

$B=\left[\begin{array}{cc}0.05& 0.4\\ -0.01& 0\end{array}\right]$

Adding 1 to positive and negative elements and taking the absolute values of the results.

$C=\left[\begin{array}{cc}1.05& 1.4\\ 0.99& 0\end{array}\right]$

After hiring, candidate 1 multiplies the impact of organization 1 by 1.05 and candidate 2 by 1.4. Organization 2’s impact after this hiring round decreases by a factor of 99% due to candidate 1 while candidate 2 does not affect this impact (since they are not hired).

Multiplying the elements in C on the same row with each other but ignoring zeros:

$d=\left[\begin{array}{c}1.47\\ 0.99\end{array}\right]$

Together, the candidates multiply the impact of organization 1 by 147% and of organization 2 by 99%.

Multiplying the elements of the same indices of d and o:

$e=\left[\begin{array}{c}14.7\\ 4.95\end{array}\right]$

After hiring, organization 1 has an impact of 14.7 units while the impact of organization 2 is 4.95 units.

The new impact (f) of the two organizations together is 19.65.

Before, the total impact was 15 units (g). So, the impact increased (h) by 4.65 units or by a factor of 131% (i).

Discussion

The solvable problem is always to find the FTE fraction F matrix in a way that maximizes the difference between the original and the new impact sum (h) (or k, see further below) given the organizations’ original impact o and the candidates’ impact multiplication at each organization, M.

The practical challenge is to estimate o and M. o can be agreed upon by experts (consensus decisionmaking utilizes the human brain to weigh the arguments accurately and relevant calibration practice may increase accuracy). For M, Bayesian updating can be used (e. g. during application process or the trial period (when the counterfactual can be assumed no change in impact due to short time)).

The assumption that the multiplicative impact factor of different candidates hired at the same does not depend on other hires can be

1. avoided by always hiring one additional candidate (in a department),

2. relatively accurate if impact of a decisionmaker includes hiring others for non-innovating roles and the newly hired candidates multiply the impact of the organization very little (e. g. due to limited contributions to innovation), or

3. misguided if the impact of team members interacts. For example, team members of complementary skills can multiply the org’s impact differently than those with similar skills.

Case 3) can be addressed by estimating the impact of different viable candidate FTE combinations in later rounds of the application process and using this in lieu of the product of the candidates’ multiplicative factors.

The above considered that a candidate can multiply an organization’s impact but not that they can add to it. Adding impact is relevant for new ventures or in the case of relatively independent work of a candidate in an existing organization. The candidate’s additional impact is usually related to the org’s impact (e. g. due to existing infrastructure) but not by a formula. Assuming that the additional impact is proportional to the candidate’s FTE, a matrix of candidates’ additional impact (J) can be simply multiplied by the relevant elements in the FTE matrix and the total impact (k) added to the new impact (f). Then, the difference between the new and original impact would be (k+f)-g.

Candidates’ additional impact matrix:

$Candidate{s}^{\prime }additionalimpact:\left[\begin{array}{cccc}cand1addorg1& cand2addorg1& \dots & candnaddorg1\\ cand1addorg2& \dots & ...& ...\\ \dots & \dots & \dots & ...\\ cand1addorgm& \dots & ...& ...\end{array}\right]$

Rows share organizations and columns candidates.

Double-counting should be avoided if both additional and multiplicative impact is used. Multiplicative impact should relate to innovations relevant to the processes that influence the entire organization while additive impact should not affect the organization’s other workings.

Another consideration relates to candidates’ learning and its efficiency (‘runway’) and the ability to keep learning for a specific period of time (e. g. due to savings). For example, it may be valuable for an organization to slightly decrease its impact while they are training a new hire so that the hire can better increase the impact of the organization in the longer term. A candidate can also spend time learning outside of an organization to further increase its additive or multiplicative impact if they are hired.

Thus, the above calculations should be done for different segments of time separately. The sum of impact over all times should be considered. I suggest that monthly impact is used because it is granular enough to account for even shorter training or learning but long enough to make estimating feasible.

Candidate’s learning efficiency could inform time and extent of decreased impact due to training a new hire. Candidate’s maximum time to learn outside of an organization can be accounted for in hiring decisions in conjunction with the above factors and alongside with the probability of the person reapplying for high-impact jobs after working in a position where making impact is relatively less feasible.

Questions and further research

• What can be some high-value next steps?

• What are the main issues with this approach (mathematical and practical/​applied)?

• Is there any other way to estimate the impact of EA-related hiring (e. g. not using matrices)?

• Is there a way to evaluate the impact of existing hires (how would one estimate the counterfactual)? (Then, the impact of hiring process aspects, such as interview type, can be estimated, such as by a regression analysis.)

• Would organizations use this approach? Why (not)?

• What is missing?

• Would quantifying the impact of EA-related hiring be net beneficial? What are the risks?

Conclusion

Using a robust quantitative analysis to optimize for EA candidates’ hiring impact (over the longer-term, using models updated monthly, considering both multiplicative and additive impact, accounting for candidates’ impact interactions and learning efficiencies, and being aware of persons’ abilities to develop skills independently and their expected willingness to re-apply for high-impact roles after working in a place less focused on impact) is possible but obtaining the input data is currently a challenge.

As of 2022-06-22, the certificate of this article is owned by brb243 (100%).

[Dawn Drescher]

Very cool formalization! What do you think of the following way of applying it:

1. Hiring managers that are looking for similar candidates meet (e.g., online) to hash out a single standardized application process for all the similar open positions.

2. When the applications are in and they have narrowed them down to the set candidates any of them find at all interesting, they start the process.

3. Is there a strong reason why they would need to agree on relative impact scores for their organizations? I imagine they’ll find it hard to agree on those. Maybe they can just assume a vector of [1, 1, 1, …] for all the impact scores?

4. They add more pseudo-organizations to the mix, which could have an impact of −100 in spaces like AGI (representing the average non-safety AGI lab) or 0 in most other spaces. (They don’t have control over the choice between other orgs, so I don’t think it makes sense to add several different values, but there need to be as many pseudo-organizations as there are candidates, in case all organizations decide not to hire.)

5. They generate the candidate-organization matrix but also include columns for “no candidate at org n,” because not hiring is also an option. (“No candidate” can “work” for multible or all organizations in parallel, so this gets a bit complicated.)

6. I think in most cases they can assume that everyone will be full time, which could simplify the process in those cases.

7. Then they pick the row that maximizes the impact.

Step 5 seems like the one that’ll require a lot more work in practice. There could be an independent team of forecasters that does the estimation or all hiring managers estimate this for all orgs, or the hiring manages of org A makes the case for/​against each candidate at org A, and then all other hiring manages estimate the impact.

There could be the risk that if one org was wrong and their eventual pick soon quits or doesn’t do a good job, that then the remaining allocation is not optimal anymore because that person would’ve been a good fit at another org that now doesn’t have capacity to hire them anymore.

[brb243]

Thank you :)

1. Sure, standardized application process for similar positions but also for their diverse set. Because a candidate’s impact in one type of role should be compared to that in another role type. 2. So, maybe there should be two ‘rounds’ of recommendations: one ‘broad’ to e. g. prevent people who would do relatively poorly in a policy think tank applying there and vice versa and then when there are final candidates for all similar policy think tank positions, there can be further estimation of best fit into specific roles. 3 and 4. Sure, that makes sense: since EA-related orgs communicate, impact could be assumed the same but some EA-unrelated orgs could have even negative or neutral impact. 5. I think that if an organization does not hire then it is just all zeros in a row in the FTE matrix. Multiple rows can have all zeros—if multiple orgs do not hire. 6) Yes, that would make the calculation easier—because only one one could be in one row and column in the candidate-org matrix. Then, for the no hiring of some orgs, it would be at most one 1 in row or column. 7) You mean if candidates are in rows and orgs in columns then the hiring managers pick for each candidate the row that maximizes their impact? (in that row there would be one for the org the candidate should work for)

Probably if each hiring manager estimates maybe top 5 candidates and their relative impact would be the easiest? Oh yes :) that is a risk but maybe hiring managers should include uncertainty which would complicate things … or should hire everyone for a trial period and then reevaluate (which could be somewhat unpopular since some candidates could change jobs ).