Some things you might want to do if you are making a weighted factor model
Weighted factor models are commonly used within EA (e.g. by Charity Entrepreneurship/AIM and 80,000 Hours). Even the formalised Scale, Solvability, Neglectedness framework can, itself, be considered a form of weighted factor model.
However, despite their wide use, weighted factor models often neglect to use important methodological techniques which could test and improve their robustness, which may threaten their validity and usefulness. RP’s Surveys and Data Analysis team previously consulted for a project who were using a WFM, and helped them understand certain things that were confusing them about the behaviour of their model using these techniques, but we’ve never had time to write up a detailed post about these methods. Such a post would discuss such topics as:
Problems with ordinal measures
When (not) to rank scores
When and how (not) to normalise
How to make interpretable rating scales
Identifying the factors that drive your outcomes
Quantifying and interpreting disagreement / uncertainty
Some things you might want to do if you are making a weighted factor model
Weighted factor models are commonly used within EA (e.g. by Charity Entrepreneurship/AIM and 80,000 Hours). Even the formalised Scale, Solvability, Neglectedness framework can, itself, be considered a form of weighted factor model.
However, despite their wide use, weighted factor models often neglect to use important methodological techniques which could test and improve their robustness, which may threaten their validity and usefulness. RP’s Surveys and Data Analysis team previously consulted for a project who were using a WFM, and helped them understand certain things that were confusing them about the behaviour of their model using these techniques, but we’ve never had time to write up a detailed post about these methods. Such a post would discuss such topics as:
Problems with ordinal measures
When (not) to rank scores
When and how (not) to normalise
How to make interpretable rating scales
Identifying the factors that drive your outcomes
Quantifying and interpreting disagreement / uncertainty
This would be great to read, I walked away from at least one application process because I couldn’t produce a decent WFM. I hope you write it!