That works out to around $167k per research output.
That seems like a lot! Maybe I should discount a bit as some of this might be for the new Special Projects team rather than research, but it still seems like it’ll be over $100k per research output.
Related questions:
Do you think the calculations above are broadly correct? If not, could you share what the ballpark figures might actually be? Obviously, this will depend a lot on the size of the project and other factors but averages are still useful!
If they are correct, how come this number is so high? Is it just due to multiple researchers spending a lot of time per report and making sure it’s extremely high-quality? FWIW I think the value of some RP projects is very high—and worth more than the costs above—but I’m still surprised at the costs.
Is the cost something you’re assessing when you decide whether to take on a research project (when it’s not driven by an external client)? Do you have some internal calculations (or external tool) where you try to calculate the value of information of a given research project and weigh that up against proposed costs?
Thanks for your thoughtful question, but I think you’re thinking about this incorrectly for a few reasons:
Firstly, while we raised $10.7M, most of that was earmarked for 2023 as we usually raise money in the current year for the following year. In 2022, we spent around $6.8M on RP core programs, not including special projects and operations to support special projects.
Secondly, we actually have published less than half of our 2022 research. My rough guess is that in 2022 we produced over 100 pieces of work, not ~64 as you estimate. This is for two reasons:
Some research is confidential for whatever reason and is never intended to be published
Some research is intended to be published but we haven’t had the resources or time to publish it yet because public outputs are not a priority for our clients and their funding does not cover it (this is actually something we’d love to get money from the EA public for).
To give a more clear substitute figure, we generally say that $20K-$40K pays for a typical short-term research project and $70K-$100K pays for a typical in-depth research project.
But more importantly I’d add that counting outputs per dollar is not a good way to view RP’s work. This is for a few reasons:
“Outputs” are a vanity metric and we don’t want to take a quantity-focused approach to our work where we aim to produce as many outputs as cheaply as possible. Instead, the quest to quantify the impact – and the impact per dollar – of Rethink Priorities is much more difficult.
“Outputs” vary a lot in size, scope, and funding and aren’t really apples-to-apples comparable in a way that would work for an aggregated metric/count. Some research reports take >12 months of full-time work whereas other research reports (especially internal ones not meant for publication) are completed in two weeks or less.
A lot of the most important work that happens with our research isn’t the time spent actually producing the report, but also engaging with stakeholders, presenting findings, and/or providing feedback on others’ research. We also spend a fair amount of time after we produce research reports engaging with the client — answering follow-up questions and/or otherwise helping people understand the research.
Doing good research at scale requires a sizable operations budget, management budget, strategy work, etc., that doesn’t immediately translate into concrete publishable research outputs.
We also produce other endpoints that aren’t research. We also incubate organizations and spend researcher time advising those organizations and we spend money to organize conferences and other events.
Relatedly, how much of the funding (both for 2022 and for 2024) is for the production of research outputs, compared to how much it is for other operations (like fiscal sponsorships or incubation)?
I think for marginal donations on RP, perhaps the best way to think about this would be in the cost to produce marginal research. A new researcher hire would cost ~$87K in salary (median, there is of course variation by title level here) and ~$28K in other costs (e.g., taxes, employment fees, benefits, equipment, employee travel). We then need to spend ~$31K in marginal spending on operations and ~$28K in marginal spending on management to support a new researcher. So the total cost for one new FTE year of research ends up being ~$174K. I think if you want to get a sense of how much it costs to support research at RP and how that balances between operations and other costs, this is a useful breakdown to look at.
In addition to research and operations, I’d say we produce roughly four other categories of things: fiscal sponsorship, incubated organizations, internal events, and external conferences. Let me go into a bit of detail about that:
Fiscal sponsorship arrangements pay for themselves out of the sponsored org’s budget, so they’re not something we’d seek public funding for.
Incubation work, or work to produce and advise new organizations based on our research (e.g., Condor Camp, Insect Institute) are things we’d raise money for and hope that they’d be impactful enough to encourage you to support. My rough guess is that in 2024 we would spend ~$150K from our animal welfare research budget and ~$900K for our Existential Security Team to work on incubating new organizations, and we are looking to fundraise for those amounts. These would be subject to similar marginal costs per FTE as mentioned above.
Internal events come out of the operations budget mentioned above.
External conferences are not something we’d seek public funding for – these have always been fully covered by getting a specific grant for the conference.
If you want to financially support only a particular part of RP or a particular thing RP does, let me know and we can discuss ways we could honor that arrangement.
I think this really depends on the research output. $100k for a report with roughly one person year’s worth of effort seems about right. Or roughly one good academic paper or master’s thesis. I suspect a lot of Rethink’s reports are more valuable than that.
That’s $100k all in cost, including costs that aren’t specific to a project. Including salary, overheads, taxes, travel, any expenses, training, recruitment etc.
I’m guessing what you mean is something like “One of RP’s aims is to advise grantmaking. How many total dollars of grantmaking have you advised?” You might then be tempted to take this number, divide it by our costs, and compare that to other organizations. But this is a tricky question to answer actually, since it never has been as straightforward of a relationship as I’d expect for a few reasons:
Our advice is marginal and we never make a sole and final decision on any grant. Also the amount of contribution varies a lot between grants. So you need some counterfactually-adjusted marginal figure.
Sometimes our advice leads to grantmakers being less likely to make a grant rather than more likely… how does that count?
The impact value of the grants themselves is not equal.
Some of our research work looks into decisions but doesn’t actually change the answer. For example, we look into an area that we think isn’t promising and confirm it isn’t promising so in absolute terms we got nowhere but the hits-based fact that it could’ve gone somewhere is valuable. It’s hard to figure out how to quantify this value.
A large portion of our research builds on itself. For example, our invertebrate work has led to some novel grantmaking that likely would not have otherwise happened, but only after three years of work. A lot of our current research is still (hopefully) in that pre-payoff period and so hasn’t lead to any concrete grants yet. It’s hard to figure out how to quantify this value.
A large portion of our research is of the form “given that this grant is being made, how can we make sure it goes as well as possible?” rather than actually advising on the initial grant. It’s hard to figure out how to quantify this value.
A lot more of our recent work has been focused on creating entirely new areas to put funding into (e.g., new incubated organizations, exploring new AI interventions). This takes time and is also hard to value.
We’ve been working this year on producing a figure that looks at itemizing decisions we’ve contributed to and estimating how much we’ve influenced that decision and how valuable that decision was, but we don’t have that work finished yet because it is complicated. Additionally, we’ve been involved in such a large number of decisions by this point that it is a lot of hard work to do all the follow-up and number crunching.
Do also keep in mind that influencing grantmaking is not RP’s sole objective and we achieve impact in other ways (e.g., talent recruitment + training + placement, conferences, incubated organizations, fiscal sponsorships).
All this to say is that I don’t actually have an answer to your question. But we did hire a Worldview Investigations Team that is working more on this.
Doing some napkin-math:
Rethink published 32 pieces of research in 2022 (according to your database)
I think roughly (?) half of your work doesn’t get published as it’s for specific clients, so let’s say you produced 64 reports overall in 2022.
Rethink raised $10.7 million in 2022.
That works out to around $167k per research output.
That seems like a lot! Maybe I should discount a bit as some of this might be for the new Special Projects team rather than research, but it still seems like it’ll be over $100k per research output.
Related questions:
Do you think the calculations above are broadly correct? If not, could you share what the ballpark figures might actually be? Obviously, this will depend a lot on the size of the project and other factors but averages are still useful!
If they are correct, how come this number is so high? Is it just due to multiple researchers spending a lot of time per report and making sure it’s extremely high-quality? FWIW I think the value of some RP projects is very high—and worth more than the costs above—but I’m still surprised at the costs.
Is the cost something you’re assessing when you decide whether to take on a research project (when it’s not driven by an external client)? Do you have some internal calculations (or external tool) where you try to calculate the value of information of a given research project and weigh that up against proposed costs?
Hi James,
Thanks for your thoughtful question, but I think you’re thinking about this incorrectly for a few reasons:
Firstly, while we raised $10.7M, most of that was earmarked for 2023 as we usually raise money in the current year for the following year. In 2022, we spent around $6.8M on RP core programs, not including special projects and operations to support special projects.
Secondly, we actually have published less than half of our 2022 research. My rough guess is that in 2022 we produced over 100 pieces of work, not ~64 as you estimate. This is for two reasons:
Some research is confidential for whatever reason and is never intended to be published
Some research is intended to be published but we haven’t had the resources or time to publish it yet because public outputs are not a priority for our clients and their funding does not cover it (this is actually something we’d love to get money from the EA public for).
To give a more clear substitute figure, we generally say that $20K-$40K pays for a typical short-term research project and $70K-$100K pays for a typical in-depth research project.
But more importantly I’d add that counting outputs per dollar is not a good way to view RP’s work. This is for a few reasons:
“Outputs” are a vanity metric and we don’t want to take a quantity-focused approach to our work where we aim to produce as many outputs as cheaply as possible. Instead, the quest to quantify the impact – and the impact per dollar – of Rethink Priorities is much more difficult.
“Outputs” vary a lot in size, scope, and funding and aren’t really apples-to-apples comparable in a way that would work for an aggregated metric/count. Some research reports take >12 months of full-time work whereas other research reports (especially internal ones not meant for publication) are completed in two weeks or less.
A lot of the most important work that happens with our research isn’t the time spent actually producing the report, but also engaging with stakeholders, presenting findings, and/or providing feedback on others’ research. We also spend a fair amount of time after we produce research reports engaging with the client — answering follow-up questions and/or otherwise helping people understand the research.
Doing good research at scale requires a sizable operations budget, management budget, strategy work, etc., that doesn’t immediately translate into concrete publishable research outputs.
We also produce other endpoints that aren’t research. We also incubate organizations and spend researcher time advising those organizations and we spend money to organize conferences and other events.
Love the question
Relatedly, how much of the funding (both for 2022 and for 2024) is for the production of research outputs, compared to how much it is for other operations (like fiscal sponsorships or incubation)?
I think for marginal donations on RP, perhaps the best way to think about this would be in the cost to produce marginal research. A new researcher hire would cost ~$87K in salary (median, there is of course variation by title level here) and ~$28K in other costs (e.g., taxes, employment fees, benefits, equipment, employee travel). We then need to spend ~$31K in marginal spending on operations and ~$28K in marginal spending on management to support a new researcher. So the total cost for one new FTE year of research ends up being ~$174K. I think if you want to get a sense of how much it costs to support research at RP and how that balances between operations and other costs, this is a useful breakdown to look at.
In addition to research and operations, I’d say we produce roughly four other categories of things: fiscal sponsorship, incubated organizations, internal events, and external conferences. Let me go into a bit of detail about that:
Fiscal sponsorship arrangements pay for themselves out of the sponsored org’s budget, so they’re not something we’d seek public funding for.
Incubation work, or work to produce and advise new organizations based on our research (e.g., Condor Camp, Insect Institute) are things we’d raise money for and hope that they’d be impactful enough to encourage you to support. My rough guess is that in 2024 we would spend ~$150K from our animal welfare research budget and ~$900K for our Existential Security Team to work on incubating new organizations, and we are looking to fundraise for those amounts. These would be subject to similar marginal costs per FTE as mentioned above.
Internal events come out of the operations budget mentioned above.
External conferences are not something we’d seek public funding for – these have always been fully covered by getting a specific grant for the conference.
If you want to financially support only a particular part of RP or a particular thing RP does, let me know and we can discuss ways we could honor that arrangement.
I think this really depends on the research output. $100k for a report with roughly one person year’s worth of effort seems about right. Or roughly one good academic paper or master’s thesis. I suspect a lot of Rethink’s reports are more valuable than that.
That’s $100k all in cost, including costs that aren’t specific to a project. Including salary, overheads, taxes, travel, any expenses, training, recruitment etc.
Do you have a sense of how much funding this informed?
I’m guessing what you mean is something like “One of RP’s aims is to advise grantmaking. How many total dollars of grantmaking have you advised?” You might then be tempted to take this number, divide it by our costs, and compare that to other organizations. But this is a tricky question to answer actually, since it never has been as straightforward of a relationship as I’d expect for a few reasons:
Our advice is marginal and we never make a sole and final decision on any grant. Also the amount of contribution varies a lot between grants. So you need some counterfactually-adjusted marginal figure.
Sometimes our advice leads to grantmakers being less likely to make a grant rather than more likely… how does that count?
The impact value of the grants themselves is not equal.
Some of our research work looks into decisions but doesn’t actually change the answer. For example, we look into an area that we think isn’t promising and confirm it isn’t promising so in absolute terms we got nowhere but the hits-based fact that it could’ve gone somewhere is valuable. It’s hard to figure out how to quantify this value.
A large portion of our research builds on itself. For example, our invertebrate work has led to some novel grantmaking that likely would not have otherwise happened, but only after three years of work. A lot of our current research is still (hopefully) in that pre-payoff period and so hasn’t lead to any concrete grants yet. It’s hard to figure out how to quantify this value.
A large portion of our research is of the form “given that this grant is being made, how can we make sure it goes as well as possible?” rather than actually advising on the initial grant. It’s hard to figure out how to quantify this value.
A lot more of our recent work has been focused on creating entirely new areas to put funding into (e.g., new incubated organizations, exploring new AI interventions). This takes time and is also hard to value.
We’ve been working this year on producing a figure that looks at itemizing decisions we’ve contributed to and estimating how much we’ve influenced that decision and how valuable that decision was, but we don’t have that work finished yet because it is complicated. Additionally, we’ve been involved in such a large number of decisions by this point that it is a lot of hard work to do all the follow-up and number crunching.
Do also keep in mind that influencing grantmaking is not RP’s sole objective and we achieve impact in other ways (e.g., talent recruitment + training + placement, conferences, incubated organizations, fiscal sponsorships).
All this to say is that I don’t actually have an answer to your question. But we did hire a Worldview Investigations Team that is working more on this.