Here are my own current thoughts on those questions. These thoughts are of course tentative, and my main aim here is to learn more on this matter (otherwise I wouldâve made a regular post rather than a question post):
On Q1: It seems like itâs rare for research orgs to make (publicly accessible) ToC diagrams? As I said, I only know of two examples of research orgs that have made such diagrams. But I havenât done a systematic or extensive search.
If itâs indeed uncommon for research orgs to make ToC diagrams, that might be due to the âpotential downsidesâ I list below.
On Q2: I have basically no evidence regarding whether many orgs make non-public ToC diagrams. Iâve never seen an example of that, but thatâd likely be the case even if there are many such non-public diagrams, since Iâve only worked at one research org thus far (Convergence), and their ToC diagram was public anyway.
On Q3: My independent impressionâi.e., what Iâd believe if I wasnât updating my beliefs based on what others seem to believeâis that, indeed, more research orgs should make ToC diagrams. But if so few research orgs do that, and most research orgs are run by smart people, maybe I should interpret that as evidence that thereâs some good reason not to do this which Iâm missing?
But Iâm not sure how strong that update should be, because it might be more a matter of norms, inertia, and/âor not having thought of doing this.
Some potential benefits of making ToC diagrams (even if they arenât publicly accessible) that came to my mind are that doing so could help the org:
Make decisions about which research questions/âdirections to pursue
E.g., maybe youâd realise that a line of research that seems fascinating to your researchers actually doesnât have any clear path to influencing your intended outcomes. This would likely push in favour of deprioritising that line of research. (But this consideration could be outweighed. See also Can we intentionally improve the world? Planners vs. Hayekians)
Make decisions about which audiences to target
E.g., you might realise that, in order to achieve your ultimate intended outcomes, you mainly need to influence decisions by [policymakers /â AI researchers /â EAs interested in global health /â company leaders /â whatever]. You can then have that in mind as you choose research questions, writing styles, types of output (e.g., EA Forum posts vs academic articles), etc.
Make decisions about what non-research activities to do, how to do them, and how much time to allocate to them
E.g., you might realise that most of the outcomes you want to bring about require specific people to read your work, such that itâs worth dedicating a decent amount of time to building relationships with those people, attending relevant conferences, etc., rather than just doing research and assuming itâll have an impact somehow.
Work out how they can assess their progress and their impact
E.g., my personal opinion is that, once Convergence had made a ToC diagram, this helped us come up with metrics for assessing our progress and impact. Itâs practically impossible to âdirectlyâ assess whether one has improved the expected value of the future (which was our ultimate aim), but easier to assess how well weâre achieving outcomes on the causal path from our actions to that outcome.
Note I said âeasierâ rather than âeasyâ, and that Iâm comparing this to something thatâs âpractically impossibleââI definitely still find assessing the impact of longtermist research very difficult!
(Note that each of those benefits could likely be captured to at least some extent by an org just describing their theory of change fairly explicitly, without actually making a diagram about it. The same point applies to the remainder of this âanswerâ.)
And some potential downsides of making ToC diagrams that came to my mind are that doing so could:
Take up a lot of time
Not actually provide much clarity anyway
Lead to focusing too much on a particular set of paths to impact that one explicitly maps out, at the expense of doing things that seem worthwhile based on intuitions and heuristics
Of course, itâs hard to say when itâd be better to operate based on intuitions and heuristics rather than a ToC
It also seems possible that the ideal would be to make a ToC, and sometimes/âoften refer to it, but not always refer to it or always require that a project âfitsâ within that ToC.
On Q4: Iâd guess that, if a research org has a ToC diagram, itâd typically make sense to make it publicly accessible. This could mean actively sharing it (e.g., in a post like this one or this one), making it prominently visible on the orgâs website, or burying it in some obscure corner of the orgâs website.
Some potential benefits of making ToC diagrams publicly accessible (rather than just making them in the first place) that came to my mind are that doing so could:
Help potential donors, potential employees, etc. understand why youâre doing what youâre doing, and thus make informed decisions about whether to donate to you, work for you, etc.
E.g., perhaps what youâre doing seems inefficient or unlikely to connect to important real-world outcomes unless people understand your ToC
Or perhaps what youâre doing is misguided, and itâd be good for the world for people to have more evidence of that if thatâs true, so that they can then support something better instead
Help other orgs learn from and adopt clever aspects of your strategy and thinking
Help stakeholders coordinate with you or support you
Iâm less sure what I could mean by this point
One vague and speculative example: If many orgsâ ToC diagrams suggest that a particular type of conference would be useful in order for those orgs to make their research impactful, and those ToC diagrams are publicly accessible, this could lead to someone setting up such a conference.
(It might be possible to capture some of these benefits by just sharing ToC diagrams with specific people, such as potential donors or employees, rather than making those diagrams totally publicly accessible.)
And some potential downsides of making ToC diagrams publicly accessible that came to my mind are that doing that could:
Lead to misperceptions about an orgâs strategy
E.g., it could lead to stakeholders focusing too much on the set of pathways to impact highlighted in the ToC diagram (rather than other, vaguer reasons to believe impact might occur), or perceiving the orgâs thinking as quite simplistic
Allow other actors to harm your org or harm things you value, as a result of them learning from your strategy and thinking
E.g., another research org might improve itself by adopting some of your ideas, and then get some funding youâd have otherwise gotten
But maybe thereâs often little harm in that, as that other org could then do useful things as well
More speculatively, if thereâs some actor that actively wants to prevent you accomplishing your mission, perhaps seeing your ToC would make that easier for them
But again, these are just my current, tentative thoughts, and Iâm primarily interested in hearing what others have to say!
My thanks to Courtney Henry and David Kristoffersson for discussions that informed this answer.
Really interesting thoughts! Thanks for writing them up.
Disclaimer: The below is descriptive, not normative, and chiefly focuses on people and organisations outside of EA
This may be overly cynical, but I think some of the reasons you list in favour of ToCs also account for why they often either donât exist or arenât publicly available, i.e. thereâs a misalignment between whatâs good for society (what youâre getting at with âwhether they should existâ) and what the organisation and/âor researchers consider to be their self-interest. For example,
maybe youâd realise that a line of research that seems fascinating to your researchers actually doesnât have any clear path to influencing your intended outcomes. This would likely push in favour of deprioritising that line of research
As you suggest, itâs very possible that sometimes whatâs most interesting and whatâs most impactful arenât perfectly aligned. Sometimes, oneâs skills and oneâs interests donât align with whatâs needed in the field in which one wants to work. This can be a bitter pill to swallow, especially for people who have already invested heavily in their education and problem area and canât just pivot to a more impactful project. Highlighting the discrepancy privately could jeopardise oneâs role or standing within an organisationâincluding if oneâs coworkers feel that youâre questioning their effectiveness/âutility; highlighting it publicly could jeopardise funding and jobs. Most people arenât EAs, but even EAs are human too.
Help potential donors, potential employees, etc. understand why youâre doing what youâre doing, and thus make informed decisions about whether to donate to you, work for you, etc.
At risk of repeating myself, sometimes people/âorgs would rather not help potential donors, potential employees, etc. make fully informed decisions about whether to fund or work for you. This is particularly the case with grifts. That said, it also applies to organisations that want to do something, act with the best of intentions, secure funding, hire a team, and then fear looking down lest they realise that, like a cartoon character, they arenât on solid footing. Sometimes itâs easier to just keep going.
Again, Iâm not saying this is right, just positing it as another explanation for why we donât see more ToCs.
On Q1: You mention only being aware of a few research orgs that have public ToC diagrams. I wanted to bring your attention to Animal Charity Evaluators, which uses ToC diagrams as a way of better communicating with the public how ACE thinks that a given recommended organization might be causing one of its animal advocacy outcomes.
ACE also uses a ToC diagram in its strategic plan, but this might not be easily searchable because it exists publicly only on pdf documents. (The webpage hosting the strategic plan doesnât use the phrase âtheory of changeâ at all, even though a ToC diagram does exist within the linked pdf there.)
I was aware of some non-research orgs who have ToC diagrams, and some research orgs who make ToC diagrams about things other than their own org (e.g., Charity Entrepreneurship make ToC diagrams for the charity ideas they researched). That seems fairly different from a ToC diagram for a research org itself. This is because research seems unusually hard to imagine the intended path to impact of, and research is where I most often get a sense people donât have a clear idea about the ToC. (Whereas with a charity like AMF, itâs pretty easy for me to imagine roughly what the ToC would look like, even without seeing a diagram AMF have made.)
So I was only a bit interested in ACEâs ToC diagrams about the orgs it recommends. But ACEâs ToC diagram for itself was indeed quite interesting for my purposes.
For the benefit of other readers, that diagram is on page 15 of the linked PDF.
(This brings the total number of know of 6, with the others being Convergence Analysis, Happier Lives Institute, Animal Charity Evaluators, and Leverageâall mentioned somewhere in the comments on this post.)
Here are my own current thoughts on those questions. These thoughts are of course tentative, and my main aim here is to learn more on this matter (otherwise I wouldâve made a regular post rather than a question post):
On Q1: It seems like itâs rare for research orgs to make (publicly accessible) ToC diagrams? As I said, I only know of two examples of research orgs that have made such diagrams. But I havenât done a systematic or extensive search.
If itâs indeed uncommon for research orgs to make ToC diagrams, that might be due to the âpotential downsidesâ I list below.
On Q2: I have basically no evidence regarding whether many orgs make non-public ToC diagrams. Iâve never seen an example of that, but thatâd likely be the case even if there are many such non-public diagrams, since Iâve only worked at one research org thus far (Convergence), and their ToC diagram was public anyway.
On Q3: My independent impressionâi.e., what Iâd believe if I wasnât updating my beliefs based on what others seem to believeâis that, indeed, more research orgs should make ToC diagrams. But if so few research orgs do that, and most research orgs are run by smart people, maybe I should interpret that as evidence that thereâs some good reason not to do this which Iâm missing?
But Iâm not sure how strong that update should be, because it might be more a matter of norms, inertia, and/âor not having thought of doing this.
Some potential benefits of making ToC diagrams (even if they arenât publicly accessible) that came to my mind are that doing so could help the org:
Make decisions about which research questions/âdirections to pursue
E.g., maybe youâd realise that a line of research that seems fascinating to your researchers actually doesnât have any clear path to influencing your intended outcomes. This would likely push in favour of deprioritising that line of research. (But this consideration could be outweighed. See also Can we intentionally improve the world? Planners vs. Hayekians)
Make decisions about which audiences to target
E.g., you might realise that, in order to achieve your ultimate intended outcomes, you mainly need to influence decisions by [policymakers /â AI researchers /â EAs interested in global health /â company leaders /â whatever]. You can then have that in mind as you choose research questions, writing styles, types of output (e.g., EA Forum posts vs academic articles), etc.
Make decisions about what non-research activities to do, how to do them, and how much time to allocate to them
E.g., you might realise that most of the outcomes you want to bring about require specific people to read your work, such that itâs worth dedicating a decent amount of time to building relationships with those people, attending relevant conferences, etc., rather than just doing research and assuming itâll have an impact somehow.
Work out how they can assess their progress and their impact
E.g., my personal opinion is that, once Convergence had made a ToC diagram, this helped us come up with metrics for assessing our progress and impact. Itâs practically impossible to âdirectlyâ assess whether one has improved the expected value of the future (which was our ultimate aim), but easier to assess how well weâre achieving outcomes on the causal path from our actions to that outcome.
Note I said âeasierâ rather than âeasyâ, and that Iâm comparing this to something thatâs âpractically impossibleââI definitely still find assessing the impact of longtermist research very difficult!
(Note that each of those benefits could likely be captured to at least some extent by an org just describing their theory of change fairly explicitly, without actually making a diagram about it. The same point applies to the remainder of this âanswerâ.)
And some potential downsides of making ToC diagrams that came to my mind are that doing so could:
Take up a lot of time
Not actually provide much clarity anyway
Lead to focusing too much on a particular set of paths to impact that one explicitly maps out, at the expense of doing things that seem worthwhile based on intuitions and heuristics
See again Can we intentionally improve the world? Planners vs. Hayekians, and perhaps also Goodhartâs law.
Of course, itâs hard to say when itâd be better to operate based on intuitions and heuristics rather than a ToC
It also seems possible that the ideal would be to make a ToC, and sometimes/âoften refer to it, but not always refer to it or always require that a project âfitsâ within that ToC.
On Q4: Iâd guess that, if a research org has a ToC diagram, itâd typically make sense to make it publicly accessible. This could mean actively sharing it (e.g., in a post like this one or this one), making it prominently visible on the orgâs website, or burying it in some obscure corner of the orgâs website.
Some potential benefits of making ToC diagrams publicly accessible (rather than just making them in the first place) that came to my mind are that doing so could:
Help potential donors, potential employees, etc. understand why youâre doing what youâre doing, and thus make informed decisions about whether to donate to you, work for you, etc.
E.g., perhaps what youâre doing seems inefficient or unlikely to connect to important real-world outcomes unless people understand your ToC
Or perhaps what youâre doing is misguided, and itâd be good for the world for people to have more evidence of that if thatâs true, so that they can then support something better instead
Help other orgs learn from and adopt clever aspects of your strategy and thinking
Help stakeholders coordinate with you or support you
Iâm less sure what I could mean by this point
One vague and speculative example: If many orgsâ ToC diagrams suggest that a particular type of conference would be useful in order for those orgs to make their research impactful, and those ToC diagrams are publicly accessible, this could lead to someone setting up such a conference.
(It might be possible to capture some of these benefits by just sharing ToC diagrams with specific people, such as potential donors or employees, rather than making those diagrams totally publicly accessible.)
And some potential downsides of making ToC diagrams publicly accessible that came to my mind are that doing that could:
Lead to misperceptions about an orgâs strategy
E.g., it could lead to stakeholders focusing too much on the set of pathways to impact highlighted in the ToC diagram (rather than other, vaguer reasons to believe impact might occur), or perceiving the orgâs thinking as quite simplistic
Allow other actors to harm your org or harm things you value, as a result of them learning from your strategy and thinking
E.g., another research org might improve itself by adopting some of your ideas, and then get some funding youâd have otherwise gotten
But maybe thereâs often little harm in that, as that other org could then do useful things as well
More speculatively, if thereâs some actor that actively wants to prevent you accomplishing your mission, perhaps seeing your ToC would make that easier for them
But again, these are just my current, tentative thoughts, and Iâm primarily interested in hearing what others have to say!
My thanks to Courtney Henry and David Kristoffersson for discussions that informed this answer.
Really interesting thoughts! Thanks for writing them up.
Disclaimer: The below is descriptive, not normative, and chiefly focuses on people and organisations outside of EA
This may be overly cynical, but I think some of the reasons you list in favour of ToCs also account for why they often either donât exist or arenât publicly available, i.e. thereâs a misalignment between whatâs good for society (what youâre getting at with âwhether they should existâ) and what the organisation and/âor researchers consider to be their self-interest. For example,
As you suggest, itâs very possible that sometimes whatâs most interesting and whatâs most impactful arenât perfectly aligned. Sometimes, oneâs skills and oneâs interests donât align with whatâs needed in the field in which one wants to work. This can be a bitter pill to swallow, especially for people who have already invested heavily in their education and problem area and canât just pivot to a more impactful project. Highlighting the discrepancy privately could jeopardise oneâs role or standing within an organisationâincluding if oneâs coworkers feel that youâre questioning their effectiveness/âutility; highlighting it publicly could jeopardise funding and jobs. Most people arenât EAs, but even EAs are human too.
At risk of repeating myself, sometimes people/âorgs would rather not help potential donors, potential employees, etc. make fully informed decisions about whether to fund or work for you. This is particularly the case with grifts. That said, it also applies to organisations that want to do something, act with the best of intentions, secure funding, hire a team, and then fear looking down lest they realise that, like a cartoon character, they arenât on solid footing. Sometimes itâs easier to just keep going.
Again, Iâm not saying this is right, just positing it as another explanation for why we donât see more ToCs.
On Q1: You mention only being aware of a few research orgs that have public ToC diagrams. I wanted to bring your attention to Animal Charity Evaluators, which uses ToC diagrams as a way of better communicating with the public how ACE thinks that a given recommended organization might be causing one of its animal advocacy outcomes.
ACE also uses a ToC diagram in its strategic plan, but this might not be easily searchable because it exists publicly only on pdf documents. (The webpage hosting the strategic plan doesnât use the phrase âtheory of changeâ at all, even though a ToC diagram does exist within the linked pdf there.)
Thanks for your comment.
I was aware of some non-research orgs who have ToC diagrams, and some research orgs who make ToC diagrams about things other than their own org (e.g., Charity Entrepreneurship make ToC diagrams for the charity ideas they researched). That seems fairly different from a ToC diagram for a research org itself. This is because research seems unusually hard to imagine the intended path to impact of, and research is where I most often get a sense people donât have a clear idea about the ToC. (Whereas with a charity like AMF, itâs pretty easy for me to imagine roughly what the ToC would look like, even without seeing a diagram AMF have made.)
So I was only a bit interested in ACEâs ToC diagrams about the orgs it recommends. But ACEâs ToC diagram for itself was indeed quite interesting for my purposes.
For the benefit of other readers, that diagram is on page 15 of the linked PDF.
On Q1, I now know of two additional research organisations which have made something like a public theory of change diagram for their work:
MIRI in 2017
Rethink Priorities in 2020
(This brings the total number of know of 6, with the others being Convergence Analysis, Happier Lives Institute, Animal Charity Evaluators, and Leverageâall mentioned somewhere in the comments on this post.)