Thanks for creating this post! Sharing some thoughts on the topic based on my experience creating and redteaming theories of change (ToCs) with various EA orgs (partly echoing your observations and partly adding new points; Two concrete project examples can be found here and here).
Neglectedness of ToC work (basically echoing your claim). Due to the non-pressing nature and required senior input, ToC/strategy work seems to get deprioritized very often, e.g., I have been deprioritizing updating our ToC for three months due to more pressing work. I think the optimal time spent thinking about your ToC/prioritization/strategy depends on the maturity of your project and is hard to get right, but based on my experience, most of us spend too little time on it (just like we tend to spend too little time exploring our career options as it is worth to invest 800 hours in our career planning when we can increase its impact by 1% in expectation). Assuming your org has ten staff, which work 200 days a year for 8 hours per day, you would want to invest 16k hours in figuring out how staff spends their time best if it is likely to result in a 1% impact increase
More than one ToC. I think most orgs should have a ToC on the org, team and individual level as well as for each main program/activity. It seems not optimal to work on something without even having thought through how this will change the world for at least 3min (and if there are alternatives, how you could achieve the same with less work)
Different levels of granularity. Depending on the purpose and the context of your ToC, you can have a three-row ToC (e.g., for small projects you are exploring), a flow-chart (e.g., to communicate the ToC of your org clearly, see examples in this post) and/or an exhaustive document showing lots of reasoning transparency, alternatives you considered etc. (e.g., to lay out the ToC of your research agenda)
Developing a ToC. One simplified approach to develop a ToC on an org level which worked well with some clients but always needs tailoring looks very roughly like this: (1) Map out all potential sources of value of today (and potentially in the future), (2) Prioritize them, (3) Create a flow chart for the most promising source of value (potentially include other sources of value or create several flow charts), (4) Think through the flow of impact/value end to end as a sanity check, (5) collect data (e.g., talk to experts, run small experiments) to reduce uncertainties, (6) re-iterate. See more here
Strategic implications and influencing decisions (previously mentioned but I think this is a point many people found useful and is not stressed enough in typical ToC literature). Your ToC should inform key decisions and ultimately how you allocate resources (your staff’s time or money). I never experienced that we were certain about all the sources of value and the causal relationships between each step or did not have a hypothesis on how to have even more impact with an adapted or new program. So the ToC work always had at least some implications for allocating time. One great example is the Fish Welfare Initiative, which included resolving uncertainties around their ToC in their yearly priorities (See slides 13 and 24)
Areas for improvement. (1) Trying to map every casual pathway and not focusing on the most important ones, (2) Deferring too much to others on what they perceive as valuable, e.g., the target group, and not doing enough first principles/independent thinking and data collection, (3) Not considering counterfactuals at all and/or not considering that there are likely several counterfactual worlds, and (4) Not laying out key assumptions and uncertainties (previously mentioned in the post but seems valuable to highlight that this also reflects my experience) among other things
Note that I likely have a significant sample bias, as organizations are unlikely to reach out to me if they have enough time to think through their ToC. Additionally, please read this as “random thoughts which came to Jona’s mind when reading the article” and not as “these are the X main things EA orgs get wrong based on a careful analysis”. I expect to update my views as I learn more
Re: #2, at Charity Entrepreneurship for example, we should have ToCs for our Incubation Program, Grantmaking Program and Research Training Program, but we don’t yet. We have a fairly polished one for the Incubation Program, and a few different ones drafted for the new Research Program we’re planning, but we haven’t written one down for our Grantmaking Program, so here I am again not practicing what I preach. Looks like we have work to do :)
Thanks for creating this post! Sharing some thoughts on the topic based on my experience creating and redteaming theories of change (ToCs) with various EA orgs (partly echoing your observations and partly adding new points; Two concrete project examples can be found here and here).
Neglectedness of ToC work (basically echoing your claim). Due to the non-pressing nature and required senior input, ToC/strategy work seems to get deprioritized very often, e.g., I have been deprioritizing updating our ToC for three months due to more pressing work. I think the optimal time spent thinking about your ToC/prioritization/strategy depends on the maturity of your project and is hard to get right, but based on my experience, most of us spend too little time on it (just like we tend to spend too little time exploring our career options as it is worth to invest 800 hours in our career planning when we can increase its impact by 1% in expectation). Assuming your org has ten staff, which work 200 days a year for 8 hours per day, you would want to invest 16k hours in figuring out how staff spends their time best if it is likely to result in a 1% impact increase
More than one ToC. I think most orgs should have a ToC on the org, team and individual level as well as for each main program/activity. It seems not optimal to work on something without even having thought through how this will change the world for at least 3min (and if there are alternatives, how you could achieve the same with less work)
Different levels of granularity. Depending on the purpose and the context of your ToC, you can have a three-row ToC (e.g., for small projects you are exploring), a flow-chart (e.g., to communicate the ToC of your org clearly, see examples in this post) and/or an exhaustive document showing lots of reasoning transparency, alternatives you considered etc. (e.g., to lay out the ToC of your research agenda)
Developing a ToC. One simplified approach to develop a ToC on an org level which worked well with some clients but always needs tailoring looks very roughly like this: (1) Map out all potential sources of value of today (and potentially in the future), (2) Prioritize them, (3) Create a flow chart for the most promising source of value (potentially include other sources of value or create several flow charts), (4) Think through the flow of impact/value end to end as a sanity check, (5) collect data (e.g., talk to experts, run small experiments) to reduce uncertainties, (6) re-iterate. See more here
Strategic implications and influencing decisions (previously mentioned but I think this is a point many people found useful and is not stressed enough in typical ToC literature). Your ToC should inform key decisions and ultimately how you allocate resources (your staff’s time or money). I never experienced that we were certain about all the sources of value and the causal relationships between each step or did not have a hypothesis on how to have even more impact with an adapted or new program. So the ToC work always had at least some implications for allocating time. One great example is the Fish Welfare Initiative, which included resolving uncertainties around their ToC in their yearly priorities (See slides 13 and 24)
Areas for improvement. (1) Trying to map every casual pathway and not focusing on the most important ones, (2) Deferring too much to others on what they perceive as valuable, e.g., the target group, and not doing enough first principles/independent thinking and data collection, (3) Not considering counterfactuals at all and/or not considering that there are likely several counterfactual worlds, and (4) Not laying out key assumptions and uncertainties (previously mentioned in the post but seems valuable to highlight that this also reflects my experience) among other things
Note that I likely have a significant sample bias, as organizations are unlikely to reach out to me if they have enough time to think through their ToC. Additionally, please read this as “random thoughts which came to Jona’s mind when reading the article” and not as “these are the X main things EA orgs get wrong based on a careful analysis”. I expect to update my views as I learn more
Good stuff Jona! I agree on all fronts.
Re: #2, at Charity Entrepreneurship for example, we should have ToCs for our Incubation Program, Grantmaking Program and Research Training Program, but we don’t yet. We have a fairly polished one for the Incubation Program, and a few different ones drafted for the new Research Program we’re planning, but we haven’t written one down for our Grantmaking Program, so here I am again not practicing what I preach. Looks like we have work to do :)