As part of the Quantified Uncertainty Research Instituteās (QURI) strategy efforts, I thought it would be a good idea to write down what I think the pathways to impact are for forecasting and evaluations. Comments are welcome, and may change what QURI focuses on in the upcoming year.
He adds in a section on āReflectionsā:
Most of the benefit of these kinds of diagrams seems to me to come from the increased clarity they allow for when thinking about their content. Otherwise, I imagine that they might make QURIās and my own work more legible to outsiders by making our assumptions or steps more explicit, which itself might allow for people to point out criticism. Note that my guess about the main pathways are highlighted in bold, so one could disagree about that without disagreeing about the rest of the diagram.
I also imagine that the forecasting and evaluations pathways could be useful to organizations other than QURI (Metaculus, other forecasting platforms, people thinking of commissioning evaluations, etc.)
It seems to me that producing these kinds of diagrams is easier over an extended period of time, rather than in one sitting because one can then come back to aspects that seem missing.
On the same post, Ozzie Gooen (Nunoās colleague at QURI) wrote:
I looked over an earlier version of this, just wanted to post my takes publicly.[1]
I like making diagrams of impact, and these seem like the right things to model. Going through them, many of the pieces seem generally right to me. I agree with many of the details, and I think this process was useful for getting us (QURI, which is just the two of us now) on the same page.
At the same time though, I think itās surprisingly difficult to make these diagrams to be understandable for many people.
Things get messy quickly. The alternatives are to make them much simpler, and/āor to try to style them better.
I think these could have been organized much neater, for example, by:
Having the flow always go left-to-right.
Using a different diagram editor that looks neater.
Reducing the number of nodes by maybe 30% or so.
Maybe neater arrow structures (having 90% lines, rather than diagonal lines) or something.
That said, this would have been a lot of work to do (required deciding on and using different software), and thereās a lot of stuff to do, so this is more āstuff to keep in mind for the future, particularly if we want to share these with many more people.ā (Nuno and I discussed this earlier)
One challenge is that some of the decisions on the particularities of the causal paths feel fairly ad-hoc, even though they make sense in isolation. I think theyāre useful for a few people to get a grasp on the main factors, but theyāre difficult to use for getting broad buy-in.
If you take a quick glance and just think, āThis looks really messy, Iām not going to botherā, I donāt particularly blame you (Iāve made very similar things that people have glanced over).
But the information is interesting, if you ever consider it worth your time/āeffort!
So, TLDR:
Impact diagrams are really hard. At these levels of details, much more so.
This is a useful exercise, and itās good to get the information out there.
I imagine some viewers will be intimidated by the diagrams.
Iām a fan of experimenting with things like this and trying out new software, so that was neat.
[1] I think itās good to share these publicly for transparency + understanding.
Nuno Sempere of QURI just published a short post on Pathways to impact for forecasting and evaluation, which has two big diagrams and opens with:
He adds in a section on āReflectionsā:
On the same post, Ozzie Gooen (Nunoās colleague at QURI) wrote: