How would something like this approach be used for decision-making?
You write:
We donāt normally estimate the value of small to medium-sized projects.
But we could!
If we could do this reliably and scalably, this might lead us to choose better projects
And:
In addition, one could also use these kinds of estimates to choose oneās own projects, or to recommend projects to others, and see how that fares.
But this post estimates the impact of already completed projects/āwriteups. So precisely this sort of method couldnāt directly be used to choose what projects to do. Instead, I see at least two broad ways something like this method could be used as an input when choosing what projects to do:
When choosing what projects to do, look at estimates like these, and either explicitly reason about or form intuitions about what these estimates suggest about the impact of different projects one is considering
One way to do this would be to create classifications for different types of projects, and then look up what has been the estimated impact per dollar of past projects in the same or similar classifications to each of the projects one is now choosing between
I think thereād be many other specific ways to do this as well
When choosing what projects to do, explicitly make estimates like these for those specific future projects
If thatās the idea, then the sort of approach taken in this post could be seen as:
just a proof of concept, or
a way to calibrate oneās intuitions/āforecasts (with the hope being that thereāll be transfer between calibration when estimating the impact of past projects and calibration when forecasting the impact of future projects), or
a way of getting reference classes /ā base rates /ā outside views
In that case, this second approach would sort-of incorporate the first approach suggested above as one part of it
Is one of these what you had in mind? Or both? Or something else?
Yeah, I think that the distinction between evaluation and forecasting is non-central. For example, these estimates can also be viewed as forecasts of what I would estimate if I spent 100x as much time on this, or as forecasts of what a really good system would output.
More to the point, if a project isnāt completed I could just estimate the distribution of expected quality, and the expected impact given each degree of quality (or, do a simplified version of that).
That said, I was thinking more about 2., though having a classification/ālookup scheme would also be a way to produce explicit estimates.
For example, these estimates can also be viewed as forecasts of what I would estimate if I spent 100x as much time on this, or as forecasts of what a really good system would output.
Agreed, but thatās still different from forecasting the impact of a project that hasnāt happened yet, and the difference intuitively seems like it might be meaningful for our purposes. I.e., itās not immediately obvious that methods and intuitions that work well for the sort of estimation/āforecasting done in this post would also work well for forecasting the impact of a project that hasnāt happened yet.
One could likewise say that itās not obvious that methods and intuitions that work well for forecasting how Iāll do in job applications would also work well for forecasting GDP growth in developing countries. So I guess my point was more fundamentally about the potential significance of the domain being different, rather than whether the thing can be seen as a type of forecasting or not.
So it sounds like youāre thinking that the sort of thing done in this post would be āa way to calibrate oneās intuitions/āforecasts (with the hope being that thereāll be transfer between calibration when estimating the impact of past projects and calibration when forecasting the impact of future projects)ā?
That does seem totally plausible to me; it just adds a step to the argument.
(I guess Iām also more generally interested in the question of how well forecasting accuracy and calibration transfers across domainsāthough at the same time I havenāt made the effort to look into it at all...)
Yes, I expect the intuitions and for estimation to generalize/āhelp a great deal with the forecasting step, and though I agree that this might not be intuitively obvious. I understand that estimation and forecasting seem like different categories, but I donāt expect that to be a significant hurdle in practice.
How would something like this approach be used for decision-making?
You write:
And:
But this post estimates the impact of already completed projects/āwriteups. So precisely this sort of method couldnāt directly be used to choose what projects to do. Instead, I see at least two broad ways something like this method could be used as an input when choosing what projects to do:
When choosing what projects to do, look at estimates like these, and either explicitly reason about or form intuitions about what these estimates suggest about the impact of different projects one is considering
One way to do this would be to create classifications for different types of projects, and then look up what has been the estimated impact per dollar of past projects in the same or similar classifications to each of the projects one is now choosing between
I think thereād be many other specific ways to do this as well
When choosing what projects to do, explicitly make estimates like these for those specific future projects
If thatās the idea, then the sort of approach taken in this post could be seen as:
just a proof of concept, or
a way to calibrate oneās intuitions/āforecasts (with the hope being that thereāll be transfer between calibration when estimating the impact of past projects and calibration when forecasting the impact of future projects), or
a way of getting reference classes /ā base rates /ā outside views
In that case, this second approach would sort-of incorporate the first approach suggested above as one part of it
Is one of these what you had in mind? Or both? Or something else?
Yeah, I think that the distinction between evaluation and forecasting is non-central. For example, these estimates can also be viewed as forecasts of what I would estimate if I spent 100x as much time on this, or as forecasts of what a really good system would output.
More to the point, if a project isnāt completed I could just estimate the distribution of expected quality, and the expected impact given each degree of quality (or, do a simplified version of that).
That said, I was thinking more about 2., though having a classification/ālookup scheme would also be a way to produce explicit estimates.
Agreed, but thatās still different from forecasting the impact of a project that hasnāt happened yet, and the difference intuitively seems like it might be meaningful for our purposes. I.e., itās not immediately obvious that methods and intuitions that work well for the sort of estimation/āforecasting done in this post would also work well for forecasting the impact of a project that hasnāt happened yet.
One could likewise say that itās not obvious that methods and intuitions that work well for forecasting how Iāll do in job applications would also work well for forecasting GDP growth in developing countries. So I guess my point was more fundamentally about the potential significance of the domain being different, rather than whether the thing can be seen as a type of forecasting or not.
So it sounds like youāre thinking that the sort of thing done in this post would be āa way to calibrate oneās intuitions/āforecasts (with the hope being that thereāll be transfer between calibration when estimating the impact of past projects and calibration when forecasting the impact of future projects)ā?
That does seem totally plausible to me; it just adds a step to the argument.
(I guess Iām also more generally interested in the question of how well forecasting accuracy and calibration transfers across domainsāthough at the same time I havenāt made the effort to look into it at all...)
Yes, I expect the intuitions and for estimation to generalize/āhelp a great deal with the forecasting step, and though I agree that this might not be intuitively obvious. I understand that estimation and forecasting seem like different categories, but I donāt expect that to be a significant hurdle in practice.