That’s fair! It’s just also reasonable for me or other non-PM forecasting folks to be concerned about people making the wrong inference. I’m currently set to pay for the SSPP and other forecasting work out of my own personal research budget next year, having not found other funding yet. I had a full proposal out to cG when they shuttered the stream, though there are some other possibilities I’m exploring.
Eva
Karma: 724
Thanks! I agree, I’m also generally skeptical of small chance * big number things—I was not intending 1% as an anchor but as an open question—and not as a probability but as a concrete percent of the funding. For example, a big org uses forecasts, but perhaps they only use them in particular workstreams responsible for X% of funding, and those workstreams could be tracked. Then out of X%, how much do they move the needle?
Anyway, happy to chat sometime!
Not Josh, and also conflicted through the Social Science Prediction Platform (though we had pretty minimal funding from EA sources), but I wonder if it would be worth pooling non-public projects we know of and making BOTE estimates of hypothetical impact. It’s tricky because I don’t know of any RCTs (though I’m working on one now). But I’m extremely confident that across us we would think of some combination of orgs/governments that collectively spend over $100 billion per year (… I can think of that alone) that are interested in forecasts in different ways. Now, imo the vast majority of places interested in forecasting are not going to do anything substantive with it, and it’s hard to know what it means for one of these places to integrate forecasts—for example, for an org spending $X, do forecasts inform 1% of their funding or what? Of the share they inform, how much do they move the needle? If estimates from people who work on forecasts may be optimistic (I’m not paid at all for it, but I choose to work on it because I think it’s useful), happy to describe the situation to an outside observer privately.
I think one should distinguish between several things here:
Prediction markets that make a lot of $ and don’t really need more because they do just fine with the profit motive
People spending a lot of time on prediction markets to prove they are a good forecaster
Infrastructure integrated into specific use cases, such as: when a funder is interested in a question so much that they will pay for forecasts, to inform and improve their other funding, or when for structural reasons the institution that has reason to benefit from the forecasts cannot fund them itself (such as some other decision-makers who do not have the mandate to support forecasting and are restricted from spending money to support it but would use forecasts in their work flow), or basic research with positive externalities.
This post really belabours the first and second bullet point, perhaps because that is where a lot of money has gone to, but there can be a lot of value in the third.
The wisdom of crowds effect kicks in with very few forecasts. In the working paper I cite elsewhere in the comments, even 5 forecasts gets you pretty far along into the WoC effect, and 10 even more so. This is for asking people what they think, not prediction markets—the latter should, theoretically, require more forecasts, since seeing the implicit beliefs of others through the market price could lead to herding etc. But the wisdom of crowds effect kicking in for very small N is well established in the literature.
I have a different takeaway as you, though, that we only know about this effect—or about the biases people have and how to adjust their forecasts—because of work on forecasting. I don’t know how we’d know this stylized fact without work on it. For the wisdom of the crowds effect specifically, perhaps you could stop funding early since that one is well known, but it’s sufficiently surprising to most people that there could be value in showing it for more domains, and it is really just one example of what we learn more generally from research on forecasting—and these other results on how to optimally weigh forecasts can shrink error much more even after taking the wisdom of the crowds effect into consideration. (In our work, WoC gets you a ~60% reduction in the MSE, but other small adjustments lead to an improvement of an additional ~60% reduction in error compared to the WoC estimate, and those aren’t even all the improvements we can make.)
Today I would never run an experiment without using forecasts to help with power calculations. And there is very recent work I’d use to adjust those forecasts, and we’re collectively not near the optimum in terms of learning what we can learn to make more accurate forecasts or integrating them into workflows. As I said elsewhere in the comments, the claims in the OP are far too strong. Even your asking a few experts—that’s something that could be improved on and integrated into workflows and is part of the titular “forecasting”. (It reads to me kind of like: don’t do forecasting, do this other thing which is itself forecasting and is informed by and improved upon by… forecasting.)
A more defensible claim imo would be that there are some projects that are self-supporting and those should not be funded, or that in some but not all cases if the market doesn’t pay for it then it’s not valuable (abstracting from coordination failures and other market failures, or the externalities of basic research).
I have at least three reasons to be hopeful:
I see forecasting catching on with researchers for experimental design, which could easily save a lot of money and help make more progress. Earlier this month we updated a working paper on forecasts using data from the Social Science Prediction Platform to explicitly include results demonstrating use in power calculations. If a year from now forecasts are used a similar amount to now in economics research then that would be evidence for your hypothesis but from my perspective the concept that forecasts could be used in this way has only just started to be socialized, at least in my field. I also personally know of at least a couple of large institutions seeking forecasts and am planning a RCT on how they affect decision-making in the field.
I think LLMs are making forecasting much cheaper and easier.
If humans don’t take up the use of forecasts in decision-making as much as they “should”, well, LLMs may be more likely to in their own pipelines.
That’s not to say that every project previously funded around forecasting was a good use of money. I would probably agree with you regarding most of the projects you have in mind, while disagreeing with the title and framing which is way too broad.
AI-foward econ pre-doc positions
Some things it could be other than hedonic adaptation (non-exhaustive):
- People being disappointed that it’s not making a bigger difference to their lives. Most experts thought there would be somewhat larger effects, so it’s plausible participants did as well and are already starting to be disappointed by year 2.
- Expectations more generally not lining up to reality. For example, when someone leaves a job, we ask them how likely they think it is that they will find another acceptable job within 6 months. Almost everyone says they think they will find such a job, but then only about a third of people do in that timeframe. (Having a harder time finding a good job than anticipated is exceptionally common in all walks of life.)
- If you have money to do more things and then start to do more things, some of those things are not going to work out for you and you may have unexpected expenses. For example, you buy a car, and someone breaks into it. You move housing units, and it costs more than you expected to move or the gas bill is too high.
- There is a lot of churn in who is low-income, and while the transfers represent 40% of baseline household income, income rises in both the treatment and control group over the course of the study. So the transfers represent a relatively smaller boost in years 2 and 3. (Still, you might think that a 40% boost from baseline would affect well-being for longer.)
- Dynamic effects on other outcomes. For example, over time, fewer people work. In general, leisure is a normal good, and I think that is also the case here, by revealed preference. However, people could choose the wrong amount of it for themselves (e.g., through over-optimism in job search) and not working could lead to unexpected expenses biting harder.
- The earliest part of the study coincided with the brunt of the pandemic. Maybe it’s easier to make people happier with cash during a pandemic (though note the effect could also go the opposite direction, given it was also easier to get other cash payments during this time).
One thing that is interesting is that well-being seems to be trending negative before the end of the study—not significantly, but still a bit concerning. If the study had only lasted one year, we might have incorrectly concluded the transfers made people durably better-off in terms of these subjective well-being measures. If the study lasted longer, though, would we see a continued decline, or are the negative point estimates just before the end a function of the transfers ending soon? (E.g., via regret or a negative anticipation effect?)
I still think hedonic adaptation is a very strong contender to explain the patterns we observe, because it shows up all over the place in the broader literature, to the point where if a study doesn’t find an effect like that I wonder why. But, there are other things it could be. Either way, one might still be surprised the effects on subjective well-being aren’t larger than they are, even in year 1. And, it’s always possible to think the transfers made a meaningful difference to people’s lives even if they did not show up in subjective well-being measures.
(Preference utilitarians might have a slightly easier time than hedonistic utilitarians here: while preference utilitarianism is more associated with life satisfaction or domain satisfaction and hedonistic utilitarianism more associated with affect balance, and we don’t see lasting positive impacts on any of those measures, the transfers do enable more choice and people prefer more cash to less. We didn’t see anyone turn down the transfers.)
Escaping hedonic adaptation is hard
Alert on the Toner-Rodgers paper
Hiring pre-docs
I would agree that econ has the potential to have more impact than a development studies degree, but neither program is an econ program. (Maybe that specific MPP has a lot of econ content, but MPP programs in general do not, and if this one does I would not know.)
If you are going to work for an international organization, either the MPhil or MPP would be fine but the MPhil might open more doors through name recognition.
Alternatively, there are a few master’s programs out there that really focus on tech-ing people up. For example, USF has an applied economics and an IDE program that are well-regarded. A bunch of master’s programs are trying to distinguish themselves on quantitative skills and I don’t expect all of them to require an econ background, so maybe it’s worth looking around more.
Even working for a year and applying to more quantitative programs saves you a year over doing two masters.
I went from the MPhil in Development Studies straight to an Econ PhD, and I know a few other scattered cases, though it was a long time ago so I’m not sure if that path would work these days, and it was pretty unusual even then. It may be doable depending on your undergrad coursework and where you’re aiming—admissions committees care a lot more about having a math background than they care about having an econ background.
Question: are you looking to continue on to a PhD later, or go to the international organization / non-profit sector, or something else? For most work in international development you don’t need a PhD and the MPhil would work just fine to get you in the door.
I wouldn’t assume the MPP would significantly help you get into the MPhil in Economics. And if you’re set on some kind of second master’s after the MPP, but not after the MPhil in Development Studies, the opportunity cost of delaying your career by 2 years would be much higher than the difference in the course costs.
Thanks, Ozzie! This is interesting. There could well be something there. Could you say more about what you have in mind?
Thanks for these questions. This probably falls on me to answer, though I am leaving GPI (I have tendered my resignation for personal reasons; Adam Bales will be Acting Director in the interim).
The funding environment is not as free as it was previously. That does suggest some recalibration and different decisions on the margin. However, I’m afraid your message paints a misleading picture and I’m concerned about the potential for inaccurate gossip. I won’t correct every inaccuracy, but funding was never as concentrated as your citation of OP’s website suggests. For one thing, they are not our only funder, grants support activities that can be many years out into the future, and their timeline does not correspond to when we applied for or received funds. For another example, the decision about the Global Priorities Fellowship was not taken due to money but due to a) the massive amounts of researcher time it took to run (which is not easily compensated for because it was focused at one time of year), b) a judgment that the program could run less frequently and still capture most of the benefits by raising the bar and being more selective. We had observed that—as is common in recruitment—the “returns” from the top few participants were much higher than the “returns” from the average participant. PhD students are in school for many years (in my field, economics, 6 years is common), and while in some of those years they may be more focused on their dissertations, running the program only occasionally still leaves ample opportunity to try to catch the students who might be a particularly good fit while they are in their PhD. Running it less frequently certainly implies lower monetary costs, but in this case that was a side benefit rather than the main consideration.
To return to your main question, the broadening of the agenda is a natural result of both a) broadening the team and b) trying to build an agenda for global priorities research that can inform external researchers. As we’ve engaged with more and more exceptional researchers at other institutions, it has shifted our overall strategy and the extent to which we try to produce research “in-house” vs. build and support an external network for global priorities research. This varies somewhat by discipline, but think of there as just being “stubs” for economics and psychology at GPI, with most of the work that we support done outside of it. I don’t mean “support” monetarily (though we have benefited from an active visitors program), but support with ideas and convenings and discussion. In the past few years, we have been actively expanding our external network, mostly in economics and psychology but also in philosophy, and we anticipate that this external engagement will be the main way through which we have impact. I can talk at length about the structural reasons why this approach makes sense in academia, but that is probably a discussion for another day. (Some considerations: faculty tend to be naturally spread out, and while one might like to have agglomeration effects, this happens more through workshops and the free exchange of ideas, or collaborations, because faculty are tied to institutions whose own incentives are to diversify. You can try to make progress with just focused work by postdocs and pre-docs, but that misses a huge swath of people, and even postdocs and pre-docs become faculty over time. In the long run, if you are being successful, the bulk of your impact has to come from external places. The fact this research is mostly done at other institutions now is a sign that global priorities research has matured.)
As a final note, consider the purpose of this research agenda. It takes time to write a good research agenda—we embarked on it when I arrived almost two years ago, and we quickly did some initial brainstorming, but then we continued on a longer and more thorough deliberative process, such as through working groups focused on exploring whether a certain topic appeared promising. In each of the growth areas you highlight—AI and psychology—we made a few hires. Those hires hit the ground running with their own research, but they also helped further develop and refine the agenda. Developing this agenda helped shape our internal priorities (though not every topic on the agenda is something we want to pursue ourselves), but the main purpose of this agenda is external. We wouldn’t have needed to put nearly so much effort into it if it were for internal use. The agenda is simultaneously forward-looking and naturally driven by the hires we already made.
Hope this helps. With apologies, I’m not likely to get into follow-ups as it takes a long time to respond.
There’s another point I don’t quite know how to put but I’ll give it a go.
Despite the comments above about having many ideas and getting feedback early about one’s projects—which both point to having and abandoning ideas quickly—there’s another sense in which actually what one needs is an ability to stick to things. And the good taste to be able to evaluate when to try something else and when to keep going. (This is less about specific projects and more about larger shifts like whether to stay in academia/a certain line of work at all.)
I feel like sometimes people get too much advice to abandon things early. It’s advice that has intuitive appeal (if you can’t pick winners, at least cut your losses early), and it’s good advice in a lot of situations. But my impression is that while there are some people who would do better failing faster, there are also some people who would do better if they were more patient. At least for myself, I started having more success when sticking with things for longer. The longer you stick to a thing, the more expertise you have in it. That may not matter in some fields, but it matters in academia.
Now, obviously, you want to be very selective about what you stick to. That’s where having good taste comes in. But I’d start by looking honestly at yourself and looking at people near you that you see doing well for themselves in your chosen field, and asking which side of the impatient-patient spectrum you fall on compared to them. Some people are too patient. Some people are too impatient. I was too impatient and improved with more patience, and for some people it’s the opposite. Which advice applies the most to you depends on your starting point and field, and of course the outside options.
For econ PhDs, I think it’s worth having a lot of ideas and discarding them quickly especially in grad school because a lot of them are bad at first, but I also think there are people who jump ship from an academic career too early, like when they are on the market or in the first few years after. I suspect this might be generally true in academia where expertise really, really matters and you need to make a long-term investment, but I can’t speak for certain about other academic fields beyond economics. And I’ve definitely met many academics who played it too safe for maximizing impact, and many people who didn’t leave quickly enough. What I’m trying to emphasize is that it’s possible to make mistakes in both directions and you should put effort into figuring out which type of error you personally are more likely to make.
Thanks. A quick, non-exhaustive list:
Get feedback early on. Talking to people can save a lot of time
You should have a very clear idea of why it is needed. Good ideas sound obvious after the fact
That’s not to say people won’t disagree with you. If your idea takes off you will need to have a thick skin
A super-easy way to have more impact is to collaborate with others. This doesn’t help for job market papers, where people tend to want to have solo-authored work. But you can get a lot more done collaborating with others and the outputs will be higher-quality, too
Apart from collaborating with people on the actual project, do what you can to get buy-in from other people who have no relationship to the project. Other people can magnify the impact in big ways and small
It can take a while before early-career researchers find a good idea. Have more ideas than you would think
UofT also has a lot of faculty you could tap into if you need support.