If you only hire people who you believe are intellectually committed to short AGI timelines (and who wonât change their minds given exposure to new evidence and analysis) to work in AGI forecasting, how can you do good AGI forecasting?
One of the co-founders of Mechanize, who formerly worked at Epoch AI, says he thinks AGI is 30 to 40 years away. That was in this video from a few weeks ago on Epoch AIâs YouTube channel.
He and one of his co-founders at Mechanize were recently on Dwarkesh Patelâs podcast (note: Dwarkesh Patel is an investor in Mechanize) and I didnât watch all of it but it seemed like they were both arguing for longer AGI timelines than Dwarkesh believes in.
I also disagree with the shortest AGI timelines and found it refreshing that within the bubble of people who are fixated on near-term AGI, at least a few people expressed a different view.
I think if you restrict who you hire to do AGI forecasting based on strong agreement with a predetermined set of views, such as short AGI timelines and views on AGI alignment and safety, then you will just produce forecasts that re-state the views you already decided were the correct ones while you were hiring.
I wasnât suggesting only hiring people who believe in short-timelines. I believe that my original post adequately lays out my position, but if any points are ambiguous, feel free to request clarification.
I donât know how Epoch AI can both âhire people with a diversity of viewpoints in order to counter biasâ and ensure that your former employees wonât try to âcash in on the AI boom in an acceleratory wayâ. These seem like incompatible goals.
I think Epoch has to either:
Accept that people have different views and will have different ideas about what actions are ethical, e.g., they may view creating an AI startup focused on automating labour as helpful to the world and benign
or
Only hire people who believe in short AGI timelines and high AGI risk and, as a result, bias its forecasts towards those conclusions
Presumably there are at least some people who have long timelines, but also believe in high risk and donât want to speed things up. Or people who are unsure about timelines, but think risk is high whenever it happens. Or people (like me) who think X-risk is low* and timelines very unclear, but even a very low X-risk is very bad. (By very low, I mean like at least 1 in 1000, not 1 in 1x10^17 or something. I agree it is probably bad to use expected value reasoning with probabilities as low as that.)
I think you are pointing at a real tension though. But maybe try to see it a bit from the point of view of people who think X-risk is real enough and raised enough by acceleration that acceleration is bad. Itâs hardly going to escape their notice that projects at least somewhat framed as reducing X-risk often end up pushing capabilities forward. They donât have to be raging dogmatists to worry about this happening again, and itâs reasonable for them to balance this risk against risks of echo chambers when hiring people or funding projects.
*Iâm less surely merely catastrophic biorisk from human misuse is low sadly.
Why donât we ask ChatGPT? (In case youâre wondering, Iâve read every word of this answer and I fully endorse it, though I think there are better analogies that the journalism example ChatGPT used).
Hopefully, this clarifies a possible third option (one that my original answer was pointing at).
I think there is a third option, though itâs messy and imperfect. The third option is to:
Maintain epistemic pluralism at the level of research methods and internal debate, while being selective about value alignment on key downstream behaviors.
In other words:
You hire researchers with a range of views on timelines, takeoff speeds, and economic impacts, so long as they are capable of good-faith engagement and epistemic humility.
But you also have clear social norms, incentives, and possibly contractual commitments around what counts as harmful conflict of interest â e.g., spinning out an acceleratory startup that would directly undermine the mission of your forecasting work.
This requires drawing a distinction between research belief diversity and behavioral alignment on high-stakes actions. Thatâs tricky! But itâs not obviously incoherent.
The key mechanism that makes this possible (if it is possible) is something like:
âWe donât need everyone to agree on the odds of doom or the value of AGI automation in theory. But we do need shared clarity on what types of action would constitute a betrayal of the mission or a dangerous misuse of privileged information.â
So you can imagine hiring someone who thinks timelines are long and AGI risk is overblown but who is fully on board with the idea that, given the stakes, forecasting institutions should err on the side of caution in their affiliations and activities.
This is analogous to how, say, journalists might disagree about political philosophy but still share norms about not taking bribes from the subjects they cover.
Caveats and Challenges:
Enforceability is hard. Noncompetes are legally dubious in many jurisdictions, and âcash in on the AI boomâ is vague enough that edge cases will be messy. But social signaling and community reputation mechanisms can still do a lot of work here.
Self-selection pressure remains. Even if you say youâre open to diverse views, the perception that Epoch is âaligned with x-risk EAsâ might still screen out applicants who donât buy the core premises. So you risk de facto ideological clustering unless you actively fight against that.
Forecasting bias could still creep in via mission alignment filtering. Even if you welcome researchers with divergent beliefs, if the only people willing to comply with your behavioral norms are those who already lean toward the doomier end of the spectrum, your epistemic diversity might still collapse in practice.
Summary:
The third option is:
Hire for epistemic virtue, not belief conformity, while maintaining strict behavioral norms around acceleratory conflict of interest.
Itâs not a magic solution â it requires constant maintenance, good hiring processes, and clarity about the boundaries between âintellectual disagreementâ and âmission betrayal.â But I think itâs at least plausible as a way to square the circle.â
So, you want to try to lock in AI forecasters to onerous and probably illegal contracts that forbid them from founding an AI startup after leaving the forecasting organization? Who would sign such a contract? This is even worse than only hiring people who are intellectually pre-committed to certain AI forecasts. Because it goes beyond a verbal affirmation of their beliefs to actually attempting to legally force them to comply with the (putative) ethical implications of certain AI forecasts.
If the suggestion is simply promoting âsocial normsâ against starting AI startups, well, that social norm already exists to some extent in this community, as evidenced by the response on the EA Forum. But if the norm is too weak, it wonât prevent the undesired outcome (the creation of an AI startup), and if the norm is too strong, I donât see how it doesnât end up selecting forecasters for intellectual conformity. Because non-conformists would not want to go along with such a norm (just like they wouldnât want to sign a contract telling them what they can and canât do after they leave the forecasting company).
If you only hire people who you believe are intellectually committed to short AGI timelines (and who wonât change their minds given exposure to new evidence and analysis) to work in AGI forecasting, how can you do good AGI forecasting?
One of the co-founders of Mechanize, who formerly worked at Epoch AI, says he thinks AGI is 30 to 40 years away. That was in this video from a few weeks ago on Epoch AIâs YouTube channel.
He and one of his co-founders at Mechanize were recently on Dwarkesh Patelâs podcast (note: Dwarkesh Patel is an investor in Mechanize) and I didnât watch all of it but it seemed like they were both arguing for longer AGI timelines than Dwarkesh believes in.
I also disagree with the shortest AGI timelines and found it refreshing that within the bubble of people who are fixated on near-term AGI, at least a few people expressed a different view.
I think if you restrict who you hire to do AGI forecasting based on strong agreement with a predetermined set of views, such as short AGI timelines and views on AGI alignment and safety, then you will just produce forecasts that re-state the views you already decided were the correct ones while you were hiring.
I wasnât suggesting only hiring people who believe in short-timelines. I believe that my original post adequately lays out my position, but if any points are ambiguous, feel free to request clarification.
I donât know how Epoch AI can both âhire people with a diversity of viewpoints in order to counter biasâ and ensure that your former employees wonât try to âcash in on the AI boom in an acceleratory wayâ. These seem like incompatible goals.
I think Epoch has to either:
Accept that people have different views and will have different ideas about what actions are ethical, e.g., they may view creating an AI startup focused on automating labour as helpful to the world and benign
or
Only hire people who believe in short AGI timelines and high AGI risk and, as a result, bias its forecasts towards those conclusions
Is there a third option?
Presumably there are at least some people who have long timelines, but also believe in high risk and donât want to speed things up. Or people who are unsure about timelines, but think risk is high whenever it happens. Or people (like me) who think X-risk is low* and timelines very unclear, but even a very low X-risk is very bad. (By very low, I mean like at least 1 in 1000, not 1 in 1x10^17 or something. I agree it is probably bad to use expected value reasoning with probabilities as low as that.)
I think you are pointing at a real tension though. But maybe try to see it a bit from the point of view of people who think X-risk is real enough and raised enough by acceleration that acceleration is bad. Itâs hardly going to escape their notice that projects at least somewhat framed as reducing X-risk often end up pushing capabilities forward. They donât have to be raging dogmatists to worry about this happening again, and itâs reasonable for them to balance this risk against risks of echo chambers when hiring people or funding projects.
*Iâm less surely merely catastrophic biorisk from human misuse is low sadly.
Why donât we ask ChatGPT? (In case youâre wondering, Iâve read every word of this answer and I fully endorse it, though I think there are better analogies that the journalism example ChatGPT used).
Hopefully, this clarifies a possible third option (one that my original answer was pointing at).
So, you want to try to lock in AI forecasters to onerous and probably illegal contracts that forbid them from founding an AI startup after leaving the forecasting organization? Who would sign such a contract? This is even worse than only hiring people who are intellectually pre-committed to certain AI forecasts. Because it goes beyond a verbal affirmation of their beliefs to actually attempting to legally force them to comply with the (putative) ethical implications of certain AI forecasts.
If the suggestion is simply promoting âsocial normsâ against starting AI startups, well, that social norm already exists to some extent in this community, as evidenced by the response on the EA Forum. But if the norm is too weak, it wonât prevent the undesired outcome (the creation of an AI startup), and if the norm is too strong, I donât see how it doesnât end up selecting forecasters for intellectual conformity. Because non-conformists would not want to go along with such a norm (just like they wouldnât want to sign a contract telling them what they can and canât do after they leave the forecasting company).