I’m surprised that there hasn’t been an attempt (as far as I know) to fund/create a competitor to Epoch.ai.
It wouldn’t have to compete on all benchmarks, but it would be good to have a forecasting organisation that could be trusted with potentially dual use insights into capabilities trajectories. I don’t believe this would require uniformity of views: it would just require people with a proper sense of responsibility.
I also think that the bad judgement displayed by some of their employees impinges on some of their research (emphasis on some, particularly the more subjective elements, Epoch is still my go-to-source in many cases). Unfortunately, I think there’s a difference between being intelligent and being wise and one common way that this distinction plays out is that some quite intelligent folks follow the incentive gradient towards being excessively and reflexively contrarian. Just to be clear, I’m not trying to attack their research, just noting that whilst a second opinion would always have been valuable, the fact that I trust them less on the margin, makes the need for such a second opinion feel more pressing to me.
In terms of producing high-quality research, I’d orient to how Epoch has done many things well, but also made a few mistakes that I would controversially call clear mistakes.
I’m also pretty sure that there’s sufficient talent in the space now to create a second such effort. It could also start small and funders could help it scale if it proves itself.
Off hand, METR, Forethought, MIT Future Tech, AI Futures Project, AI Index, HAL, and Artificial Analysis all substantially overlap with us in our research focuses and other work, though no two orgs have exactly the same remit. The list of orgs and individual researchers whose work at least partially overlaps with us is far larger.
Personally I think there is a huge amount of descriptive and forecasting research to be done around AI, far more than any one organization can or should take on. I would welcome more “competitors” and I don’t want anyone who is interested in our research areas to feel like we have these topics “covered”. And I’m confident there are many good critiques to be made of our work and much better analyses to can be done on the questions we’ve tackled.
I’m surprised that there hasn’t been an attempt (as far as I know) to fund/create a competitor to Epoch.ai.
It wouldn’t have to compete on all benchmarks, but it would be good to have a forecasting organisation that could be trusted with potentially dual use insights into capabilities trajectories. I don’t believe this would require uniformity of views: it would just require people with a proper sense of responsibility.
I also think that the bad judgement displayed by some of their employees impinges on some of their research (emphasis on some, particularly the more subjective elements, Epoch is still my go-to-source in many cases). Unfortunately, I think there’s a difference between being intelligent and being wise and one common way that this distinction plays out is that some quite intelligent folks follow the incentive gradient towards being excessively and reflexively contrarian. Just to be clear, I’m not trying to attack their research, just noting that whilst a second opinion would always have been valuable, the fact that I trust them less on the margin, makes the need for such a second opinion feel more pressing to me.
In terms of producing high-quality research, I’d orient to how Epoch has done many things well, but also made a few mistakes that I would controversially call clear mistakes.
I’m also pretty sure that there’s sufficient talent in the space now to create a second such effort. It could also start small and funders could help it scale if it proves itself.
Off hand, METR, Forethought, MIT Future Tech, AI Futures Project, AI Index, HAL, and Artificial Analysis all substantially overlap with us in our research focuses and other work, though no two orgs have exactly the same remit. The list of orgs and individual researchers whose work at least partially overlaps with us is far larger.
Personally I think there is a huge amount of descriptive and forecasting research to be done around AI, far more than any one organization can or should take on. I would welcome more “competitors” and I don’t want anyone who is interested in our research areas to feel like we have these topics “covered”. And I’m confident there are many good critiques to be made of our work and much better analyses to can be done on the questions we’ve tackled.
Putting aside mechanize, what do you think are the mistakes Epoch has made?
I thought some of their analysis was weak. I made comments at the time, but unfortunately, I don’t have time at the moment to go back and find them.