I think the distance between our current understanding of AI safety and the required one is of similar order of magnitude to the distance between invention of Dirac sea in 1930 and discovery of asymptotic freedom in non-Abelian gauge theory in 1973. This is 43 years of well-funded research by the top minds of mankind. And that without taking into account the engineering part of the project.
If remaining time frame for solving FAI is 25 years than:
We’re probably screwed anyway
We need invest all possible effort into FAI since the tail the probability distribution is probably fast falling
On the other hand, my personal estimate regarding time to human level AI is about 80 years. This is still not that long.
Could you say something about why your subjective probability distribution for the difficulty is so tight? I think it is very hard to predict in advance how difficult these problems are; witness the distribution of solution times for Hilbert’s problems.
Even if you’re right, I think that says that we should try to quickly get to the point with a serious large programme. It’s not clear that the route to that means doing focusing on direct work at the margin now. It will involve some, but mostly because of the instrumental benefits in helping increase the growth of people working on it, and because it’s hard to scale up later overnight.
My distribution isn’t tight, I’m just saying there is a significant probability of large serial depth. You are right that much of the benefit of current work is “instrumental”: interesting results will convince other people to join the effort.
I think the distance between our current understanding of AI safety and the required one is of similar order of magnitude to the distance between invention of Dirac sea in 1930 and discovery of asymptotic freedom in non-Abelian gauge theory in 1973. This is 43 years of well-funded research by the top minds of mankind. And that without taking into account the engineering part of the project.
If remaining time frame for solving FAI is 25 years than:
We’re probably screwed anyway
We need invest all possible effort into FAI since the tail the probability distribution is probably fast falling
On the other hand, my personal estimate regarding time to human level AI is about 80 years. This is still not that long.
Could you say something about why your subjective probability distribution for the difficulty is so tight? I think it is very hard to predict in advance how difficult these problems are; witness the distribution of solution times for Hilbert’s problems.
Even if you’re right, I think that says that we should try to quickly get to the point with a serious large programme. It’s not clear that the route to that means doing focusing on direct work at the margin now. It will involve some, but mostly because of the instrumental benefits in helping increase the growth of people working on it, and because it’s hard to scale up later overnight.
My distribution isn’t tight, I’m just saying there is a significant probability of large serial depth. You are right that much of the benefit of current work is “instrumental”: interesting results will convince other people to join the effort.