I think the crux boils down to you basically saying “we can’t be certain that it would be very dangerous, therefore we should build it to find out”. This, to me, it totally reckless when it comes to the stakes being extinction (we really do not want to be FAFO-ing this! Where is your security mindset?). You don’t seem to put much (any?) weight on lab leaks during training (as a result of emergent situational awareness). “Responsible Scaling” isanythingbut, in the situation we are now in.
Also, the disadvantages you mention around Goodharting make me think that the only sensible way to proceed is to just shut it all down.
You say that you disagree with Nora over alignment optimism, but then also that you “strongly disagree” with “the premise that if smarter-than-human AI is developed in the near future, then we almost surely die, regardless of who builds it” (Rob’s post). I think you are also way too optimistic about alignment work on it’s current trajectory actually leading to x-safety by saying this.
I think the crux boils down to you basically saying “we can’t be certain that it would be very dangerous, therefore we should build it to find out”. This, to me, it totally reckless when it comes to the stakes being extinction (we really do not want to be FAFO-ing this! Where is your security mindset?). You don’t seem to put much (any?) weight on lab leaks during training (as a result of emergent situational awareness). “Responsible Scaling” is anything but, in the situation we are now in.
Also, the disadvantages you mention around Goodharting make me think that the only sensible way to proceed is to just shut it all down.
You say that you disagree with Nora over alignment optimism, but then also that you “strongly disagree” with “the premise that if smarter-than-human AI is developed in the near future, then we almost surely die, regardless of who builds it” (Rob’s post). I think you are also way too optimistic about alignment work on it’s current trajectory actually leading to x-safety by saying this.