4. Cotra aims to predict when it will be possible for “a single computer program [to] perform a large enough diversity of intellectual labor at a high enough level of performance that it alone can drive a transition similar to the Industrial Revolution.”—that is a “growth rate [of the world economy of] 20%-30% per year if used everywhere it would be profitable to use”
Your scenario is premise 4 “Some deployed APS systems will be exposed to inputs where they seek power in unintended and high-impact ways (say, collectively causing >$1 trillion dollars of damage), because of problems with their objectives” (italics added).
Your bar is (much?) lower, so we should expect your scenario to come (much?) earlier.
Oh and:
4. Cotra aims to predict when it will be possible for “a single computer program [to] perform a large enough diversity of intellectual labor at a high enough level of performance that it alone can drive a transition similar to the Industrial Revolution.”—that is a “growth rate [of the world economy of] 20%-30% per year if used everywhere it would be profitable to use”
Your scenario is premise 4 “Some deployed APS systems will be exposed to inputs where they seek power in unintended and high-impact ways (say, collectively causing >$1 trillion dollars of damage), because of problems with their objectives” (italics added).
Your bar is (much?) lower, so we should expect your scenario to come (much?) earlier.