Practical/âeconomic reasons why companies might not want to build AGI systems
(Originally posted on the EA Corner Discord server.)
First, most companies that are using ML or data science are not using SOTA neural network models with a billion parameters, at least not directly; theyâre using simple models, because no competent data scientist would use a sophisticated model where a simpler one would do. Only a small number of tech companies have the resources or motivation to build large, sophisticated models (here Iâm assuming, like OpenAI does, that model size correlates with âsophisticated-nessâ).
Second, increasing model size has diminishing returns with respect to model performance. Scaling laws usually relate model size to training loss via a power law, so every doubling of model size results in a smaller increase in training performance. And this is training performance, which is not the same as test set performanceâincreases in training performance above a certain threshold are considered not to matter for the modelâs ultimate performance. (This is why techniques like early stopping existâyou just stop training the model once its true performance stops increasing.)
(Counterpoint: Software systems typically have superstar economicsâe.g. the best search engine is 100x more profitable than the second-best search engine. So there could be a non-linear relationship between model performance and profitability, such that increasing a modelâs performance from 97% to 98% makes a huge difference in profits whereas going from 96% to 97% does not.)
Thirdâand this reason only applies to AGI, not powerful narrow AIsâitâs not clear to me how you would design an engineering process to ensure that an AGI system can perform multiple tasks very well and generalize to new tasks. Typically, when we design software, we create a test suite that evaluates its suitability for the tasks for which itâs designed. Before releasing a new version of an AI system, we have to run the entire test suite on it and make sure it passes. Itâs obviously easier to design a test suite for an AI that is designed to do a few tasks well than for an AI thatâs supposed to be able to do any task. (On the flip side, this means that anyone seeking to design an AGI would have to design a way to test it to ensure that itâs (1) actually an AGI and (2) performant.) (While generality isnât strictly necessary for AIs to be dangerous, I believe many of us would agree that AGIs are more dangerous as x-risks than narrow AIs.)
Fourth, setting out to create AGI would have a huge opportunity cost. Yes, technically, humans are probably not the absolute smartest, most capable beings that evolution could have built, but that doesnât mean that building a smarter AGI machine would be profitable. It seems to me that humans have a comparative advantage in planning etc. while âtechnology as a wholeâ will have a comparative advantage in e.g. doing machine vision at scale. So most firms ought to just hire a bunch of humans and design/âpurchase technological systems that complement humansâ skill sets (this is a common idea about how future AI development will go, called âintelligence augmentationâ).
Practical/âeconomic reasons why companies might not want to build AGI systems
(Originally posted on the EA Corner Discord server.)
First, most companies that are using ML or data science are not using SOTA neural network models with a billion parameters, at least not directly; theyâre using simple models, because no competent data scientist would use a sophisticated model where a simpler one would do. Only a small number of tech companies have the resources or motivation to build large, sophisticated models (here Iâm assuming, like OpenAI does, that model size correlates with âsophisticated-nessâ).
Second, increasing model size has diminishing returns with respect to model performance. Scaling laws usually relate model size to training loss via a power law, so every doubling of model size results in a smaller increase in training performance. And this is training performance, which is not the same as test set performanceâincreases in training performance above a certain threshold are considered not to matter for the modelâs ultimate performance. (This is why techniques like early stopping existâyou just stop training the model once its true performance stops increasing.)
(Counterpoint: Software systems typically have superstar economicsâe.g. the best search engine is 100x more profitable than the second-best search engine. So there could be a non-linear relationship between model performance and profitability, such that increasing a modelâs performance from 97% to 98% makes a huge difference in profits whereas going from 96% to 97% does not.)
Thirdâand this reason only applies to AGI, not powerful narrow AIsâitâs not clear to me how you would design an engineering process to ensure that an AGI system can perform multiple tasks very well and generalize to new tasks. Typically, when we design software, we create a test suite that evaluates its suitability for the tasks for which itâs designed. Before releasing a new version of an AI system, we have to run the entire test suite on it and make sure it passes. Itâs obviously easier to design a test suite for an AI that is designed to do a few tasks well than for an AI thatâs supposed to be able to do any task. (On the flip side, this means that anyone seeking to design an AGI would have to design a way to test it to ensure that itâs (1) actually an AGI and (2) performant.) (While generality isnât strictly necessary for AIs to be dangerous, I believe many of us would agree that AGIs are more dangerous as x-risks than narrow AIs.)
Fourth, setting out to create AGI would have a huge opportunity cost. Yes, technically, humans are probably not the absolute smartest, most capable beings that evolution could have built, but that doesnât mean that building a smarter AGI machine would be profitable. It seems to me that humans have a comparative advantage in planning etc. while âtechnology as a wholeâ will have a comparative advantage in e.g. doing machine vision at scale. So most firms ought to just hire a bunch of humans and design/âpurchase technological systems that complement humansâ skill sets (this is a common idea about how future AI development will go, called âintelligence augmentationâ).