In this paper, Strubell & al (2019) outline the hidden cost of machine learning (from inception to training and fine tuning) and found emissions for 1 model is about 360 tCo2.
The highest estimate they find is for Neural Architecture Search, which they estimated as emitting 313 tons of C02 after training for over 30 years. This suggests to me that they’re using an inappropriate hardware choice! Additionally, the work they reference—here—does not seem to be the sort of work you’d expect to see widely used. Cars emit a lot of CO2 because everyone has one; most people have no need to search for new transformer architecture. The answers from one search could presumably be used for many applications.
Most of the models they train produce dramatically lower estimates.
I also don’t really understand how their estimates for renewable generation for the cloud companies are so low. Amazon say they were 50% renewable in 2018, but the paper only gives them 18% credit, and Google say they are CO2 neutral now. It makes sense that they should look quite efficient, given that cloud datacenters are often located near geothermal or similar power sources. This 18% is based on a Greenpeace report which I do not really trust.
Finally, I found this unintentionally very funny:
Academic researchers need equitable access to computation resources. Recent advances in available compute come at a high price not attainable to all who desire access. … . Limiting this style of research to industry labs hurts the NLP research community in many ways. … This even more deeply promotes the already problematic “rich get richer” cycle of research funding, where groups that are already successful and thus well-funded tend to receive more funding due to their existing accomplishments. Third, the prohibitive start-up cost of building in-house resources forces resource-poor groups to rely on cloud compute services such as AWS, Google Cloud and Microsoft Azure.
While these services provide valuable, flexible, and often relatively environmentally friendly compute resources …
This whole paragraph is totally different to the rest of the paper. It appears in the conclusion section, but isn’t really concluding from anything in the main body—it appears the authors simply wanted to share some left wing opinions at the end. But this ‘conclusion’ is exactly backwards—if training models is bad for the environment, it is good to prevent too many people doing it! And if cloud computing is more environmentally friendly than buying your own GPU, it is good that people are forced into using it!
Overall this paper was not very convincing that training models will be a significant driver of climate change. And there is compelling reason to be less worried about climate change than AGI. So I don’t think this was very convincing that the main AI risk concern is the secondary effect on climate change.
Yes my apologies! I’ve added the necessary corrections.
Thanks.
The highest estimate they find is for Neural Architecture Search, which they estimated as emitting 313 tons of C02 after training for over 30 years. This suggests to me that they’re using an inappropriate hardware choice! Additionally, the work they reference—here—does not seem to be the sort of work you’d expect to see widely used. Cars emit a lot of CO2 because everyone has one; most people have no need to search for new transformer architecture. The answers from one search could presumably be used for many applications.
Most of the models they train produce dramatically lower estimates.
I also don’t really understand how their estimates for renewable generation for the cloud companies are so low. Amazon say they were 50% renewable in 2018, but the paper only gives them 18% credit, and Google say they are CO2 neutral now. It makes sense that they should look quite efficient, given that cloud datacenters are often located near geothermal or similar power sources. This 18% is based on a Greenpeace report which I do not really trust.
Finally, I found this unintentionally very funny:
This whole paragraph is totally different to the rest of the paper. It appears in the conclusion section, but isn’t really concluding from anything in the main body—it appears the authors simply wanted to share some left wing opinions at the end. But this ‘conclusion’ is exactly backwards—if training models is bad for the environment, it is good to prevent too many people doing it! And if cloud computing is more environmentally friendly than buying your own GPU, it is good that people are forced into using it!
Overall this paper was not very convincing that training models will be a significant driver of climate change. And there is compelling reason to be less worried about climate change than AGI. So I don’t think this was very convincing that the main AI risk concern is the secondary effect on climate change.