Thanks for thisāI really appreciated how clear and focused-on-actual-takeaways (rather than āIāll discuss X, Y, Z topicā) this summary is.
OpenAI observed in 2018 that since 2012 the amount of compute used in the largest AI training runs has been doubling every 3.4 months.
In our updated analysis (n=57, 1957 to 2021), we observe a doubling time of 6.2 months between 2012 and mid-2021.
What is n counting here? Like studies included in the lit review?
Why is n referring to things from 1957-2021 if the conclusion given is just about 2012-2021? It seems like the relevant n would be just the items about 2012-2021?
Is the difference between your estimate and OpenAIās because 2018-2021 involved much slower doubling times (perhaps three times as long?), because OpenAI missed or misinterpreted some things that happened in 2012-2018, because youāre including pre-2012 things and those involved slower doubling times, or some mix of those factors?
(Probably I can find the answers in your next post, but maybe this is worth clarifying here too?)
n is counting the number of ML systems in the analysis at the point of writing. (We have added more systems in the meantime). An example for such a system is GPT-3, AlphaFold, etc. - basically a row in our dataset.
Right, good point. Iāll add the number of systems for the given time period.
Thatās hard to answer. I donāt think OpenAI misinterpreted anything. For the moment, I think itās probably a mixture of:
the inclusion criteria for the systems on which we base this trend
actual slower doubling times for reasons which we should figure out
Nonetheless, as outlined in Part 1 - Section 2.3, I did not interpret those trends yet but Iām interested in a discussion and trying to write up my thoughts on this in the future.
Thanks for thisāI really appreciated how clear and focused-on-actual-takeaways (rather than āIāll discuss X, Y, Z topicā) this summary is.
What is n counting here? Like studies included in the lit review?
Why is n referring to things from 1957-2021 if the conclusion given is just about 2012-2021? It seems like the relevant n would be just the items about 2012-2021?
Is the difference between your estimate and OpenAIās because 2018-2021 involved much slower doubling times (perhaps three times as long?), because OpenAI missed or misinterpreted some things that happened in 2012-2018, because youāre including pre-2012 things and those involved slower doubling times, or some mix of those factors?
(Probably I can find the answers in your next post, but maybe this is worth clarifying here too?)
Thanks, Michael.
nis counting the number of ML systems in the analysis at the point of writing. (We have added more systems in the meantime). An example for such a system is GPT-3, AlphaFold, etc. - basically a row in our dataset.Right, good point. Iāll add the number of systems for the given time period.
Thatās hard to answer. I donāt think OpenAI misinterpreted anything. For the moment, I think itās probably a mixture of:
the inclusion criteria for the systems on which we base this trend
actual slower doubling times for reasons which we should figure out Nonetheless, as outlined in Part 1 - Section 2.3, I did not interpret those trends yet but Iām interested in a discussion and trying to write up my thoughts on this in the future.