Hi CAISID, thanks for letting me know that you think the podcast is interesting!
(For context to others, the original comment is referring to an aside that I give at 27:04 about “foundation models”)
I definitely did have a difficult time finding concrete answers regarding training costs of current cutting edge foundation models. The only numbers I could find from the big companies that were primary sources, i.e. from people within the company/press releases, were very loose numbers that included salaries, which basically makes them meaningless (at least, for the information I want out of them).
There has been some work done on estimating training costs (e.g., Epoch or the AI Index Report), but it seemed that I would need to spend a significant amount of time collecting data and even doing some forecasting to actually get approximations for current state of the art.
Would love to hear your thoughts on this either here or whatever messaging format you prefer.
Hi CAISID, thanks for letting me know that you think the podcast is interesting!
(For context to others, the original comment is referring to an aside that I give at 27:04 about “foundation models”)
I definitely did have a difficult time finding concrete answers regarding training costs of current cutting edge foundation models. The only numbers I could find from the big companies that were primary sources, i.e. from people within the company/press releases, were very loose numbers that included salaries, which basically makes them meaningless (at least, for the information I want out of them).
There has been some work done on estimating training costs (e.g., Epoch or the AI Index Report), but it seemed that I would need to spend a significant amount of time collecting data and even doing some forecasting to actually get approximations for current state of the art.
Would love to hear your thoughts on this either here or whatever messaging format you prefer.