Thanks for your work here, it’s a useful overview for the compute metrics project I’m working on with Peter. Minor errors:
Also commonly used is Petaflop/s-day. It’s also a quantity of operations. A petaflop/s is 1015 floating point operations per second for one day. A day has 84,400seconds≈1015. That makes 1020 FLOPs.
For an NVIDIA A100, the on-board memory bandwidth is around 2GB/s
I think this should be 2TB/s?
And ping!
We are working on a piece with more insights on the utilizations and some advice on how to estimate training compute and the connected utilization of the system (link to be added by the end of 2021; ping me if not).
Thanks for your work here, it’s a useful overview for the compute metrics project I’m working on with Peter. Minor errors:
A petaflop/s-day is 1015
A day has 10^5 seconds
I think this should be 2TB/s?
And ping!