I’d say it’s a medium-sized deal. Academics can often propose ideas and show that they work on smaller (eg 7b) models. However, then it requires someone with a larger compute budget to like the idea & results and implement it at a larger scale.
There are some areas where access to compute is less important like MechInterp, RedTeaming, creating Benchmarks or more theoretical areas of AI research. Areas are more amenable to academic research if they don’t require training Frontier models. Eg inference or small fine-tuning runs on Frontier Models are actually not super expensive and can be done by academic labs. Also some areas of research can be done well on smaller models (eg MechInterp), so it’s fine if your uni doesn’t have so many GPUs.
But my experience (and also that of some others I know) was that I would regularly think of experiments or research ideas that I didn’t end up running or pursuing because I didn’t think I had the necessary compute.
I’d say it’s a medium-sized deal. Academics can often propose ideas and show that they work on smaller (eg 7b) models. However, then it requires someone with a larger compute budget to like the idea & results and implement it at a larger scale.
There are some areas where access to compute is less important like MechInterp, RedTeaming, creating Benchmarks or more theoretical areas of AI research. Areas are more amenable to academic research if they don’t require training Frontier models. Eg inference or small fine-tuning runs on Frontier Models are actually not super expensive and can be done by academic labs. Also some areas of research can be done well on smaller models (eg MechInterp), so it’s fine if your uni doesn’t have so many GPUs.
But my experience (and also that of some others I know) was that I would regularly think of experiments or research ideas that I didn’t end up running or pursuing because I didn’t think I had the necessary compute.