Thanks for writing this. It feels useful to have these diffusion numbers. However, in practice, I am much less interested in “how much longer did it take for these other models to be built?” but “by how much did the release of GPT3 move these projects forward in time?”. An answer to the latter question seems to tell us more about what we can do about diffusion and whether publications have a counterfactual effect on timelines. What is your current best guess on the latter question? [sorry if I missed this somewhere]
Hi Charlotte—as you can imagine, estimating the latter is much more difficult due to reasoning about counterfactuals. But I do have some thoughts on it in this section of a post in the sequence.
I think the key claim you’d be looking for there is:
My best guess is that the knowledge of GPT-3’s existence sped up both DeepMind and Google’s work scaling up language models by six months (90% CI: 1–18 months). But I have not been able to distinguish whether this acceleration was driven by insider knowledge, or the publication of GPT-3, or the hype generated after publication, or some combination of those factors.
As my 90% confidence interval shows, I’m very uncertain, but I hope this helps.
Thanks for writing this. It feels useful to have these diffusion numbers. However, in practice, I am much less interested in “how much longer did it take for these other models to be built?” but “by how much did the release of GPT3 move these projects forward in time?”. An answer to the latter question seems to tell us more about what we can do about diffusion and whether publications have a counterfactual effect on timelines. What is your current best guess on the latter question? [sorry if I missed this somewhere]
Hi Charlotte—as you can imagine, estimating the latter is much more difficult due to reasoning about counterfactuals. But I do have some thoughts on it in this section of a post in the sequence.
I think the key claim you’d be looking for there is:
As my 90% confidence interval shows, I’m very uncertain, but I hope this helps.