It’s hard to imagine a newsletter that could have picked out that paper at the time as among the most important of the hundreds included. For comparison, I think probably that at the time, there was much more hype and discussion of Hinton and students’ capsule nets (also had a NIPS 2017 paper).
People at the time thought it was a big deal: https://twitter.com/Miles_Brundage/status/1356083229183201281 Even the ones who were not saying it would be “radically new” or “spicy” or “this is going to be a big deal” or a “paradigm shift” were still at least asking if it might be (out of all the hundreds of things they could have been asking about but weren’t).
Incidentally, I don’t know if I count, but “Attention Is All You Need” was in my June 2017 newsletter & end-of-year best-of list (and capsule nets were not—I didn’t like them, and still don’t, it struck me as overly-hardwired and inflexible compared to existing attention methods even prior to Transformers, hardware-unfriendly, weak on toy problems, and essentially something only of interest because Hinton had been hinting at or talking about it for years; my opinion of CapsuleNets has not improved since*). So, I don’t find it hard to imagine a newsletter doing it because I did it myself.
* eg as of April 2024, despite 5600+ citations, I still have found no reason to ever cite CapsuleNets on gwern.net.
Wow, that certainly is more “attention” than I remember at the time.
I think filtering on that level of hype alone would still leave you reading way too many papers.
But I can see that it might be more plausible for someone with good judgment + finger on the pulse to do a decent job predicting what will matter (although then maybe that person should be doing research themselves).
People at the time thought it was a big deal: https://twitter.com/Miles_Brundage/status/1356083229183201281 Even the ones who were not saying it would be “radically new” or “spicy” or “this is going to be a big deal” or a “paradigm shift” were still at least asking if it might be (out of all the hundreds of things they could have been asking about but weren’t).
Incidentally, I don’t know if I count, but “Attention Is All You Need” was in my June 2017 newsletter & end-of-year best-of list (and capsule nets were not—I didn’t like them, and still don’t, it struck me as overly-hardwired and inflexible compared to existing attention methods even prior to Transformers, hardware-unfriendly, weak on toy problems, and essentially something only of interest because Hinton had been hinting at or talking about it for years; my opinion of CapsuleNets has not improved since*). So, I don’t find it hard to imagine a newsletter doing it because I did it myself.
* eg as of April 2024, despite 5600+ citations, I still have found no reason to ever cite CapsuleNets on gwern.net.
Wow, that certainly is more “attention” than I remember at the time.
I think filtering on that level of hype alone would still leave you reading way too many papers.
But I can see that it might be more plausible for someone with good judgment + finger on the pulse to do a decent job predicting what will matter (although then maybe that person should be doing research themselves).