Quote from the executive summary of the MIT Media Lab study, on page 3:
Despite $30–40 billion in enterprise investment into GenAI, this report uncovers a surprising result in that 95% of organizations are getting zero return. The outcomes are so starkly divided across both buyers (enterprises, mid-market, SMBs) and builders (startups, vendors, consultancies) that we call it the GenAI Divide. Just 5% of integrated AI pilots are extracting millions in value, while the vast majority remain stuck with no measurable P&L impact.
Quote from page 6:
The GenAI Divide is starkest in deployment rates, only 5% of custom enterprise
AI tools reach production. Chatbots succeed because they’re easy to try and flexible, but fail
in critical workflows due to lack of memory and customization. This fundamental gap
explains why most organizations remain on the wrong side of the divide.
Page 7:
The 95% failure rate for enterprise AI solutions represents the clearest manifestation of the GenAI Divide.
I agree that the authors encourage this misreading of the data by eg saying “95% of organizations are getting zero return” and failing to note the caveats listed in my comment. If you believe that this statement is referencing a different data set than the one I was quoting which doesn’t have those caveats, I’d be interested to hear it.
I’m saying that the authors summarized their findings without caveats, and that those caveats would dramatically change how most people interpret the results.
(Note that, despite the “MIT” name being attached, this isn’t an academic paper, and doesn’t seem to be trying to hold itself to those standards.)
I recommend emailing the authors and asking for clarification. I’ve done this more than once in the past when I’ve had thoughts about papers I’ve read and have gotten some extremely helpful, illuminating replies.
I always worry about bothering people, but I get the sense that, rather than being annoyed, people find it rewarding that anyone took an interest in their work, or at least don’t mind answering a quick email.
Quote from the executive summary of the MIT Media Lab study, on page 3:
Quote from page 6:
Page 7:
I agree that the authors encourage this misreading of the data by eg saying “95% of organizations are getting zero return” and failing to note the caveats listed in my comment. If you believe that this statement is referencing a different data set than the one I was quoting which doesn’t have those caveats, I’d be interested to hear it.
Are you saying the authors of the study are misreporting their own results?
I’m saying that the authors summarized their findings without caveats, and that those caveats would dramatically change how most people interpret the results.
(Note that, despite the “MIT” name being attached, this isn’t an academic paper, and doesn’t seem to be trying to hold itself to those standards.)
I recommend emailing the authors and asking for clarification. I’ve done this more than once in the past when I’ve had thoughts about papers I’ve read and have gotten some extremely helpful, illuminating replies.
I always worry about bothering people, but I get the sense that, rather than being annoyed, people find it rewarding that anyone took an interest in their work, or at least don’t mind answering a quick email.
6 months later, readers might be interested in this recent 80,000 Hours article about the 95% statistic.