You are a Lottery Ticket
(This is posted on the EA forum since a lot of altruists believe that entrepreneurship may be one of the most effective ways of earning to give.)
One of society’s classic mistakes is a belief that people’s success is due to their hard work and determination, when it’s really due to external factors. The startup world lionizes repeat successes like Elon Musk and Steve Jobs as evidence that certain people are more skilled at creating new companies than others, yet the academic literature on startups contains gems like
Human capital variables have limited impact on startup performance, and the few significant effects are split equally between enhancing and impeding performance[1]
In this post, I discuss the extent to which startup success depends on things outside of the founders’ control and what to do about it.
Part 1: The necessary role of luck
If you are like me, you’ve understood that entrepreneurship is more risky than dependent employment, but you’ve never really thought about why. There’s no law which says that entrepreneurship has to be risky, so what’s going on? Are VCs just getting together and deciding to pay entrepreneurs based on a dice roll? The answer is: basically, yes.
If a founder leaves a startup, that company is in much more dire straits than if an entry-level employee leaves IBM. This means that startup founders need to signal their commitment to investors much more strongly than entry-level employees do; that signaling usually takes the form of the founder rejecting a salary and instead gambling their finances on the company success.[2] Because salaries are much more stable than business values, this means that entrepreneurs are forced by investors to accept higher levels of risk.
This is important so I’m going to call it out in very clear terms:
Entrepreneurship must involve an element of risk for the founder (i.e. luck). If it didn’t, no one would invest.
The Importance of Industry Selection
The greatest startups seem to create markets out of thin air: no one cared about search until Google and only petty criminals ran non-medallioned taxis before Uber. But in reality Google couldn’t have been successful without the dot-com boom, nor could Uber have gotten traction if smartphone use was low – factors completely outside the founders’ control.
This is true more broadly: studies have found that the strongest predictor of a startup’s success is whether they are in a growth industry, rather than the inherent ability of founders (beyond their ability to select a growth industry).[3]
My favorite example of this is a paper entitled “How much does industry matter, really?”[4] In which the authors performed a linear regression on companies’ profitability which included a coefficient for every single company (i.e. it was insanely biased towards finding that individual performance matters). This lead to the following conclusions:
Your industry’s performance explains 19% of your company’s performance.
Everything else about your company combined (growth, employees, IP,…) explains 32% of performance.
This means that if you are in a growth industry you can do a lot of things wrong and still end up with a better company than an awesome team with great traction, ironclad intellectual property and a crystal-clear value proposition if that second company’s industry happens to take a downturn.
This is not to say that all entrepreneurs are created equal, because clearly they are not. But the point is that skill in entrepreneurship is less about being able to steer your ship through stormy waters and more about checking the weather before you set off.
Example: Amazon
Amazon’s success is often attributed to the frugality of its founder, Jeff Bezos, including tidbits like that their desks were initially made out of office doors, and that they removed all the light bulbs from their vending machines to save on electricity.
But the actual story of Amazon’s founding is more interesting. Jeff had heard two interesting things in 1994:
Internet usage was rapidly growing, and
The Supreme Court had just ruled that online retailers don’t have to collect sales tax
He guessed that this would mean an increase in e-commerce, and his guess was right.
Importantly, he didn’t think that he personally had specific talents in e-commerce (his background was in finance), nor did he start a company to pursue his passion of selling books online. He took the outside view that e-commerce in general was going to be successful, and hitched himself to that wagon.
Conclusion of Part One
Some people invest in market size and market potential and the idea itself. We start with the people first. We think the ideas that entrepreneurs start with evolve and change dramatically from the beginning and sometimes end up unrecognizable, so we believe in investing in the people. - Ron Conway
The idea that there are “superstar” entrepreneurs with a Midas touch is just factually wrong. For a long time, evidence indicated that there was no such thing as entrepreneurial skill at all, and while larger data sets are starting to change that perception, even the most “pro-skill” papers find skillful entrepreneurs to have only modest benefits over their less skilled counterparts (for example, increasing the chance of success from 20% to 30%).[5]
It’s not surprising that people believe individual performance matters even when it doesn’t; this error is so common that it’s known as the “fundamental” attribution error and interested readers can view the Wikipedia page for a list of ridiculous circumstances in which people attribute performance to internal characteristics even when there’s absolutely no way internal characteristics could be involved.
One of the differentiators between entrepreneurs and everyone else is that entrepreneurs have a stronger illusion of control, believing that they can accurately predict things like when competitors will enter the market.[6] This causes the startup community to be obsessed with topics like growth hacking, building “culture”, and financing. Those are all important, but there is no question that an hour spent looking at sample investment documents would be better spent looking at market projections.
Part Two: What to Do About It
Reference class forecasting
Humans so frequently attribute success to internal characteristics instead of external ones that this bias is known as the “fundamental attribution error”. Because it’s so frequent, there’s a huge amount of research on how to avoid it and one technique I found useful is known as “reference class forecasting”.
At a high level, this involves identifying companies that are similar to yours and then estimating your own likelihood of success based on those companies’ success rates.
For example, Amazon’s claim was that a new technology (the Internet) and new tax advantages would cause people to buy books in a new way. If previous tax cuts had caused huge spikes in book purchasing, or if other products had seen great sales on the Internet, this would cause us to increase our guess as to Amazon’s success. If it turned out that people loved buying books in person so that even when previous technologies like telephones spread no one bought books over the phone, this would decrease our estimate.
(There’s a cliché that startups describe themselves as “it’s like X for Y.” There’s even a site which generates these clichés. Reference class forecasting is basically taking that cliché and turning it into an investment strategy.)
Example: Amazon (Redux)
I gave empirical evidence before about the importance of industry selection, but reference class forecasting gives a strong intuition for it. Take our standard Amazon example:
Claim: Amazon was successful because its founder was frugal. Examination: make a reference class of all frugal founders. What fraction of them have billion-dollar companies?
Claim: Amazon was successful because its industry took off. Examination: make a reference class of all founders whose industries took off. What fraction of them have billion-dollar companies?
Eating my own dog food
My first idea for a company was predictive analytics around code quality. While I was at my previous job I had been able to show that certain coding techniques were associated with a greater number of defects, and I figured I could monetize this.
The outside view didn’t look great though. Static code analysis tools have existed for a long time, but few people use them as a core piece of the development process. There were no recent advances in IDEs or version control which I could use to explain why my tools will be more successful. Even though this idea met many of the standard startup idea criteria (I had domain expertise, it solved a problem I myself had, I was passionate about it), I abandoned it.
My current projects deals with quality measurement in healthcare setting. For those of you unfamiliar with American politics, health care reform is kind of a big deal and quality-based payments are expected to triple in the next three years. The market is clearly growing and while I think I have some advantages that my competitors lack, I think the outside view looks pretty good for all of us.
Summary
When I explain my unorthodox views on the importance of teams, people often say something like:
It’s better to have an A team with a B idea than a B team with an A idea.
My point isn’t that this is wrong; it’s that it doesn’t make sense.
Industry selection is one of the most, if not the most, important skills of a founding team. If the founding team did their market research, customer validation, etc. and still didn’t realize that their bad idea was bad, then in what sense are they a good founding team?[7]
Also: reference class forecasting is cool, and people should do more of it.
Lastly: if you want to work for a startup with an excellent reference class, we’re hiring.
Footnotes
See “Entrepreneurship: a game of poker, not roulette” for a different spin on roughly the same facts.
I would like to thank Brian Tomasik, Gina Stuessy and especially Ben Todd for comments on earlier versions of this article.
[1] Baum, Joel AC, and Brian S. Silverman. “Picking winners or building them? Alliance, intellectual, and human capital as selection criteria in venture financing and performance of biotechnology startups.” Journal of business venturing 19.3 (2004): 411-436.
[2] See Wikipedia for the general phenomenon, or e.g. Cumming, Douglas. “Adverse selection and capital structure: evidence from venture capital.” Entrepreneurship Theory and Practice 30.2 (2006): 155-183.
[3] Gartner, William B, Jennifer A Starr, and Subodh Bhat. “Predicting new venture survival: an analysis of “anatomy of a start-up.” cases from Inc. Magazine.” Journal of Business Venturing 14.2 (1999): 215-232.
[4] McGahan, Anita M., and Michael E. Porter. “How much does industry matter, really?.” (1997).
[5] Gompers, Paul, et al. Skill vs. luck in entrepreneurship and venture capital: Evidence from serial entrepreneurs. No. w12592. National Bureau of Economic Research, 2006. For a model which has reasonably accurate predictions but assumes that entrepreneurial success is entirely due to luck see: Kihlstrom, Richard, and Jean-Jacques Laffont, A General Equilibrium Entrepreneurial Theory of Firm Formation Based on Risk Aversion, Journal of Political Economy 87. 1979. 719-748.
[6] Simon, Mark, Susan M. Houghton, and Karl Aquino. “Cognitive biases, risk perception, and venture formation: How individuals decide to start companies.” Journal of business venturing 15.2 (2000): 113-134.
[7] Of course, you might argue that there are ex-ante good ideas which turn out to be bad, and ex-ante bad ideas which turn out to be good. To which I respond: exactly.
Isn’t this what you would see if your “human capital variables” just don’t have much predictive power? Their four human capital variables, described in section 4.3.3, seem to be the following: 1. number of people in management roles, 2. number of management roles held by the president, 3. number of prior startups which the founder was president of, 4. number of roles at other companies currently held by president.
I’m not too surprised that these factors had limited predictive ability, or that their sign was mixed. Factor 3 doesn’t seem to incorporate anything about the success of those startups, just their number.
I don’t understand the logic here. In any domain with principal agent problems, you expect the principal to pay the agent a fraction of the output, i.e. for the agent to retain equity. In fact, financial theory suggests that startup founders should take more equity when performance was more dependent on their behavior / skill, and should take less equity when performance is more out of their control. So the fact that startup founders receive variable payouts is direct evidence against the luck hypothesis.
I agree this why startup founders have such a wide spread in their compensation, but I don’t think it helps attribute that spread to luck vs. skill.
It seems like this is mostly a claim about what skills are useful, rather than about the importance of skill. The fact that smartphone use was high was not out of Uber’s control, as you point out, since they founded the businesses when it looked reasonable. So it’s strange to call it a “factor completely outside of the founder’s control.”
This looks like a surprisingly small effect! Even a 50-50 split would leave room for the best entrepreneurs to succeed with very high probability if dropped into a random industry. You say:
This may be true (though I have the opposite intuition), but even if so it seems almost guaranteed to be a smaller share of the variance—just because there is so much more variance.
More importantly, variation in performance of an industry has a huge impact on the returns to industry-specific capital, even in efficient-market land. But in efficient-market land, it’s not very important as a consideration about what startup to start, because there is more competition in industries that are poised to grow.
The study in question showed that entrepreneurs believe they are better at predicting things. It assumed that this was because they were more biased rather than being better predictors, but both explanations seem plausible. Similarly, it showed that entrepreneurs had better views of themselves, and assumed that was due to overconfidence rather than ability, but didn’t try to distinguish the two possibilities.
Thanks for the feedback Paul.
Factor 3′s non-significance tells us that, if there is such a thing as founding skill it’s not a skill which improves through practice. This certainly doesn’t disprove the skill hypothesis, but it’s a mark against.
For what it’s worth: anecdotally these are major factors that VCs look at, so it’s very surprising to people in the startup world that they don’t correlate with success. (To the extent that most investors bluntly tell me that they don’t believe the result.)
I don’t think I understand. To me a good formalization of “luck” is “ex-ante variance” – I think you disagree?
Keep in mind that most of the variance is still unexplained; it’s not 19% industry and 81% founder.
I agree that under perfect competition profits are zero, and that this mechanism is enforced by increased competition in profitable sectors, but it seems weird to assume perfect competition here? I think I’m not understanding you again.
I don’t think this is a fair summary of their paper. For example, they asked people to give 90% confidence intervals for various things, and recorded how frequently the true answer was outside that range. This seems to me to be a legitimate definition of “overconfident”.
The authors also cite evidence that people’s answers to questions like “I could succeed at making this venture a success, even though many other managers would fail” is uncorrelated with their actual ability, although I agree that it’s not 100% clear.
I agree that this is a better measure of overconfidence. But it looks like startup founders did insignificantly better at this task, rather than doing worse at it. I may be misreading Table 1.
That said, I also agree that startup founders tend to be unrealistically optimistic about their own prospects (or at least answer survey questions in ways that are unreallistically optimistic).
Suppose that each startup’s valuation is a deterministic function of the founder’s skill (which is known to the founder). The we expect startup founders to be paid mostly in equity. And we could also see great variation in founder compensation. But I think we would agree that there is no luck in this case—it seems like exactly the kind of model you are trying to argue against.
I’m skeptical that the study has enough power to detect a realistic effect for this factor, given that it’s a small piece of human capital and all of the measures are extremely noisy.
I also have concerns about confounding. For example, prior foundings had the largest negative coefficient of any variable on VC funding. The author’s interpretation is: “presidents who have experienced entrepreneurial failures in the past may find it more difficult to obtain VC financing for future startups.” I don’t know if that’s the real explanation, but I’m pretty hesitant to make causal inferences in this case—whatever confounder produced that large negative effect, I expect it swamps a modest practice effect.
I expect the real world is somewhere in between perfect competition and no competition.
No, you are correct and I misread your statement. Entrepreneurs don’t appear to have this sort of overconfidence (at least according to this study’s results). They (we) appeared to have more self aggrandizing confidence.
Cool, I thought you might be making this point but I didn’t want to put words in your mouth.
It’s unquestionably true that part of the variance is explained by the fact that any jackass can call themselves an entrepreneur and therefore the set of “entrepreneurs” has a greater range of skills than e.g. the set of software developers. But I’m skeptical that this is the entire reason; to cite a fact from above: an entrepreneur who has a successful exit only has a 30% chance of success in their next venture. It seems unlikely that entrepreneurs’ skills decline rapidly after a successful exit.
I would agree with this more generally: there are probably modest skill effects, but they are incredibly hard to define and are swamped by the nosiness of startups. Especially once you look at “qualified” founders (e.g. those with venture backing), the skill differentiation explains a very small piece of the variance.
Quoth Gompers et al.:
“While it may be better to be lucky than smart, the evidence presented here indicates that being smart has value too.”
Hard to argue with that :-)
I’m still not entirely sure what lesson you were drawing from the efficient market hypothesis, but perhaps it doesn’t matter.
Thanks for engaging with objections, and sorry for being so critical!
It’s possible we just have a quantitative disagreement.
For example, I agree that there are very few people who could start a successful startup with high reliability. But quantitatively, I don’t know how much variation in skill there is and how important it is. I think the most compelling statistic here is the 30% IPO rate for second startup (given success) vs. 18% IPO rate for first startups. But that seems to me like a pretty big effect, so I’m not sure quite what to make of it.
I think you could just as well argue “Submitting an academic paper to a good conference is a lottery ticket.” (In fact, the numbers are comparable). In some sense this is true, but I still wouldn’t say “The idea that there are some excellent papers is just factually wrong.”
Thanks Paul for the feedback, and for the reminder that we are criticizing ideas and not people :-)
The thing with academic papers was really interesting, and gave me pause. I would point out two similarities:
The set of all papers submitted to a specific conference is a lot more homogeneous than the set of all papers period. Similarly, the set of all entrepreneurs who get VC funding is a lot more homogenous than the set of all people who think about starting companies. So the statement “performance within some limited subset is mostly due to chance” isn’t necessarily conflicting with the idea that there is such a thing as entrepreneurial skill/paper quality.
Instead of drawing a lesson that there is no such thing as skill we might conclude that acceptance to a conference or having an IPO is just not a very good indicator of skill.
I also agree that this is largely a quantitative disagreement. I’ve spent the last year being surrounded by people who believe that variance in startups is completely determined by the founder’s skill, and that gives me a framing for what I write.
My thoughts exactly—except “Your industry’s performance explains 19% of your company’s performance.
This looks like a surprisingly small effect! Even a 50-50 split would leave room for the best entrepreneurs to succeed with very high probability if dropped into a random industry.”
doesn’t take account of the proportion of businesses that fail—if 10% (as I understand it) succeed according to some measure, then a 50-50 split would leave a fairly small probability if dropped into a random industry, as would 80-20.
Very interesting, Ben. Thanks for posting.
Here’s an idea indirectly related to your article: The EA community has an incredible amount of intellectual talent. And it is unusual as far as communities go in that everyone’s motive to make money is selfless. For that reason, I am indifferent to whether I make a million dollars all by myself or whether I make it with the help of 40 other people (aside from differences in the initial investment). Given that, isn’t the EA community uniquely positioned to crowd source a business idea, fund that idea with an EA-friendly VC, hire EA types to run the business, and then give the vast majority of the returns (if any) to EA causes? Would it be a good investment for EA Ventures, for example, to organize an entrepreneurial think tank?
I have some concerns with this paper:
It’s not about startups (average assets of included companies were something like $500m) and is highly weighted towards commoditized industries.
They allow persistent shocks in unexplained variance, which is going to seriously attenuate any firm-specific effect, basically shifting some of the “growth, employees, IP, …” effects into the unexplained variance.
Finally, they don’t mention logging dollar variables, which makes me highly suspicious (perhaps it’s so obvious they don’t need to mention it?)--dollars are power-law distributed, which means large values have an outsized influence on unlogged linear regression fits.
Thanks for the comments Ben! This paper is the canonical response to the poorly researched “All great CEOs do these 10 things” sorts of articles that Harvard Business Review etc. loves to publish, which is why I included it.
Fair enough. If you look at table 4, you can see that the basic trend of “industry matters more than segment specific effects” holds for everything except, confusingly, for manufacturing. So while it may not be robust across different company sizes, it’s at least robust across different industries. (And my intuition is that small companies are even more affected by industry downturns.)
If I’m understanding you correctly, you think they should be using the Rumelt model described on pages 17 and 18? They ran that model on the data in table 4 with basically the same result (17% of variance explained by industry, 29% by company versus 19% and 32% with their model).
Interesting, good point. I skimmed some citing articles and couldn’t find anyone talking about this, so I’m not sure.
This is irrelevant to your point if it doesn’t hold across sizes, though. Compustat’s data is entirely on public companies. I really don’t see any reason to expect this to generalize to tech startups.
In fact, one would expect that most of the influence of founder/IP/team/whatever would be concentrated in navigating from startup to IPO, rather than in post-IPO profits. If the fixed effect is already on the order of 30% of variance post-IPO, I think it’s hard to defend the primarily-lottery-ticket interpretation in the pre-IPO regime.
Overall, I agree with the point that this data set may not be valid, and I sincerely appreciate you pointing that out. I will have to do more research and figure out what can be done with that section.
However, I disagree with the statement:
Startups pivot all the time, and are constantly changing their products. Therefore we should expect that a startup’s profitability in year T doesn’t predict very well their profitability in year T+1.
Big companies almost never change, and my experience is that a large part of the reason they make sales this year is because of the sales, products etc. they made last year. Therefore we should expect their revenue stream to be highly correlated over time.
So I think the big company numbers place an upper bound on the results you would see from small companies, not a lower one.
(One way of coming to this conclusion that I assume is uncontroversial is just to note that the valuations of startups fluctuate a lot more rapidly than the valuations of public companies.)
Let me know what you think about this, and your thoughts around including some language about this in that section.
Yearly profitability isn’t the right measure before IPO; most startups aren’t optimizing for profitability (and indeed never become profitable until after IPO). High losses can indicate failure, or they can indicate rapid scaling of a successful model. So unpredictable profits aren’t really evidence of eventual success being unpredictable. Even if eventual success was relatively predictable we would expect profits to fluctuate along the way depending on the company’s specific growth trajectory.
Sure. For better or for worse that’s what the paper I cited measured, so that’s why I was using that metric.
If you prefer though: startups have significantly higher fluctuations in valuation than public companies, and presumably valuation captures whatever metrics are relevant.
Very interesting and thorough post. Thanks for doing all this research and posting this.
Since so much luck is involved in start-up success, does that mean we might be too focused on creating better EAs when we should be focused more on just creating more EAs? This could apply more broadly than just start-ups, to things like politics, academia, or other forms of influence. Maybe we just need a shotgun approach.
Not really. To avoid principal-agent problems, it’s only necessary that the founders share in the returns, not that some part of the returns is nondeterministic/unknowable-a-priori. For instance, if God appeared to you and Marc Andreessen and told you both (truthfully) that Health eFilings was going to be the next Facebook, I’m pretty sure Andreessen would invest, even though the outcome was now 100% knowable-a-priori.
Thank Ben! I agree. This is why I said “Because salaries are much more stable than business values”—if that antecedent doesn’t hold, then the conclusion doesn’t either.
Your post read: “This is important so I’m going to call it out in very clear terms: Entrepreneurship must involve an element of risk for the founder (i.e. luck). If it didn’t, no one would invest.” You weren’t very clear about the fact that this had “salaries are much more [deterministic] than business values” as a possibly-false antecedent.* (It needs to be a possibly false antecedent because that’s exactly what you’re trying to prove in the rest of the essay. To assert it here would beg the question.)
I’m not sure I understand. It’s just empirically true that business values fluctuate a lot more than salaries, so while it’s a possibly false antecedent it just isn’t false. I’m not sure I understand how you think that should be indicated?
You said “salaries are much more stable than business values” above but for your conclusion to go through you need salaries not only to be more stable, but more ex-ante predictable. If business values fluctuate more than salaries, but the fluctuation is ex-ante predictable, then there is no element of luck in founding a startup (and to the extent the ex-ante variance is small, the role of luck is small).
On the other hand, the ex-ante variance being large is exactly what you’re trying to prove in the rest of the post, so it would be circular to assume it there and then use this to support your subsequent arguments in favor of it.
Huh, interesting point! I will have to reread my copy of The Economics of Entrepreneurship; I wonder if they addressed that or just said “adverse selection” and left it at that.