An Equilibrium of No Free Energy

EliezerYudkowsky31 Oct 2017 22:25 UTC
18 points
7 comments27 min readEA link

Previous: Inadequacy and Modesty

I am now going to introduce some concepts that lack established names in the economics literature—though I don’t believe that any of the basic ideas are new to economics.

First, I want to distinguish between the standard economic concept of efficiency (as in efficient pricing) and the related but distinct concepts of inexploitability and adequacy, which are what usually matter in real life.

i.

Depending on the strength of your filter bubble, you may have met people who become angry when they hear the phrase “efficient markets,” taking the expression to mean that hedge fund managers are particularly wise, or that markets are particularly just.1

Part of where this interpretation appears to be coming from is a misconception that market prices reflect a judgment on anyone’s part about what price would be “best”—fairest, say, or kindest.

In a pre-market economy, when you offer somebody fifty carrots for a roasted antelope leg, your offer says something about how impressed you are with their work hunting down the antelope and how much reward you think that deserves from you. If they’ve dealt generously with you in the past, perhaps you ought to offer them more. This is the only instinctive notion people start with for what a price could mean: a personal interaction between Alice and Bob reflecting past friendships and a balance of social judgments.

In contrast, the economic notion of a market price is that for every loaf of bread bought, there is a loaf of bread sold; and therefore actual demand and actual supply are always equal. The market price is the input that makes the decreasing curve for demand as a function of price meet the increasing curve for supply as a function of price. This price is an “is” statement rather than an “ought” statement, an observation and not a wish.

In particular, an efficient market, from an economist’s perspective, is just one whose average price movement can’t be predicted by you.

If that way of putting it sounds odd, consider an analogy. Suppose you asked a well-designed superintelligent AI system to estimate how many hydrogen atoms are in the Sun. You don’t expect the superintelligence to produce an answer that is exactly right down to the last atom, because this would require measuring the mass of the Sun more finely than any measuring instrument you expect it to possess. At the same time, it would be very odd for you to say, “Well, I think the superintelligence will underestimate the number of atoms in the Sun by 10%, because hydrogen atoms are very light and the AI system might not take that into account.” Yes, hydrogen atoms are light, but the AI system knows that too. Any reason you can devise for how a superintelligence could underestimate the amount of hydrogen in the Sun is a possibility that the superintelligence can also see and take into account. So while you don’t expect the system to get the answer exactly right, you don’t expect that you yourself will be able to predict the average value of the error—to predict that the system will underestimate the amount by 10%, for example.

This is the property that an economist thinks an “efficient” price has. An efficient price can update sharply: the company can do worse or better than expected, and the stock can move sharply up or down on the news. In some cases, you can rationally expect volatility; you can predict that good news might arrive tomorrow and make the stock go up, balanced by a counter-possibility that the news will fail to arrive and the stock will go down. You could think the stock is 30% likely to rise by $10 and 20% likely to drop by $15 and 50% likely to stay the same. But you can’t predict in advance the average value by which the price will change, which is what it would take to make an expected profit by buying the stock or short-selling it.2

When an economist says that a market price is efficient over a two-year time horizon, they mean: “The current price that balances the supply and demand of this financial instrument well reflects all public information affecting a boundedly rational estimate of the future supply-demand balancing point of this financial instrument in two years.” They’re relating the present intersection of these two curves to an idealized cognitive estimate of the curves’ future intersection.

But this is a long sentence in the language of a hunter-gatherer. If somebody doesn’t have all the terms of that sentence precompiled in their head, then they’re likely to interpret the sentence in the idiom of ordinary human life and ordinary human relationships.

People have an innate understanding of “true” in the sense of a map that reflects the territory, and they can imagine processes that produce good maps; but probability and microeconomics are less intuitive.3 What people hear when you talk about “efficient prices” is that a cold-blooded machine has determined that some people ought to be paid $9/​hour. And they hear the economist saying nice things about the machine, praising it as “efficient,” implying that the machine is right about this $9/​hour price being good for society, that this price well reflects what someone’s efforts are justly worth. They hear you agreeing with this pitiless machine’s judgment about how the intuitive web of obligations and incentives and reputation ought properly to cash out for a human interaction.

And in the domain of stocks, when stock prices are observed to swing widely, this intuitive view says that the market can’t be that smart after all. For if it were smart, would it keep turning out to be “wrong” and need to change its mind?

I once read a rather clueless magazine article that made fun of a political prediction market on the basis that when a new poll came out, the price of the prediction market moved. “It just tracks the polls!” the author proclaimed. But the point of the prediction market is not that it knows some fixed, objective chance with high accuracy. The point of a prediction market is that it summarizes all the information available to the market participants. If the poll moved prices, then the poll was new information that the market thought was important, and the market updated its belief, and this is just the way things should be.

In a liquid market, “price moves whose average direction you can predict in advance” correspond to both “places you can make a profit” and “places where you know better than the market.” A market that knows everything you know is a market where prices are “efficient” in the conventional economic sense—one where you can’t predict the net direction in which the price will change.

This means that the efficiency of a market is assessed relative to your own intelligence, which is fine. Indeed, it’s possible that the concept should be called “relative efficiency.” Yes, a superintelligence might be able to predict price trends that no modern human hedge fund manager could; but economists don’t think that today’s markets are efficient relative to a superintelligence.

Today’s markets may not be efficient relative to the smartest hedge fund managers, or efficient relative to corporate insiders with secret knowledge that hasn’t yet leaked. But the stock markets are efficient relative to you, and to me, and to your Uncle Albert who thinks he tripled his money through his incredible acumen in buying NetBet.com.

ii.

Not everything that involves a financial price is efficient. There was recently a startup called Color Labs, aka Color.com, whose putative purpose was to let people share photos with their friends and see other photos that had been taken nearby. They closed $41 million in funding, including $20 million from the prestigious Sequoia Capital.

When the news of their funding broke, practically everyone on the online Hacker News forum was rolling their eyes and predicting failure. It seemed like a nitwit me-too idea to me too. And then, yes, Color Labs failed and the 20-person team sold themselves to Apple for $7 million and the venture capitalists didn’t make back their money. And yes, it sounds to me like the prestigious Sequoia Capital bought into the wrong startup.

If that’s all true, it’s not a coincidence that neither I nor any of the other onlookers could make money on our advance prediction. The startup equity market was inefficient (a price underwent a predictable decline), but it wasn’t exploitable.4 There was no way to make a profit just by predicting that Sequoia had overpaid for the stock it bought. Because, at least as of 2017, the market lacks a certain type and direction of liquidity: you can’t short-sell startup equity.5

What about houses? Millions of residential houses change hands every year, and they cost more than stock shares. If we expect the stock market to be well-priced, shouldn’t we expect the same of houses?

The answer is “no,” because you can’t short-sell a house. Sure, there are some ways to bet against aggregate housing markets, like shorting real estate investment trusts or home manufacturers. But in the end, hedge fund managers can’t make a synthetic financial instrument that behaves just like the house on 6702 West St. and sell it into the same housing market frequented by consumers like you. Which is why you might do very well to think for yourself about whether the price seems sensible to you before buying a house: because you might know better than the market price, even as a non-specialist relying only on publicly available information.

Let’s imagine there are 100,000 houses in Boomville, of which 10,000 have been for sale in the last year or so. Suppose there are 20,000 fools who think that housing prices in Boomville can only go up, and 10,000 rational hedge fund managers who think that the shale-oil business may collapse and lead to a predictable decline in Boomville house prices. There’s no way for the hedge fund managers to short Boomville house prices—not in a way that satisfies the optimistic demand of 20,000 fools for Boomville houses, not in a way that causes house prices to actually decline. The 20,000 fools just bid on the 10,000 available houses until the skyrocketing price of the houses makes 10,000 of the fools give up.

Some smarter agents might decline to buy, and so somewhat reduce demand. But the smarter agents can’t actually visit Boomville and make hundreds of thousands of dollars off of the overpriced houses. The price is too high and will predictably decline, relative to public information, but there’s no way you can make a profit on knowing that. An individual who owns an existing house can exploit the inefficiency by selling that house, but rational market actors can’t crowd around the inefficiency and exploit it until it’s all gone.

Whereas a predictably underpriced house, put on the market for predictably much less than its future price, would be an asset that any of a hundred thousand rational investors could come in and snap up.

So a frothy housing market may see many overpriced houses, but few underpriced ones.

Thus it will be easy to lose money in this market by buying stupidly, and much harder to make money by buying cleverly. The market prices will be inefficient—in a certain sense stupid—but they will not be exploitable.

In contrast, in a thickly traded market where it is easy to short an overpriced asset, prices will be efficient in both directions, and any day is as good a day to buy as any other. You may end up exposed to excess volatility (an asset with a 50% chance of doubling and a 50% chance of going bankrupt, for example), but you won’t actually have bought anything overpriced—if it were predictably overpriced, it would have been short-sold.6

We can see the notion of an inexploitable market as generalizing the notion of an efficient market as follows: in both cases, there’s no free energy inside the system. In both markets, there’s a horde of hungry organisms moving around trying to eat up all the free energy. In the efficient market, every predictable price change corresponds to free energy (easy money) and so the equilibrium where hungry organisms have eaten all the free energy corresponds to an equilibrium of no predictable price changes. In a merely inexploitable market, there are predictable price changes that don’t correspond to free energy, like an overpriced house that will decline later, and so the no-free-energy equilibrium can still involve predictable price changes.7

Our ability to say, within the context of the general theory of “efficient markets,” that houses in Boomville may still be overpriced—and, additionally, to say that they are much less likely to be underpriced—is what makes this style of reasoning powerful. It doesn’t just say, “Prices are usually right when lots of money is flowing.” It gives us detailed conditions for when we should and shouldn’t expect efficiency. There’s an underlying logic about powerfully smart organisms, any single one of which can consume free energy if it is available in worthwhile quantities, in a way that produces a global equilibrium of no free energy; and if one of the premises is invalidated, we get a different prediction.

iii.

At one point during the 2016 presidential election, the PredictIt prediction market—the only one legally open to US citizens (and only US citizens)—had Hillary Clinton at a 60% probability of winning the general election. The bigger, international prediction market BetFair had Clinton at 80% at that time.

So I looked into buying Clinton shares on PredictIt—but discovered, alas, that PredictIt charged a 10% fee on profits, a 5% fee on withdrawals, had an $850 limit per contract bet… and on top of all that, I’d also have to pay 28% federal and 9.3% state income taxes on any gains. Which, in sum, meant I wouldn’t be getting much more than $30 in expected return for the time and hassle of buying the contracts.

Oh, if only PredictIt didn’t charge that 10% fee on profits, that 5% fee on withdrawals! If only they didn’t have the $850 limit! If only the US didn’t have such high income taxes, and didn’t limit participation in overseas prediction markets! I could have bought Clinton shares at 60 cents on PredictIt and Trump shares at 20 cents on Betfair, winning a dollar either way and getting a near-guaranteed 25% return until the prices were in line! Curse those silly rules, preventing me from picking up that free money!

Does that complaint sound reasonable to you?

If so, then you haven’t yet fully internalized the notion of an inefficient-but-inexploitable market.

If the taxes, fees, and betting limits hadn’t been there, the PredictIt and BetFair prices would have been the same.

iv.

Suppose it were the case that some cases of Seasonal Affective Disorder proved resistant to sitting in front of a 10,000-lux lightbox for 30 minutes (the standard treatment), but would nonetheless respond if you bought 130 or so 60-watt-equivalent high-CRI LED bulbs, in a mix of 5000K and 2700K color temperatures, and strung them up over your two-bedroom apartment.

Would you expect that, supposing this were true, there would already exist a journal report somewhere on it?

Would you expect that, supposing this were true, it would already be widely discussed (or at least rumored) on the Internet?

Would you expect that, supposing this were true, doctors would already know about it and it would be on standard medical pages about Seasonal Affective Disorder?

And would you, failing to observe anything on the subject after a couple of hours of Googling, conclude that your civilization must have some unknown good reason why not everyone was doing this already?

To answer a question like this, we need an analysis not of the world’s efficiency or inexploitability but rather of its adequacy—whether all the low-hanging fruit have been plucked.

A duly modest skepticism, translated into the terms we’ve been using so far, might say something like this: “Around 7% of the population has severe Seasonal Affective Disorder, and another 20% or so has weak Seasonal Affective Disorder. Around 50% of tested cases respond to standard lightboxes. So if the intervention of stringing up a hundred LED bulbs actually worked, it could provide a major improvement to the lives of 3% of the US population, costing on the order of $1000 each (without economies of scale). Many of those 9 million US citizens would be rich enough to afford that as a treatment for major winter depression. If you could prove that your system worked, you could create a company to sell SAD-grade lighting systems and have a large market. So by postulating that you can cure SAD this way, you’re postulating a world in which there’s a huge quantity of metaphorical free energy—a big energy gradient that society hasn’t traversed. Therefore, I’m skeptical of this medical theory for more or less the same reason that I’m skeptical you can make money on the stock market: it postulates a $20 bill lying around that nobody has already picked up.”

So the distinction is:

  • Efficiency: “Microsoft’s stock price is neither too low nor too high, relative to anything you can possibly know about Microsoft’s stock price.”

  • Inexploitability: “Some houses and housing markets are overpriced, but you can’t make a profit by short-selling them, and you’re unlikely to find any substantially underpriced houses—the market as a whole isn’t rational, but it contains participants who have money and understand housing markets as well as you do.”

  • Adequacy: “Okay, the medical sector is a wildly crazy place where different interventions have orders-of-magnitude differences in cost-effectiveness, but at least there’s no well-known but unused way to save ten thousand lives for just ten dollars each, right? Somebody would have picked up on it! Right?!”

Let’s say that within some slice through society, the obvious low-hanging fruit that save more than ten thousand lives for less than a hundred thousand dollars total have, in fact, been picked up. Then I propose the following terminology: let us say that that part of society is adequate at saving 10,000 lives for $100,000.

And if there’s a convincing case that this property does not hold, we’ll say this subsector is inadequate (at saving 10,000 lives for $100,000).

To see how an inadequate equilibrium might arise, let’s start by focusing on one tiny subfactor of the human system, namely academic research.

We’ll even further oversimplify our model of academia and pretend that research is a two-factor system containing academics and grantmakers, and that a project can only happen if there’s both a participating academic and a participating grantmaker.

We next suppose that in some academic field, there exists a population of researchers who are individually eager and collectively opportunistic for publications—papers accepted to journals, especially high-impact journal publications that constitute strong progress toward tenure. For any clearly visible opportunity to get a sufficiently large number of citations with a small enough amount of work, there are collectively enough academics in this field that somebody will snap up the opportunity. We could say, to make the example more precise, that the field is collectively opportunistic in 2 citations per workday—if there’s any clearly visible opportunity to do 40 days of work and get 80 citations, somebody in the field will go for it.

This level of opportunism might be much more than the average paper gets in citations per day of work. Maybe the average is more like 10 citations per year of work, and lots of researchers work for a year on a paper that ends up garnering only 3 citations. We’re not trying to ask about the average price of a citation; we’re trying to ask how cheap a citation has to be before somebody somewhere is virtually guaranteed to try for it.

But academic paper-writers are only half the equation; the other half is a population of grantmakers.

In this model, can we suppose for argument’s sake that grantmakers are motivated by the pure love of all sentient life, and yet we still end up with an academic system that is inadequate?

I might naively reply: “Sure. Let’s say that those selfish academics are collectively opportunistic at two citations per workday, and the blameless and benevolent grantmakers are collectively opportunistic at one quality-adjusted life-year (QALY) per $100.8 Then everything which produces one QALY per $100 and two citations per workday gets funded. Which means there could be an obvious, clearly visible project that would produce a thousand QALYs per dollar, and so long as it doesn’t produce enough citations, nobody will work on it. That’s what the model says, right?”

Ah, but this model has a fragile equilibrium of inadequacy. It only takes one researcher who is opportunistic in QALYs and willing to take a hit in citations to snatch up the biggest, lowest-hanging altruistic fruit if there’s a population of grantmakers eager to fund projects like that.

Assume the most altruistically neglected project produces 1,000 QALYs per dollar. If we add a single rational and altruistic researcher to this model, then they will work on that project, whereupon the equilibrium will be adequate at 1,000 QALYs per dollar. If there are two rational and altruistic researchers, the second one to arrive will start work on the next-most-neglected project—say, a project that has 500 QALYs/​$ but wouldn’t garner enough citations for whatever reason—and then the field will be adequate at 500 QALYs/​$. As this free energy gets eaten up (it’s tasty energy from the perspective of an altruist eager for QALYs), the whole field becomes less inadequate in the relevant respect.

But this assumes the grantmakers are eager to fund highly efficient QALY-increasing projects.

Suppose instead that the grantmakers are not cause-neutral scope-sensitive effective altruists assessing QALYs/​$. Suppose that most grantmakers pursue, say, prestige per dollar. (Robin Hanson offers an elementary argument that most grantmaking to academia is about prestige.9 In any case, we can provisionally assume the prestige model for purposes of this toy example.)

From the perspective of most grantmakers, the ideal grant is one that gets their individual name, or their boss’s name, or their organization’s name, in newspapers around the world in close vicinity to phrases like “Stephen Hawking” or “Harvard professor.” Let’s say for the purpose of this thought experiment that the population of grantmakers is collectively opportunistic in 20 microHawkings per dollar, such that at least one of them will definitely jump on any clearly visible opportunity to affiliate themselves with Stephen Hawking for $50,000. Then at equilibrium, everything that provides at least 2 citations per workday and 20 microHawkings per dollar will get done.

This doesn’t quite follow logically, because the stock market is far more efficient at matching bids between buyers and sellers than academia is at matching researchers to grantmakers. (It’s not like anyone in our civilization has put as much effort into rationalizing the academic matching process as, say, OkCupid has put into their software for hooking up dates. It’s not like anyone who did produce this public good would get paid more than they could have made as a Google programmer.)

But even if the argument is still missing some pieces, you can see the general shape of this style of analysis. If a piece of research will clearly visibly yield lots of citations with a reasonable amount of labor, and make the grantmakers on the committee look good for not too much money committed, then a researcher eager to do it can probably find a grantmaker eager to fund it.

But what if there’s some intervention which could save 100 QALYs/​$, yet produces neither great citations nor great prestige? Then if we add a few altruistic researchers to the model, they probably won’t be able to find a grantmaker to fund it; and if we add a few altruistic grantmakers to the model, they probably won’t be able to find a qualified researcher to work on it.

One systemic problem can often be overcome by one altruist in the right place. Two systemic problems are another matter entirely.

Usually when we find trillion-dollar bills lying on the ground in real life, it’s a symptom of (1) a central-command bottleneck that nobody else is allowed to fix, as with the European Central Bank wrecking Europe, or (2) a system with enough moving parts that at least two parts are simultaneously broken, meaning that single actors cannot defy the system. To modify an old aphorism: usually, when things suck, it’s because they suck in a way that’s a Nash equilibrium.

In the same way that inefficient markets tend systematically to be inexploitable, grossly inadequate systems tend systematically to be unfixable by individual non-billionaires.

But then you can sometimes still insert a wedge for yourself, even if you can’t save the whole system. Something that’s systemically hard to fix for the whole planet is sometimes possible to fix in your own two-bedroom apartment. So inadequacy is even more important than exploitability on a day-to-day basis, because it’s inadequacy-generating situations that lead to low-hanging fruits large enough to be worthwhile at the individual level.

v.

A critical analogy between an inadequate system and an efficient market is this: even systems that are horribly inadequate from our own perspective are still in a competitive equilibrium. There’s still an equilibrium of incentives, an equilibrium of supply and demand, an equilibrium where (in the central example above) all the researchers are vigorously competing for prestigious publications and using up all available grant money in the course of doing so. There’s no free energy anywhere in the system.

I’ve seen a number of novice rationalists committing what I shall term the Free Energy Fallacy, which is something along the lines of, “This system’s purpose is supposed to be to cook omelettes, and yet it produces terrible omelettes. So why don’t I use my amazing skills to cook some better omelettes and take over?”

And generally the answer is that maybe the system from your perspective is broken, but everyone within the system is intensely competing along other dimensions and you can’t keep up with that competition. They’re all chasing whatever things people in that system actually pursue—instead of the lost purposes they wistfully remember, but don’t have a chance to pursue because it would be career suicide. You won’t become competitive along those dimensions just by cooking better omelettes.

No researcher has any spare attention to give your improved omelette-cooking idea because they are already using all of their labor to try to get publications into high-impact journals; they have no free work hours.

The journals won’t take your omelette-cooking paper because they get lots of attempted submissions that they screen, for example, by looking for whether the researcher is from a high-prestige institution or whether the paper is written in a style that makes it look technically difficult. Being good at cooking omelettes doesn’t make you the best competitor at writing papers to appeal to prestigious journals—any publication slot would have to be given to you rather than someone else who is intensely trying to get it. Your good omelette technique might be a bonus, but only if you were already doing everything else right (which you’re not).

The grantmakers have no free money to give you to run your omelette-cooking experiment, because there are thousands of researchers competing for their money, and you are not competitive at convincing grantmaking committees that you’re a safe, reputable, prestigious option. Maybe they feel wistfully fond of the ideal of better omelettes, but it would be career suicide for them to give money to the wrong person because of that.

What inadequate systems and efficient markets have in common is the lack of any free energy in the equilibrium. We can see the equilibrium in both cases as defined by an absence of free energy. In an efficient market, any predictable price change corresponds to free energy, so thousands of hungry organisms trying to eat the free energy produce a lack of predictable price changes. In a system like academia, the competition for free energy may not correspond to anything good from your own standpoint, and as a result you may label the outcome “inadequate”; but there is still no free energy. Trying to feed within the system, or do anything within the system that uses a resource the other competing organisms want—money, publication space, prestige, attention—will generally be as hard for you as it is for any other organism.

Indeed, if the system gave priority to rewarding better performance along the most useful or socially beneficial dimensions over all competing ways of feeding, the system wouldn’t be inadequate in the first place. It’s like wishing PredictIt didn’t have fees and betting limits so that you could snap up those mispriced contracts.

In a way, it’s this very lack of free energy, this intense competition without space to draw a breath, that keeps the inadequacy around and makes it non-fragile. In the case of US science, there was a brief period after World War II where there was new funding coming in faster than universities could create new grad students, and scientists had a chance to pursue ideas that they liked. Today Malthus has reasserted himself, and it’s no longer generally feasible for people to achieve career success while going off and just pursuing the research they most enjoy, or just going off and pursuing the research with the largest altruistic benefits. For any actor to do the best thing from an altruistic standpoint, they’d need to ignore all of the system’s internal incentives pointing somewhere else, and there’s no free energy in the system to feed someone who does that.10

vi.

Since the idea of civilizational adequacy seems fairly useful and general, I initially wondered whether it might be a known idea (under some other name) in economics textbooks. But my friend Robin Hanson, a professional economist at an academic institution well-known for its economists, has written a lot of material that I see (from this theoretical perspective) as doing backwards reasoning from inadequacy to incentives.11 If there were a widespread economic notion of adequacy that he were invoking, or standard models of academic incentives and academic inadequacy, I would expect him to cite them.

Now look at the above paragraph. Can you spot the two implicit arguments from adequacy?

The first sentence says, “To the extent that this way of generalizing the notion of an efficient market is conceptually useful, we should expect the field of economics to have been adequate to have already explored it in papers, and adequate at the task of disseminating the resulting knowledge to the point where my economist friends would be familiar with it.”

The second and third sentences say, “If something like inadequacy analysis were already a well-known idea in economics, then I would expect my smart economist friend Robin Hanson to cite it. Even if Robin started out not knowing, I expect his other economist friends would tell him, or that one of the many economists reading his blog would comment on it. I expect the population of economists reading Robin’s blog and papers to be adequate to the task of telling Robin about an existing field here, if one already existed.”

Adequacy arguments are ubiquitous, and they’re much more common in everyday reasoning than arguments about efficiency or exploitability.

vii.

Returning to that business of stringing up 130 light bulbs around the house to treat my wife’s Seasonal Affective Disorder:

Before I started, I tried to Google whether anyone had given “put up a ton of high-quality lights” a shot as a treatment for resistant SAD, and didn’t find anything. Whereupon I shrugged, and started putting up LED bulbs.

Observing these choices of mine, we can infer that my inadequacy analysis was something like this: First, I did spend a fair amount of time Googling, and tried harder after the first search terms failed. This implies I started out thinking my civilization might have been adequate to think of the more light treatment and test it.

Then when I didn’t find anything on Google, I went ahead and tested the idea myself, at considerable expense. I didn’t assign such a high probability to “if this is a good idea, people will have tested it and propagated it to the point where I could find it” that in the absence of Google results, I could infer that the idea was bad.

I initially tried ordering the cheapest LED lights from Hong Kong that I could find on eBay. I didn’t feel like I could rely on the US lighting market to equalize prices with Hong Kong, and so I wasn’t confident that the premium price for US LED bulbs represented a quality difference. But when the cheap lights finally arrived from Hong Kong, they were dim, inefficient, and of visibly low color quality. So I decided to buy the more expensive US light bulbs for my next design iteration.

That is: I tried to save money based on a possible local inefficiency, but it turned out not to be inefficient, or at least not inefficient enough to be easily exploited by me. So I updated on that observation, discarded my previous belief, and changed my behavior.

Sometime after putting up the first 100 light bulbs or so, I was working on an earlier draft of this chapter and therefore reflecting more intensively on my process than I usually do. It occurred to me that sometimes the best academic content isn’t online and that it might not be expensive to test that. So I ordered a used $6 edited volume on Seasonal Affective Disorder, in case my Google-fu had failed me, hoping that a standard collection of papers would mention a light-intensity response curve that went past “standard lightbox.”

Well, I’ve flipped through that volume, and so far it doesn’t seem to contain any account of anyone having ever tried to cure resistant SAD using more light, either substantially higher-intensity or substantially higher-duration. I didn’t find any table of response curves to light levels above 10,000 lux, or any experiments with all-day artificial light levels comparable to my apartment’s roughly 2,000-lux luminance.

I say this to emphasize that I didn’t lock myself into my attempted reasoning about adequacy when I realized it would cost $6 to perform a further observational check. And to be clear, ordering one book still isn’t a strong check. It wouldn’t surprise me in the least to learn that at least one researcher somewhere on Earth had tested the obvious thought of more light and published the response curve. But I’d also hesitate to bet at odds very far from 1:1 in either direction.

And the higher-intensity light therapy does seems to have mostly cured Brienne’s SAD. It wasn’t cheap, but it was cheaper than sending her to Chile for 4 months.

If more light really is a simple and effective treatment for a large percentage of otherwise resistant patients, is it truly plausible that no academic researcher out there has ever conducted the first investigation to cross my own mind? “Well, since the Sun itself clearly does work, let’s try more light throughout the whole house—never mind these dinky lightboxes or 30-minute exposure times—and then just keep adding more light until it frickin’ works.” Is that really so non-obvious? With so many people around the world suffering from severe or subclinical SAD that resists lightboxes, with whole countries in the far North or South where the syndrome is common, could that experiment really have never been tried in a formal research setting?

On my model of the world? Sure.

Am I running out and trying to get a SAD researcher interested in my anecdotal data? No, because when something like this doesn’t get done, there’s usually a deeper reason than “nobody thought of it.”

Even if nobody did think of it, that says something about a lack of incentives to be creative. If academics expected working solutions to SAD to be rewarded, there would already be a much larger body of literature on weird things researchers had tried, not just lightbox variant after lightbox variant. Inadequate systems tend systematically to be systemically unfixable; I don’t know the exact details in this case, but there’s probably something somewhere.

So I don’t expect to get rich or famous, because I don’t expect the system to be that exploitable in dollars or esteem, even though it is exploitable in personalized SAD treatments. Empirically, lots of people want money and acclaim, and base their short- and long-term career decisions around its pursuit; so achieving it in unusually large quantities shouldn’t be as simple as having one bright idea. But there aren’t large groups of competent people visibly organizing their day-to-day lives around producing outside-the-box new lightbox alternatives with the same intensity we can observe people organizing their lives around paying the bills, winning prestige or the acclaim of peers, etc.

People presumably care about curing SAD—if they could effortlessly push a button to instantly cure SAD, they would do so—but there’s a big difference between “caring” and “caring enough to prioritize this over nearly everything else I care about,” and it’s the latter that would be needed for researchers to be willing to personally trade away non-small amounts of expected money or esteem for new treatment ideas.12

In the case of Japan’s monetary policy, it wasn’t a coincidence that I couldn’t get rich by understanding macroeconomics better than the Bank of Japan. Japanese asset markets shot up as soon as it became known that the Bank of Japan would create more money, without any need to wait and see—so it turns out that the markets also understood macroeconomics better than the Bank of Japan. Part of our civilization was being, in a certain sense, stupid: there were trillion-dollar bills lying around for the taking. But they weren’t trillion-dollar bills that just anyone could walk over and pick up.

From the standpoint of a single agent like myself, that ecology didn’t contain the particular kind of free energy that lots of other agents were competing to eat. I could be unusually right about macroeconomics compared to the PhD-bearing professionals at the Bank of Japan, but that weirdly low-hanging epistemic fruit wasn’t a low-hanging financial fruit; I couldn’t use the excess knowledge to easily get excess money deliverable the next day.

Where reward doesn’t follow success, or where not everyone can individually pick up the reward, institutions and countries and whole civilizations can fail at what is usually imagined to be their tasks. And then it is very much easier to do better in some dimensions than to profit in others.

To state all of this more precisely: Suppose there is some space of strategies that you’re competent enough to think up and execute on. Inexploitability has a single unit attached, like “$” or “effective SAD treatments,” and says that you can’t find a strategy in this space that knowably gets you much more of the resource in question than other agents. The kind of inexploitability I’m interested in typically arises when a large ecosystem of competing agents is genuinely trying to get the resource in question, and has access to strategies at least as good (for acquiring that resource) as the best options in your strategy space.

Inadequacy with respect to a strategy space has two units attached, like “effective SAD treatments /​ research hours” or “QALYs /​ $,” and says that there is some set of strategies a large ecosystem of agents could pursue that would convert the denominator unit into the numerator unit at some desired rate, but the agents are pursuing strategies that in fact result in a lower conversion rate. The kind of inadequacy I’m most interested in arises when many of the agents in the ecosystem would prefer that the conversion occur at the rate in question, but there’s some systemic blockage preventing this from happening.

Systems tend to be inexploitable with respect to the resources that large ecosystems of competent agents are trying their hardest to pursue, like fame and money, regardless of how adequate or inadequate they are. And if there are other resources the agents aren’t adequate at converting fame, money, etc. into at a widely desired rate, it will often be due to some systemic blockage. Insofar as agents have overlapping goals, it will therefore often be harder than it looks to find real instances of exploitability, and harder than it looks to outperform an inadequate equilibrium. But more local goals tend to overlap less: there isn’t a large community of specialists specifically trying to improve my wife’s well-being.

The academic and medical system probably isn’t that easy to exploit in dollars or esteem, but so far it does look like maybe the system is exploitable in SAD innovations, due to being inadequate to the task of converting dollars, esteem, researcher hours, etc. into new SAD cures at a reasonable rate—inadequate, for example, at investigating some SAD cures that Randall Munroe would have considered obvious,13 or at doing the basic investigative experiments that I would have considered obvious. And when the world is like that, it’s possible to cure someone’s crippling SAD by thinking carefully about the problem yourself, even if your civilization doesn’t have a mainstream answer.

viii.

There’s a whole lot more to be said about how to think about inadequate systems: common conceptual tools include Nash equilibria, commons problems, asymmetrical information, principal-agent problems, and more. There’s also a whole lot more to be said about how not to think about inadequate systems.

In particular, if you relax your self-skepticism even slightly, it’s trivial to come up with an a priori inadequacy argument for just about anything. Talk about “efficient markets” in any less than stellar forum, and you’ll soon get half a dozen comments from people deriding the stupidity of hedge fund managers. And, yes, the financial system is broken in a lot of ways, but you still can’t double your money trading S&P 500 stocks. “Find one thing to deride, conclude inadequacy” is not a good rule.

At the same time, lots of real-world social systems do have inadequate equilibria and it is important to be able to understand that, especially when we have clear observational evidence that this is the case. A blanket distrust of inadequacy arguments won’t get us very far either.

This is one of those ideas where other cognitive skills are required to use it correctly, and you can shoot off your own foot by thinking wrongly. So if you’ve read this far, it’s probably a good idea to keep reading.

Cross-posted to equilibriabook.com and Less Wrong. Next: Moloch’s Toolbox.

  1. If the person gets angry and starts talking about lack of liquidity, rather than about the pitfalls of capitalism, then that is an entirely separate class of dispute. ↩

  2. You can often predict the likely direction of a move in such a market, even though on average your best guess for the change in price will always be 0. This is because the median market move will usually not equal the mean market move. For similar reasons, a rational agent can usually predict the direction of a future Bayesian update, even though the average value by which their probability changes should be 0. A high probability of a small update in the expected direction can be offset by a low probability of a larger update in the opposite direction. ↩

  3. Anyone who tries to spread probability literacy quickly runs into the problem that a weather forecast giving an 80% chance of clear skies is deemed “wrong” on the 1-in–5 occasions when it in fact rains, prompting people to wonder what mistake the weather forecaster made this time around. ↩

  4. More precisely, I would say that the market was inexploitable in money, but inefficiently priced. ↩

  5. To short-sell is to borrow the asset, sell it, and then buy it back later after the price declines; or sometimes to create a synthetic copy of an asset, so you can sell that. Shorting an asset allows you to make money if the price goes down in the future, and has the effect of lowering the asset’s price by increasing supply. ↩

  6. Though beware that even in a stock market, some stocks are harder to short than others—like stocks that have just IPOed. Drechsler and Drechsler found that creating a broad market fund of only assets that are easy to short in recent years would have produced 5% higher returns (!) than index funds that don’t kick out hard-to-short assets. Unfortunately, I don’t know of any index fund that actually tracks this strategy, or it’s what I’d own as my main financial asset. ↩

  7. Robert Shiller cites Edward Miller as having observed in 1977 that efficiency requires short sales, and either Shiller or Miller observes that houses can’t be shorted. But I don’t know of any standard economic term for markets that are inefficient but “inexploitable” (as I termed it). It’s not a new idea, but I don’t know if it has an old name.

    I mention parenthetically that a regulator that genuinely and deeply cared about protecting retail financial customers would just concentrate on making everything in that market easy to short-sell. This is the obvious and only way to ensure the asset is not overpriced. If the Very Serious People behind the JOBS Act to enable crowdfunded startups had honestly wanted to protect normal people and understood this phenomenon, they would mandate that all equity sales go through an exchange where it was easy to bet against the equity of dumb startups, and then declare their work done and go on permanent vacation in Aruba. This is the easy and only way to protect consumers from overpriced financial assets. ↩

  8. “Quality-adjusted life year” is a measure used to compare the effectiveness of medical interventions. QALYs are a popular way of relating the costs of death and disease, though they’re generally defined in ways that exclude non-health contributors to quality of life. ↩

  9. Hanson, “Academia’s Function.” ↩

  10. This is also why, for example, you can’t get your project funded by appealing to Bill Gates. Every minute of Bill Gates’s time that Bill Gates makes available to philanthropists is a highly prized and fought-over resource. Every dollar of Gates’s that he makes available to philanthropy is already highly fought over. You won’t even get a chance to talk to him. Bill Gates is surrounded by a cloud of money, but you’re very naive if you think that corresponds to him being surrounded by a cloud of free energy. ↩

  11. Robin often says things like, for example: “X doesn’t use a prediction market, so X must not really care about accurate estimates.” That is to say: “If system X were driven mainly by incentive Y, then it would have a Y-adequate equilibrium that would pick low-hanging fruit Z. But system X doesn’t do Z, so X must not be driven mainly by incentive Y.” ↩

  12. Even the attention and awareness needed to explicitly consider the option of making such a tradeoff, in an environment where such tradeoffs aren’t already normally made or discussed, is a limited resource. Researchers will not be motivated to take the time to think about pursuing more socially beneficial research strategies if they’re currently pouring all their attention and strategic thinking into finding ways to achieve more of the other things they want in life.

    Conventional cynical economics doesn’t require us to posit Machiavellian researchers who explicitly considered pursuing better strategies for treating SAD and decided against them for selfish reasons; they can just be too busy and distracted pursuing more obvious and immediate rewards, and never have a perceptible near-term incentive to even think very much about some other considerations. ↩

  13. See: What If? Laser Pointer. ↩

EliezerYudkowsky31 Oct 2017 22:25 UTC
18 points
7 comments27 min readEA link