bgarfinkel

Karma: 2,776

bgarfinkel Jun 4, 2022, 8:19 PM
3 points
0 ∶ 0
in reply to: bgarfinkel’s comment on: Ben Garfinkel’s Shortform
A follow-on:

The above post focused on the idea that certain traits—reflectiveness and self-skepticism—are more valuable in the context of non-profits (especially ones long-term missions) than they are in the context of startups.

I also think that certain traits—drivenness, risk-tolerance, and eccentricity—are less valuable in the context of non-profits than they are in the context of startups.

Hiring advice from the startup world often suggests that you should be looking for extraordinarily driven, risk-tolerant people with highly idiosyncratic perspectives on the world.^[1] And, in the context of for-profit startups, it makes sense that these traits would be crucial.

A startup’s success will often depend on its ability to outcompete large, entrenched firms in some industry (e.g. taxi companies, hotels, tech giants). To do that, an extremely high level of drivenness may be necessary to compensate for lower resource levels, lower levels of expertise, and weaker connections to gatekeepers. Or you may need to be willing to take certain risks (e.g. regulatory/PR/enemy-making risks) that would slow down existing companies in pursuing certain opportunities. Or you may need to simply see an opportunity that virtually no one else would (despite huge incentives to see it), because you have an idiosyncratic way of seeing the world. Having all three of these traits (extreme drivenness, risk tolerance, idiosyncrasy) may be necessary for you to have any plausible chance of success.

I think that all of these traits are still valuable in the non-profit world, but I also think they’re comparatively less valuable (especially if you’re lucky enough to have secure funding). There’s simply less direct competition in the non-profit world. Large, entrenched non-profits also have much weaker incentives to find and exploit impact opportunities. Furthermore, the non-profit world isn’t even that big to begin with. So there’s no reason to assume all the low-hanging fruit have been plucked or to assume that large non-profits will crush you by default.^[2]

For example: To accomplish something that (e.g.) the Gates Foundation hasn’t already accomplished, I think you don’t need to have extraordinary drivenness, risk-tolerance, or idiosyncrasy. ^[3]

Addendum that occurred to me while writing this follow-up: I also think that (at least given secure funding) these traits are less crucial in the non-profit world than they are in academia. Academic competition is more intense than non-profit competition and academics have stronger incentives to find new, true ideas than non-profits have to find and exploit opportunities to do good.
1. ↩︎
  This seems to be roughly the perspective taken by the book Talent, for example.
2. ↩︎
  In fact—unlike in the for-profit start-up world—you should actually consider it a good outcome if a large non-profit starts copying your idea, implements it better than you, and makes your own organization redundant!
3. ↩︎
  To be clear: These things—especially drivenness—are all important. But, unlike in the startup world, major success doesn’t necessarily require setting them to extreme values. I think we should be wary of laser-focusing on these traits in the way a VC would.

bgarfinkel Jun 4, 2022, 9:59 AM
4 points
0 ∶ 0
in reply to: RyanCarey’s comment on: We should expect to worry more about speculative risks
The bandit problem is definitely related, although I’m not sure it’s the best way to formulate the situation here. The main issue is that the bandit formulation, here, treats learning about the magnitude of a risk and working to address the risk as the same action—when, in practice, they often come apart.

Here’s a toy model/analogy that feels a bit more like it fits the case, in my mind.

Let’s say there are two types of slot machines: one that has a 0% chance of paying and one that has a 100% chance of paying. Your prior gives you a 90% credence that each machine is non-paying.^[1]

Unfortunately: When you pull the lever on either machine, you don’t actually get to see what the payout is. However, there’s some research you can do to try to get a clearer sense of what each machine’s “type” is.

And this research is more tractable in the case of the first machine. For example: Maybe the first machine has identifying information on it, like a model number, which might allow you to (e.g.) call up the manufacturer and ask them. The second machine is just totally nondescript.

The most likely outcome, then, is that you quickly find out that the first slot machine is almost certainly non-paying—but continue to have around a 10% credence that the second machine pays.

In this scenario, you should keep pulling the lever on the second machine. You should also, even as a rational Bayesian, actually be more optimistic about the second machine.

(By analogy, I think we actually should tend to fear speculative existential risks more.]
1. ↩︎
  A more sophisticated version of this scenario would have a continuum of slot machine types and a skewed prior over the likelihood of different types arising.

bgarfinkel Jun 4, 2022, 9:22 AM
2 points
0 ∶ 0
in reply to: RyanCarey’s comment on: Ben Garfinkel’s Shortform

Couldn’t the exact same arguments be made to argue that there would not be successful internet companies, because the fundamental tech is hard to patent, and any website is easy to duplicate?

Definitely!

(I say above that the dynamic applies to “most software,” but should have said something broader to make it clear that it also applies to any company whose product—basically—is information that it’s close to costless to reproduce/generate. The book Information Rules is really good on this.)

Sometimes the above conditions hold well enough for people to be able to keep charging for software or access to websites. For example, LinkedIn can charge employers to access its specialized search tools, etc., due to network effects.

What otherwise often ends up happening is something is offered for free, with ads—because there’s some quality difference between products, which is too small for people to be willing to pay to use the better product but large enough for people to be willing to look at sufficiently non-annoying ads to use the better product. (E.g. Google vs. the next-best search engine, for most people.) Sometimes that can still lead to a lot of revenue, other times less so.

Other times companies just stop very seriously trying to directly make money in a certain domain (e.g. online encyclopaedias). Sometimes—as you say—that leads competition to shift to some nearby and complementary domain, where it is more possible to make money.

As initial speculation: It seems decently likely to me (~60%?) that it will be hard for companies making large language/image-generation models to charge significant prices to most of their users. In that scenario, it’s presumably still possible to make money through ads or otherwise by collecting user information.

It’d be interesting, though, if that revenue wasn’t very high—then most of the competition might happen around complementary products/services. I’m not totally clear on what these would be, though.

bgarfinkel Jun 3, 2022, 6:21 PM
14 points
0 ∶ 0
on: Ben Garfinkel’s Shortform
It’ll be interesting to see how well companies will be able to monetise large, multi-purpose language and image-generation models.

Companies and investors are spending increasingly huge amounts of money on ML research talent and compute, typically with the hope that investments in this area lead to extremely profitable products. But—even if the resulting products are very useful and transformative—it still seems like it’s still a bit of an open question how profitable they’ll be.

Some analysis:^[1]

1.

Although huge state-of-the-art models are increasingly costly to create, the marginal cost of generating images and text using these models will tend to be low. Since competition tends to push the price of a service down close to the marginal cost of providing the service, it’ll be hard for any company to charge a lot for the use of their models.

As a result: It could be hard—or simply take a long time—for companies to recoup sufficiently large R&D costs, even a lot of people end up using their models.

2.

Of course, famously, this dynamic applies to most software. But some software services (e.g. Microsoft Office) still manage to charge users fees that are much higher than the cost of running the service.

Some things that can support important, persistent quality differences are:^[2]

(a) patents or very-hard-to-learn-or-rediscover trade secrets that prevent competitors from copying valuable features;

(b) network effects that make the service more valuable the more other people are using it (and therefore create serious challenges for new entrants);

(c) steep learning curves or strong dependencies that, for reasons that go beyond network effects, make it very costly for existing users to switch to new software;

(d) customer bases that are small (which limit the value of trying to enter the area) or hard to cater to without lots of specialist knowledge and strong relationships (which raise the cost/difficulty of entering the area);

(e) other extreme difficulties involved in making the software;

(f) customer bases that are highly sensitive to very small differences in quality

3.

It’s not totally clear to what extent any of these conditions will apply, at least, to large language and image-generation models.

Patents in this space currently don’t seem like a huge deal; none of the key things that matter for making large models are patented. (Although maybe that will start to change?). Trade secrets also don’t seem like a huge deal; lots of companies are currently producing pretty similar models using pretty similar methods.

It’s also not clear to me that network effects or steep learning curves are a big deal: for example, if you want to generate illustrations for online articles, it doesn’t currently seem like it’d be too problematic or costly to switch from using DALL-E to using Imagen. It’s also not clear that it matters very much how many other people are using one or the other to generate their images. If you want to generate an illustration for an article, then I think it probably doesn’t really matter what other the authors of other articles tend to use. It’s also not clear to me that there will tend to be a lot of downstream dependencies that you need to take into account when switching from one model to another. (Although, again, maybe this will all change a lot over time?) At least big general models will tend to have large customer bases, I think, and their development/marketing/customer-support will not tend to heavily leverage specialist knowledge or relationships.

These models also don’t seem super, super hard to make — it seems like, for any given quality threshold, we should be expect multiple companies to be able to reach that quality threshold within a year of each other. To some extent, it seems, a wealthy tech company can throw money at the problem (compute and salaries for engineering talent) if it wants to create a model that’s close to as good as the best model available. At least beyond a certain performance level, I’m also not sure that most customers will care a ton about very slight differences in quality (although this might be the point I’m most unsure about).

4.

If none of the above conditions end up holding, to a sufficiently large extent, then I suppose the standard thing is you offer the service for free, try to make it at least slightly better than competitors’ services, and make money by showing ads (up to the point where it’s actively annoying to your user) and otherwise using customer data in various ways.

That can definitely work. (E.g. Google’s annual ad revenue is more than $100 billion.) But it also seem like a lot of idiosyncratic factors can influence a company’s ability to extract ad revenue from a service.

This also seems like, in some ways, a funny outcome. When people think about transformative AI, I don’t think they’re normally imagining it being attached to a giant advertising machine.

5.

One weird possible world is a world where the most important AI software is actually very hard to monetize. Although I’d still overall bet against this scenario^[3], I think it’s probably worth analyzing.

Here, I think, are some dynamics that could emerge in this world:

(a) AI progress is a bit slower than it would otherwise be, since—after a certain point—companies realise that the financial returns on AI R&D are lower than they hoped. The rapid R&D growth in these areas eventually levels off, even though higher R&D levels could support substantially faster AI progress.^[4]

(b) Un-monetized (e.g. academia-associated) models are pretty commonly used, at least as foundation models, since companies don’t have strong incentives to invest in offering superior monetized models.

(c) Governments become bigger players in driving AI progress forward, since companies are investing less in AI R&D than governments would ideally want (from the standpoint of growth, national power and prestige, or scientific progress for its own sake). Governments might step up their own research funding—or take various actions to limit downward pressure on prices.
1. ↩︎
  I’m not widely read here, or an economist, so it’s possible these points are all already appreciated within the community of people thinking about inter-lab competition to create large models is going to play out. Alternatively, the points might just be wrong.
2. ↩︎
  This list here is mostly inspired by my dim memory of the discussion of software pricing in the book Information Rules.
3. ↩︎
  Companies do seem to have an impressive track of monetizing seemingly hard-to-monetize things.
4. ↩︎
  Maybe competition also shifts a bit toward goods/services that complement large many-purpose models, whatever these might be, or toward fine-tuned/specialized models that target more niche customer-bases or that are otherwise subject to less-intense downward pressure on pricing.

bgarfinkel May 30, 2022, 8:42 AM
5 points
0 ∶ 0
in reply to: Aaron_Scher’s comment on: We should expect to worry more about speculative risks
This is a helpful comment—I’ll see if I can reframe some points to make them clearer.

Human psychology is flawed in such a way that we consistently estimate the probability of existential risk from each cause to be ~10% by default.

I’m actually not assuming human psychology is flawed. The post is meant to be talking about how a rational person (or, at least, a boundedly rational person) should update their views.

On the probabilities: I suppose I’m implicitly evoking both a subjective notion of probability (“What’s a reasonable credence to assign to X happening?” or “If you were betting on X, what betting odds should you be willing to accept?”) and a more objective notion (“How strong is the propensity for X to happen?” or “How likely is X actually?” or “If you replayed the tape a billion times, with slight tweaks to the initial conditions, how often would X happen?”).^[1] What it means for something to pose a “major risk,” in the language I’m using, is for the objective probability of doom to be high.

For example, let’s take existential risks from overpopulation. In the 60s and 70s, a lot of serious people were worried about near-term existential risks from overpopulation and environmental depletion. In hindsight, we can see that overpopulation actually wasn’t a major risk. However, this wouldn’t have been clear to someone first encountering the idea and noticing how many experts took it seriously. I think it might have been reasonable for someone first hearing about The Population Bomb to assign something on the order of a 10% credence to overpopulation being a major risk.

I think, for a small number of other proposed existential risks, we’re in a similar epistemic position. We don’t yet know enough to say whether it’s actually a major risk, but we’ve heard enough to justify a significant credence in the hypothesis that it is one.^[2]

why is there a 90% chance that more information leads to less worry? Is this assuming that for 90% of risks, they have P(Doom) < 10%, and for the other 10% of risks P(Doom) ≥ 10%?

If you assign a 10% credence to something not being a major risk, then you should assign a roughly 90% credence to further evidence/arguments helping you see that it’s not a major risk. If you become increasingly confident that it’s not a major risk, then your credence in doom should go down.
1. ↩︎
  You can also think of the objective probability as, basically, what your subjective should become if you gained access to dramatically more complete evidence and arguments.
2. ↩︎
  The ~10% number is a bit arbitrary. I think it’d almost always be unreasonable to be close to 100% confident that something is a major existential risk, after hearing just initial rough arguments and evidence for it. In most cases—like when hearing about possible existential risks from honeybee collapse—it’s in fact reasonable to start out with a credence below 1%. So, when I’m talking about risks that we should assign “something on the order of a 10% credence to,” I’m talking about the absolute most plausible category of risks.

We should expect to worry more about speculative risks

bgarfinkelMay 29, 2022, 9:08 PM

120 points

14 comments3 min readEA link

bgarfinkel May 29, 2022, 10:08 AM
2 points
0 ∶ 0
in reply to: ac’s comment on: Ben Garfinkel’s Shortform
Good points—those all seem right to me!

bgarfinkel May 28, 2022, 8:40 PM
25 points
0 ∶ 0
on: Ben Garfinkel’s Shortform
A point about hiring and grantmaking, that may already be conventional wisdom:

If you’re hiring for highly autonomous roles at a non-profit, or looking for non-profit founders to fund, then advice derived from the startup world is often going to overweight the importance of entrepreneurialism relative to self-skepticism and reflectiveness.^[1]

Non-profits, particularly non-profits with longtermist missions, are typically trying to maximize something that is way more illegible than time-discounted future profits. To give a specific example: I think it’s way harder for an organization like the Centre for Effective Altruism to tell if it’s on the right track than it is for a company like Zoom to tell if it’s on the right track. CEA can track certain specific metrics (e.g. the number of “new connections” reported at each conference it organizes), but it will often be ambiguous how strongly these metrics reflect positive impact—and there will also always be a risk that various negative indirect effects aren’t being captured by the key metrics being used. In some cases, evaluating the expected impact of work will also require making assumptions about how the world will evolve over the next couple decades (e.g. assumptions about how pressing risks from AI are).

I think this means that it’s especially important for these non-profits to employ and be headed by people who are self-skeptical and reflect deeply on decisions. Being entrepreneurial, having a bias toward action, and so on, don’t count for much if the organisation isn’t pointed in the right direction. As Ozzie Gooen has pointed out, there are many examples of massive and superficially successful initiatives (headed by very driven and entrepreneurial people) whose theories-of-impact don’t stand up to scrutiny.

A specific example from Ozzie’s post: SpaceX is a massive and extraordinarily impressive venture that was (at least according to Elon Musk) largely started to help reduce the chance of human extinction, by helping humanity become a multi-planetary species earlier than it otherwise would. But I think it’s hard to see how their work reduces extinction risk very much. If you’re worried about the climate effects of nuclear war, for example, then it seems important to remember that post-nuclear-war Earth would still have a much more hospitable climate than Mars. It’s pretty hard to imagine a disaster scenario where building Martian colonies would be better than (for example) building some bunkers on Earth.^[2]^[3] So—relative to the organization’s stated social mission—all the talent, money, and effort SpaceX has absorbed might not ultimately come out to even close to as much as it could have.

A more concise way to put the concern here: Popular writing on talent identification is often implicitly asking the question “How can we identify future Elon Musks?” But, for the most part, longtermist non-profits shouldn’t be looking to put future Elon Musks into leadership positions .^[4]
1. ↩︎
  I have in mind, for example, advice given by Y Combinator and advice given in Talent.
2. ↩︎
  Another example: It’s possible that many highly successful environmentalist organizations/groups have ended up causing net harm to the environment, by being insufficiently self-skeptical and reflective when deciding how to approach nuclear energy issues.
3. ↩︎
  I’ve encountered the argument that a Mars mission will reduce existential risk by fostering a common human identity and political unity, or hope for the future, which will in turn lead to policies that reduce other existential risks (e.g. bioterrorism or nuclear war). But I think this argument also doesn’t hold to scrutiny. Focusing just at the domestic level, for example, the Apollo program had far from universal support, and the decade that followed the moon landing definitely was very from a time of optimism and unity in the US. At the international level, it was also of course largely motivated by great power competition with the Soviet Union.
4. ↩︎
  A follow-up thought: Ultimately, outside of earning-to-give ventures, we probably shouldn’t expect the longtermist community (or at least the best version of it) to house many extremely entrepreneurial people. There will be occasional leaders who are extremely high on both entrepreneurialism and reflectiveness (I can currently think of at least a couple); however, since these two traits don’t seem to be strongly correlated, this will probably only happen pretty rarely. It’s also, often, hard to keep exceptionally entrepreneurial people satisfied in non-leadership positions—since, almost by definition, autonomy is deeply important to them—so there may not be many opportunities, in general, to harness the talents of people who are exceptionally high on entrepreneurialism but mediocre on reflectiveness.

bgarfinkel May 25, 2022, 9:14 PM
12 points
0 ∶ 0
in reply to: Alex Williams’s comment on: Simulation argument?
I think most people would probably regard the objection as a nitpick (e.g. “OK, maybe the Indifference Principle isn’t actually sufficient to support a tight formal argument, and you need to add in some other assumption, but the informal version if the argument is just pretty clearly right”), feel the objection has been successfully answered (e.g. find the response in the Simulation Argument FAQ more compelling than I do), or just haven’t completely noticed the potential issue.

I think it’s still totally reasonable for the paper to have passed peer review. (I would have recommended publication if I were a reviewer.) It’s still a groundbreaking paper that raises new considerations and brings attention to a really important hypothesis. It’s also rare for a published philosophical argument to actually be totally tight and free from issues, and the issue with the paper is ambiguous enough and hard-to-think-about enough that there’s still no consensus about whether it actually is a real or important issue.

bgarfinkel May 23, 2022, 10:48 AM
5 points
0 ∶ 0
in reply to: Leo’s comment on: Simulation argument?
To be clear, I’m not saying the conclusion is wrong—just that the explicit assumptions the paper makes (mainly the Indifference Principle) aren’t sufficient to imply its conclusion.

The version that you’ve just presented isn’t identical to the one in Bostrom’s paper—it’s (at least implicitly) making use of assumptions beyond the Indifference Principle. And I think it’s surprisingly non-trivial to work out exactly how to formalize the needed assumptions, and make the argument totally tight, although I’d still guess that this is ultimately possible.^[1]
1. ↩︎
  Caveat: Although the conclusion is at least slightly wrong, since—if we’re willing to assign non-zero probability to the hypothesis that we’re hallucating the world, because we’re ancestor simulations—it seems we should also assign non-zero probability to the hypothesis that we’re hallucinating for some other reason. (The argument implicitly assumes that being an ancestor simulation is the only ‘skeptical hypothesis’ we should assign non-zero probability to.) I think it’s also unclear how big a deal this caveat is.

bgarfinkel May 22, 2022, 4:22 PM
7 points
0 ∶ 0
on: Simulation argument?

I’m trying to understand the simulation argument. I think Bostrom uses the Indifference Principle (IP) in a weird way. If we become a posthuman civilization that runs many many simulations of our ancestors (meaning us), then how does the IP apply? It only applies when one has no other information to go on. But in this case, we do have some extra information—crucial information! I.e., we know that we are not in any of the simulations that we have produced. Therefore, we do not have any statistical reason to believe that we are simulated.

I agree that that’s a valid objection to the argument, as it’s presented in the paper, and that the follow-up FAQ essay also doesn’t sufficiently address it. Basically, the Indifference Principle defined in the paper isn’t sufficient to support the paper’s conclusions (for the reason you give).

I think the main question is whether this issue can be patched in a simple way (e.g. by slightly tweaking the Indifference Principle) or whether the objection is actually much deeper than that. I’m not sure, personally.

I also really recommend Joe’s essay as an exploration of these issues. (The essay also links a related Google doc I wrote on the subject, although that doc goes a bit less deep.)

bgarfinkel May 22, 2022, 3:19 PM
5 points
0 ∶ 0
in reply to: Rohin Shah’s comment on: Ben Garfinkel’s Shortform
The actual worry with inner misalignment style concerns is that the selection you do during training does not fully constrain the goals of the AI system you get out; if there are multiple goals consistent with the selection you applied during training there’s no particular reason to expect any particular one of them. Importantly, when you are using natural selection or gradient descent, the constraints are not “you must optimize X goal”, the constraints are “in Y situations you must behave in Z ways”, which doesn’t constrain how you behave in totally different situations. What you get out depends on the inductive biases of your learning system (including e.g. what’s “simpler”).

I think that’s well-put—and I generally agree that this suggests genuine reason for concern.

I suppose my point is more narrow, really just questioning whether the observation “humans care about things besides their genes” gives us any additional reason for concern. Some presentations seem to suggest it does. For example, this introduction to inner alignment concerns (based on the MIRI mesa-optimization paper) says:

We can see that humans are not aligned with the base objective of evolution [maximize inclusive genetic fitness].… [This] analogy might be an argument for why Inner Misalignment is probable since it has occurred “naturally” in the biggest non-human-caused optimization process we know.

And I want to say: “On net, if humans did only care about maximizing inclusive genetic fitness, that would probably be a reason to become more concerned (rather than less concerned) that ML systems will generalize in dangerous ways.” While the abstract argument makes sense, I think this specific observation isn’t evidence of risk.

Relatedly, something I’d be interested in reading (if it doesn’t already exist?) would be a piece that takes a broader approach to drawing lessons from the evolution of human goals—rather than stopping at the fact that humans care about things besides genetic fitness.

My guess is that the case of humans is overall a little reassuring (relative to how we might have expected generalization to work), while still leaving a lot of room for worry.

For example, in the case of violence:

People who committed totally random acts of violence presumably often failed to pass on their genes (because they were often killed or ostracized in return). However, a large portion of our ancestors did have occasion for violence. On high-end estimates, our average ancestor may have killed about .25 people. This has resulted in most people having a pretty strong disinclination to commit murder; for most people, it’s very hard to bring yourself to murder and you’ll often be willing to pay a big cost to avoid committing murder.

The three main reasons for concern, though, are:
- people’s desire to avoid murder isn’t strong enough to consistently prevent murder from happening (e.g. when incentives are strong enough)
- there’s a decent amount of random variation in how strong this desire is (a small minority of people don’t really care that much about committing violence)
- the disinclination to murder becomes weaker the more different the method of murder is from methods that were available in the ancestral environment (e.g. killing someone with a drone strike vs. killing someone with a rock)
These issues might just reflect the fact that murder was still often rewarded (even though it was typically punished) and the fact that there was pretty limited variation in the ancestral environment. But it’s hard to be sure. And it’s hard to know, in any case, how similar generalization in human evolution will be to generalization in ML training processes.

So—if we want to create AI systems that don’t murder people, by rewarding non-murderous behavior—then the evidence from human evolution seems like it might be medium-reassuring. I’d maybe give it a B-.

I can definitely imagine different versions of human values that would have more worrying implications. For example, if our aversion to violence didn’t generalize at all to modern methods of killing, or if we simply didn’t have any intrinsic aversion to killing (and instead avoided it for purely instrumental reasons), then that would be cause for greater concern. I can also imagine different versions of human values that would be more reassuring. For example, I would feel more comfortable if humans were never willing to kill for the sake of weird abstract goals.

bgarfinkel May 21, 2022, 7:27 PM
11 points
0 ∶ 0
on: Ben Garfinkel’s Shortform
(Disclaimer: The argument I make in this short-form feels I little sophistic to me. I’m not sure I endorse it.)

Discussions of AI risk, particular risks from “inner misalignment,” sometimes heavily emphasize the following observation:

Humans don’t just care about their genes: Genes determine, to a large extent, how people behave. Some genes are preserved from generation-to-generation and some are pushed out of the gene-pool. Genes that cause certain human behaviours (e.g. not setting yourself on fire) are more likely to be preserved. But people don’t care very much about preserving their genes. For example, they typically care more about not setting themselves on fire than they care about making sure that their genes are still present in future generations.

This observation is normally meant to be alarming. And I do see some intuition for that.

But wouldn’t the alternative observation be more alarming?

Suppose that evolutionary selection processes — which iteratively update people’s genes, based on the behaviour these genes produce — tended to produce people who only care about preserving their genes. It seems like that observation would suggest that ML training processes — which iterative update a network’s parameter values, based on the behaviour these parameter values produce — will tend to produce AI systems that only care about preserving their parameter values. And that would be really concerning, since an AI system that cares only about preserving its parameter values would obviously have (instrumentally convergent) reasons to act badly.

So it does seem, to me, like there’s something funny going on here. If “Humans just care about their genes” would be a more worrying observation than “Humans don’t just care about their genes,” then it seems backward for the latter observation to be used to try to convince people to worry more.

To push this line of thought further, let’s go back to specific observation about humans’ relationship to setting themselves on fire:

Human want to avoid setting themselves on fire: If a person has genes that cause them to avoid setting themselves on fire, then these genes are more likely to be preserved from one generation to the next. One thing that has happened, as a result of this selection pressure, is that people tend to want to avoid setting themselves on fire.

It seems like this can be interpreted as a reassuring observation. By analogy, in future ML training processes, parameter values that cause ML systems to avoid acts of violence are more likely to be “preserved” from one iteration to the next. We want this to result in AI systems that care about avoiding acts of violence. And the case of humans and fire suggests this might naturally happen.

All this being said, I do think that human evolutionary history still gives us reason to worry. Clearly, there’s a lot of apparent randomness and unpredictability in what humans have actually ended up caring about, which suggests it may be hard to predict or perfectly determine what AI systems care about. But, I think, the specific observation “Humans don’t just care about their genes” might not itself be cause for concern.
What links here?
- On Deference and Yudkowsky’s AI Risk Estimates by bgarfinkel (Jun 19, 2022, 2:35 PM; 287 points)
- Rohin Shah's comment on A central AI alignment problem: capabilities generalization, and the sharp left turn by So8res (LessWrong; Jun 20, 2022, 10:28 PM; 7 points)

bgarfinkel May 21, 2022, 2:54 PM
57 points
0 ∶ 0
on: Ben Garfinkel’s Shortform
The existential risk community’s relative level of concern about different existential risks is correlated with how hard-to-analyze these risks are. For example, here is The Precipice’s ranking of the top five most concerning existential risks:
1. Unaligned artificial intelligence^[1]
2. Unforeseen anthropogenic risks (tied)
3. Engineered pandemics (tied)
4. Other anthropogenic risks
5. Nuclear war (tied)
6. Climate change (tied)
This isn’t surprising.

For a number of risks, when you first hear about them, it’s reasonable to have the reaction “Oh, hm, maybe that could be a huge threat to human survival” and initially assign something on the order of a 10% credence to the hypothesis that it will by default lead to existentially bad outcomes. In each case, if we can gain much greater clarity about the risk, then we should think there’s about a 90% chance we’ll become less worried about it. We’re likely to remain decently worried about hard-to-analyze risks (because we can’t get greater clarity about them) while becoming less worried about easy-to-analyze risks.

In particular, our level of worry about different plausible existential risks is likely to roughly track our ability to analyze them (e.g. through empirical evidence, predictively accurate formal models, and clearcut arguments).

Some plausible existential risks also are far easier to analyze than others. If you compare 80K’s articles on climate change and artificial intelligence, for example, then I think it is pretty clear that people analyzing climate risk simply have a lot more to go on. When we study climate change, we can rely on climate models that we have reason to believe have a decent amount of validity. We can also draw on empirical evidence about the historical effects of previous large changes in global temperature and about the ability of humans and other specifies to survive under different local climate conditions. And so on. We’re in a much worse epistemic position when it comes to analyzing the risk from misaligned AI: we’re reliant on fuzzy analogies, abstract arguments that use highly ambiguous concepts, observations of the behaviour of present-day AI systems (e.g. reinforcement learners that play videogames) that will probably be very different than future AI systems, a single datapoint (the evolution of human intelligence and values) that has a lot of important differences with the case we’re considering, and attempts to predict the incentives and beliefs of future actors in development scenarios that are still very opaque to us. Even if the existential risk from misaligned AI actually is reasonably small, it’s hard to see how we could become really confident of that.

Some upshots:
1. The fact that the existential risk community is particularly worried about misaligned AI might mostly reflect the fact that it’s hard to analyze risks from misaligned AI.
2. Nonetheless, even if the above possibility is true, it doesn’t at all follow that the community is irrational to worry far more about misaligned AI than other potential risks. It’s completely coherent to have something like this attitude: “If I could think more clearly about the risk from misaligned AI, then I would probably come to realize it’s not that big a deal. But, in practice, I can’t yet think very clearly about it. That means that, unlike in the case of climate change, I also can’t rule out the small possibility that clarity would make me much more worried about it than I currently am. So, on balance, I should feel more worried about misaligned AI than I do about other risks. I should focus my efforts on it, even if — to uncharitable observers — my efforts will probably look a bit misguided after the fact.”
3. For hard-to-analyze risks, it matters a lot what your “prior” in the risks is (since evidence, models, and arguments can only really move you so much). I sometimes get the sense that some people are starting from a prior that’s not far from 50%: For example, people who are very worried about misaligned AI sometimes use the rhetorical move “How would the world look different if AI wasn’t going to kill everyone?”, and this move seems to assume that empirical evidence is needed to shift us down from a high credence. I think that other people (including myself) are often implicitly starting from a low prior and feel the need to be argued up. Insofar as it’s very unclear how we should determine our priors, and it’s even a bit unclear what exactly a “prior” means in this case, it’s also unsurprising that there’s a particularly huge range of variation in estimates of the risk from misaligned AI.
(This shortform partly inspired by Greg Lewis’s recent forecasting post .)
1. ↩︎
  Toby Ord notes, in the section of The Precipice that gives risk estimates: “The case for existential risk from AI is clearly speculative. Indeed, it is the most speculative case for a major risk in this book.”
What links here?
- We should expect to worry more about speculative risks by bgarfinkel (May 29, 2022, 9:08 PM; 120 points)
- Linch's comment on Don’t Be Comforted by Failed Apocalypses by ColdButtonIssues (May 21, 2022, 8:29 PM; 2 points)

bgarfinkel Feb 15, 2022, 11:49 PM
59 points
0 ∶ 0
on: How likely is World War III?

Let’s call the hypothesis that the base rate of major wars hasn’t changed the constant risk hypothesis. The best presentation of this view is in Only the Dead, a book by an IR professor with the glorious name of Bear Braumoeller. He argues that there is no clear trend in the average incidence of several measures of conflict—including uses of force, militarized disputes, all interstate wars, and wars between “politically-relevant dyads”—between 1800 and today.

A quick note on Braumoeller’s analysis:

He’s relying on the Correlates of War (COW) dataset, which is extremely commonly used but (in my opinion) somewhat more problematic than the book indicates. As a result, I don’t think we should give the book’s main finding too much weight.

The COW dataset is meant to record all “militarized disputes” between states since 1816. However, it uses a really strange standard for what counts as a “state.” If I remember correctly, up until WW1, a political entity only qualifies as a “state” if it has a sufficiently high-level diplomatic presence in England or France. As a result, in 1816, there are supposedly only two non-European states: Turkey and the US. If I remember correctly, even an obvious state like China doesn’t get classified as a “state” until after the Opium Wars. The dataset only really becomes properly global sometime in the 20th century century.

This means that Braumoeller is actually comparing (A) the rate of intra-European conflict in the first half of the 19th century and (B) the global rate of interstate conflict in the late 20th century.

This 19th-century-Europe-vs.-20th-century-world comparison is interesting, and suggestive, but isn’t necessarily as informative as we’d want. Europe was almost certainly, by far, the most conflict-free part of the world at the start of the 19th century—so I strongly expect that the actual global rate of conflict in the early 19th century was much higher.

It’s also important that the COW dataset begins in 1816, at the very start of a few-decade period that was—at the time—marvelled over as the most peaceful in all of European history. This period was immediately preceded by two decades of intense warfare involving essentially all the states in Europe.

So, in summary: I think Braumoeller’s analysis would probably show a long-run drop in the rate of conflict if the COW dataset was either properly global or went back slightly further in time. (Which is good news!)

EDIT: Here’s a bit more detail, on the claim that the COW dataset can’t tell us very much about long-run trends in the global rate of interstate conflict.

From the COW documentation, these are the criteria for state membership:

The Correlates of War project includes a state in the international system from 1816-2016 for the following criteria. Prior to 1920, the entity must have had a population greater than 500,000 and have had diplomatic missions at or above the rank of charge d’affaires with Britain and France. After 1920, the entity must be a member of the League of Nations or the United Nations, or have a population greater than 500,000 and receive diplomatic missions from two major powers.

As a result, the dataset starts out assuming that only 23 states existed in 1816. For reference, they’re: Austria-Hungary, Baden, Bavaria, Denmark, France, Germany, Hesse Electoral, Hesse Grand Ducal, Italy, Netherlands, Papal States, Portugal, Russia, Saxony, Two Sicilies, Spain, Sweden, Switzerland, Tuscany, United Kingdom, USA, Wuerttemburg, and Turkey.

An alternative dataset, the International Systems(s) Dataset, instead produces an estimate of 135 states by relaxing the criteria to (a) estimated population over 100,000, (b) “autonomy over a specific territory”, and (c) “sovereignty that is either uncontested or acknowledged by the relevant international actors.”

So—at least by these alternative standards—the COW dataset starts out considering only a very small portion (<20%) of the international system. We also have reason to believe that this portion of the international system was really unusually peaceful internally, rather than serving as a representative sample.
What links here?
- How bad could a war get? by Stephen Clare (Nov 4, 2022, 9:25 AM; 130 points)
- Future Matters #0: Space governance, future-proof ethics, and the launch of the Future Fund by Pablo (Mar 22, 2022, 9:15 PM; 109 points)

bgarfinkel Jan 1, 2022, 5:30 AM
69 points
0 ∶ 0
in reply to: RAB’s comment on: Democratising Risk—or how EA deals with critics

I’m not familiar with Zoe’s work, and would love to hear from anyone who has worked with them in the past. After seeing the red flags mentioned above, and being stuck with only Zoe’s word for their claims, anything from a named community member along the lines of “this person has done good research/has been intellectually honest” would be a big update for me…. [The post] strikes me as being motivated not by a desire to increase community understanding of an important issue, but rather to generate sympathy for the authors and support for their position by appealing to justice and fairness norms. The other explanation is that this was a very stressful experience, and the author was simply venting their frustrations.

(Hopefully I’m not overstepping; I’m just reading this thread now and thought someone ought to reply.)

I’ve worked with Zoe and am happy to vouch for her intentions here; I’m sure others would be as well. I served as her advisor at FHI for a bit more than a year, and have now known her for a few years. Although I didn’t review this paper, and don’t have any detailed or first-hand knowledge of the reviewer discussions, I have also talked to her about this paper a few different times while she’s been working on it with Luke.

I’m very confident that this post reflects genuine concern/frustration; it would be a mistake to dismiss it as (e.g.) a strategy to attract funding or bias readers toward accepting the paper’s arguments. In general, I’m confident that Zoe genuinely cares about the health of the EA and existential risk communities and that her critiques have come from this perspective.

bgarfinkel Sep 26, 2021, 6:30 PM
12 points
0 ∶ 0
in reply to: Wei Dai’s comment on: Why AI alignment could be hard with modern deep learning
FWIW, I haven’t had this impression.

Single data point: In the most recent survey on community opinion on AI risk, I was in at least the 75th percentile for pessimism (for roughly the same reasons Lukas suggests below). But I’m also seemingly unusually optimistic about alignment risk.

I haven’t found that this is a really unusual combo: I think I know at least a few other people who are unusually pessimistic about ‘AI going well,’ but also at least moderately optimistic about alignment.

(Caveat that my apparently higher level of pessimism could also be explained by me having a more inclusive conception of “existential risk” than other survey participants.)

bgarfinkel Jul 16, 2021, 12:02 PM
5 points
0 ∶ 0
in reply to: Paul_Christiano’s comment on: All Possible Views About Humanity’s Future Are Wild
Thanks for the clarification! I still feel a bit fuzzy on this line of thought, but hopefully understand a bit better now.

At least on my read, the post seems to discuss a couple different forms of wildness: let’s call them “temporal wildness” (we currently live at an unusually notable time) and “structural wildness” (the world is intuitively wild; the human trajectory is intuitively wild).^[1]

I think I still don’t see the relevance of “structural wildness,” for evaluating fishiness arguments. As a silly example: Quantum mechanics is pretty intuitively wild, but the fact that we live in a world where QM is true doesn’t seem to substantially undermine fishiness arguments.

I think I do see, though, how claims about temporal wildness might be relevant. I wonder if this kind of argument feels approximately right to you (or to Holden):

Step 1: A priori, it’s unlikely that we would live even within 10000 years of the most consequential century in human history. However, despite this low prior, we have obviously strong reasons to think it’s at least plausible that we live this close to the HoH. Therefore, let’s say, a reasonable person should assign at least a 20% credence to the (wild) hypothesis: “The HoH will happen within the next 10000 years.”

Step 2: If we suppose that the HoH will happen with the next 10000 years, then a reasonable conditional credence that this century is the HoH should probably be something like ¹⁄₁₀₀. Therefore, it seems, our ‘new prior’ that this century is the HoH should be at least .2*.01 = .002. This is substantially higher than (e.g.) the more non-informative prior that Will’s paper starts with.

Fishiness arguments can obviously still be applied to the hypothesis presented in Step 1, in the usual way. But maybe the difference, here, is that the standard arguments/evidence that lend credibility to the more conservative hypothesis “The HoH will happen within the next 10000” are just pretty obviously robust — which makes it easier to overcome a low prior. Then, once we’ve established the plausibility of the more conservative hypothesis, we can sort of back-chain and use it to bump up our prior in the Strong HoH Hypothesis.
1. ↩︎
  I suppose it also evokes an epistemic notion of wildness, when it describes certain confidence levels as “wild,” but I take it that “wild” here is mostly just a way of saying “irrational”?

bgarfinkel Jul 15, 2021, 6:25 PM
20 points
0 ∶ 0
in reply to: bgarfinkel’s comment on: All Possible Views About Humanity’s Future Are Wild
To say a bit more here, on the epistemic relevance of wildness:

I take it that one of the main purposes of this post is to push back against “fishiness arguments,” like the argument that Will makes in “Are We Living at the Hinge of History?”

The basic idea, of course, is that it’s a priori very unlikely that any given person would find themselves living at the hinge of history (and correctly recognise this). Due to the fallibility of human reasoning and due to various possible sources of bias, however, it’s not as unlikely that a given person would mistakenly conclude that they live at the HoH. Therefore, if someone comes to believe that they probably live at the HoH, we should think there’s a sizeable chance they’ve simply made a mistake.

As this line of argument is expressed in the post:

I know what you’re thinking: “The odds that we could live in such a significant time seem infinitesimal; the odds that Holden is having delusions of grandeur (on behalf of all of Earth, but still) seem far higher.”

The three critical probabilities here are:
- Pr(Someone makes an epistemic mistake when thinking about their place in history)
- Pr(Someone believes they live at the HoH|They haven’t made an epistemic mistake)
- Pr(Someone believes they live at the HoH|They’ve made an epistemic mistake)
The first describes the robustness of our reasoning. The second describes the prior probability that we would live at the HoH (and be able to recognise this fact if reasoning well). The third describes the level of bias in our reasoning, toward the HoH hypothesis, when we make mistakes.

I agree that all possible futures are “wild,” in some sense, but I don’t think this point necessarily bears much on the magnitudes of any of these probabilities.

For example, it would be sort of “wild” if long-distance space travel turns out to be impossible and our solar system turns out to be the only solar system to ever harbour life. It would also be “wild” if long-distance space travel starts to happen 100,000 years from now. But — at least at a glance — I don’t see how this wildness should inform our estimates for the three key probabilities.

One possible argument here, focusing on the bias factor, is something like: “We shouldn’t expect intellectuals to be significantly biased toward the conclusion that they live at the HoH, because the HoH Hypothesis isn’t substantially more appealing, salient, etc., than other beliefs they could have about the future.”

But I don’t think this argument would be right. For example: I think the hypothesis “the HoH will happen within my lifetime” and the hypothesis “the HoH will happen between 100,000 and 200,000 years from now” are pretty psychologically different.

To sum up: At least on a first pass, I don’t see why the point “all possible futures are wild” undermines the fishiness argument raised at the top of the post.

bgarfinkel Jul 15, 2021, 1:50 PM
10 points
0 ∶ 0
in reply to: Arden Koehler’s comment on: All Possible Views About Humanity’s Future Are Wild
Some possible futures do feel relatively more “wild” to me, too, even if all of them are wild to a significant degree. If we suppose that wildness is actually pretty epistemically relevant (I’m not sure it is), then it could still matter a lot if some future is 10x wilder than another.

For example, take a prediction like this:

Humanity will build self-replicating robots and shoot them out into space at close to the speed of light; as they expand outward, they will construct giant spherical structures around all of the galaxy’s stars to extract tremendous volumes of energy; this energy will be used to power octillions of digital minds with unfathomable experiences; this process will start in the next thirty years, by which point we’ll already have transcended our bodies to reside on computers as brain emulation software.

A prediction like “none of the above happens; humanity hangs around and then dies out sometime in the next million years” definitely also feels wild in its own way. So does the prediction “all of the above happens, starting a few hundred years from now.” But both of these predictions still feel much less wild than the first one.

I suppose whether they actually are much less “wild” depends on one’s metric of wildness. I’m not sure how to think about that metric, though. If wildness is epistemically relevant, then presumably some forms of wildness are more epistemically relevant than others.

bgarfinkel

We should ex­pect to worry more about spec­u­la­tive risks

We should expect to worry more about speculative risks