Davidmanheim

Karma: 6,292

Davidmanheim 24 Sep 2023 15:13 UTC
2 points
0 ∶ 0
in reply to: alexherwix’s comment on: Pausing AI vs Degrowth in rich countries
I’m explaining why people haven’t engaged with this—the specifics are missing, or have little to do with degrowth, or are wrong. You can cite the study “justifying” limits to growth, (which I’ve discussed on this forum before!) but they said that there would be a collapse decades ago, so it’s hard to take that seriously.
I’m sure there is a steelmanned version of this that deserves some credit, and I initially said that there are some ideas from that movement that deserve credit—but I don’t understand what it has to do with the degrowth movement, which is pretty explicit about what it wants and aims for.

Davidmanheim 24 Sep 2023 10:36 UTC
2 points
1 ∶ 1
in reply to: alexherwix’s comment on: Pausing AI vs Degrowth in rich countries
The last sentence in that quote gives away the game. The hypothesis—the one I’m saying is not supported by any evidence, and which has been falsified in the past—is that you can do degrowth without the downsides. The concrete proposals are to stop doing the things that increase economic growth. For example, they are opposed to mining more minerals, regardless of environmental damage, because they want less resource usage. Less isn’t more.
You say their point is worthy of discussion. Which point? That there are finite limits? No, it’s not worth discussing. Yes, there are limits to growth, but they aren’t relevant. They are busy telling people energy is finite, so we should use less—ignoring the fact that energy can be plentiful with solar and other renewable sources.
These are the same people—literally the same, in some cases—as the “limits to growth” folks from decades ago, and the fact that they were wrong hasn’t deterred them in the least. They are STILL telling people that we will run out of minerals, ignoring the fact that discoverable reserves are orders of magnitude larger than we need in the foreseeable future, and in most cases reserves have been getting larger over time.
But sure, you can tell me I haven’t engaged with this, and that it needs more thought. I’m even happy to give it more thought—I just need you, or someone else, to point to what you think we should consider that isn’t either philosophy about finitude ungrounded in any facts, or that is flat out wrong, instead of saying “consider this general area,” one which I’m broadly familiar with already.

Davidmanheim 24 Sep 2023 6:52 UTC
2 points
1 ∶ 1
in reply to: alexherwix’s comment on: Pausing AI vs Degrowth in rich countries
Current gross world income is enough to provide everyone on earth with $10,000 per year. (And that overstates a lot of things because much of that wealth is in forms that aren’t distributable.)
But $10,000/year is under the US federal poverty level. It seems a lot like degrowth is embracing a world where everyone lives under the level that a poor person in the developed world lives, which seems in pretty stark contrast to the goals of making the world prosper.
The claim of degrowth is that there’s no way to have more than this consistent with long-term flourishing of humanity. That seems to fly in the face of every theoretical and observed claim about resource constraints and growth—climate is a real problem, but degrowth doesn’t come anywhere close to solving it.
There are other claims that degrowth makes that seem unobjectionable, and worthy of debate—global poverty is being perpetuated by debt burden in the developing world, so the developed world needs to forgive that debt, foreign aid doesn’t accomplish its stated aims is is a force for neo-colonialism, austerity overwhelmingly harms the poor and should be unacceptable, the enforcement of global intellectual property laws harms the developing world, and similar. But the central claim, that we need to have fewer goods, fewer people, and less prosperity, isn’t really worth debate.

Davidmanheim 24 Sep 2023 4:57 UTC
3 points
0 ∶ 0
on: “We can Prevent AI Disaster Like We Prevented Nuclear Catastrophe”
The key missing component here is getting buy-in from the major actors—countries and companies—not the specific solution getting proposed.

Davidmanheim 23 Sep 2023 16:58 UTC
2 points
0 ∶ 0
on: It’s not obvious that getting dangerous AI later is better
Great post, including lots of useful things to think through! I do have a couple points I’d like to note, however.
One key assumption I think is questionable is that the choice presented is dangerous AI now or later, so that the two are mutually exclusive. In fact, if we get dangerous AI soon, that doesn’t imply additional and even more dangerous systems would be created later. The potential reason for believing this is that there will be governance mechanisms put in place after a failure—which does seem likely, but it also argues strongly for trying to figure out how to build those mechanisms as soon as possible, since it’s entirely plausible that “later” isn’t that far away.
Another assumption is that we can’t stop dangerous AI completely, that is, the possibility of preventing its development long enough to build aligned AI that would be more capable of ensuring safety. To make that assumption, you need to be pessimistic that governance can control these systems that long—which is a plausible concern, but not a certainty, and again raises the earlier point that we could have dangerous AI both soon and later.

Davidmanheim 23 Sep 2023 16:47 UTC
2 points
0 ∶ 0
in reply to: Matthew_Barnett’s comment on: How could a moratorium fail?
I apologize that my intent here was unclear—I edited it to say “the treaties [being discussed] aren’t permanent,” which I thought was clear from context.

How could a moratorium fail?

Davidmanheim22 Sep 2023 15:11 UTC

28 points

3 comments9 min readEA link

Davidmanheim 22 Sep 2023 7:56 UTC
7 points
1 ∶ 0
in reply to: Vaipan’s comment on: The Case for AI Safety Advocacy to the Public
I don’t assume that they are convinced, I think that they are aware of the issues. They are also a tiny group compared to the general population—so I think you need a far stronger reason to focus on such a small group instead of the public than what has been suggested.
And I think you’re misconstruing my position about EA versus AI safety—I strongly agree that they should be separate, as I’ve said elsewhere.

Davidmanheim 22 Sep 2023 7:30 UTC
2 points
0 ∶ 0
in reply to: cmeinel’s comment on: What’s in a Pause?
These are difficult problems, but thankfully not the ones we need to deal with immediately. None of Iran, Russia and North Korea are chip producers nor are they particularly close to SOTA in ML—if there is on-chip monitoring for manufacturers, and cloud compute has restrictions, there is little chance they accelerate. And we stopped the first two from getting nukes for decades, so export controls are definitely a useful mechanism. In addition, the incentive for nuclear states or otherwise dangerous rogue actors to develop AGI as a strategic asset is lessened if they aren’t needed for balance of power—so a global moratorium makes these states less likely to feel a need to keep up in order to stay in power.
That said, a moratorium isn’t a permanent solution to proliferation of dangerous tech, even if the regime were to end up being permanent. Like with nuclear weapons, we expect to raise costs of violating norms to be prohibitively high, and we can delay things for quite a long time, but if we don’t have further progress on safety, and we remain / become convinced that unaligned ASI is an existential threat, we would need to continually reassess how strong sanctions and enforcement needs to be to prevent existential catastrophe. But if we get a moratorium in non-rogue states, thankfully, we don’t need to answer these questions this decade, or maybe even next.

Davidmanheim 22 Sep 2023 7:19 UTC
2 points
0 ∶ 0
in reply to: Gerald Monroe’s comment on: AI Pause Will Likely Backfire
For individual tasks, sure, you can implement verifiers, though I think it becomes quickly unwieldy, but there’s no in-principle reason we cannot do this. But you cannot create AGI with a restricted model—we cannot define the space of what outputs we want, otherwise it’s by definition a narrow AI.

Davidmanheim 21 Sep 2023 21:07 UTC
2 points
0 ∶ 0
in reply to: Nora Belrose’s comment on: AI Pause Will Likely Backfire
Conditional on “the efforts” working is hooribly underspecified. A global governance mechanism run by a new extranational body with military powers monitoring and stopping production of GPUs, or a standard treaty with a multi-party inspection regime?

Davidmanheim 21 Sep 2023 20:30 UTC
2 points
0 ∶ 0
in reply to: Gerald Monroe’s comment on: AI Pause Will Likely Backfire
If your action space is small enough to have what you want it to not be able to do programmatically described in terms of its outputs, and your threat model is complete, it works fine.

Davidmanheim 21 Sep 2023 10:49 UTC
2 points
0 ∶ 0
in reply to: Matthew_Barnett’s comment on: The Case for AI Safety Advocacy to the Public
That makes sense—I was confused, since you said different things, and some of them were subjunctive, and some were speaking about why you disagree with proposed analogies.

Given your perspective, is loss-of-control from more capable and larger models not a foreseeable harm? If we see a single example of this, and we manage to shut it down, would you then be in favor of a regulate-before-training approach?

Davidmanheim 21 Sep 2023 8:21 UTC
4 points
1 ∶ 0
in reply to: Gerald Monroe’s comment on: AI Pause Will Likely Backfire
I disagree with a lot of particulars here, but don’t want to engage beyond this response because your post feels like it’s not about the substantive topic any more, it’s just trying to mock an assumed / claimed lack of understanding on my part. (Which would be far worse to have done if it were correct.)
That said, if you want to know more about my background, I’m eminently google-able, and while you clearly have more background in embedded safety compliant systems, I think you’re wrong on almost all of the details of what you wrote as it applies to AGI.
Regarding your analogy to Mobileye’s approach, I’ve certainly read the papers, and had long conversations with people at Mobileye about their safety systems. I even had one of their former That’s why I think it’s fair to say that you’re fundamentally mischaracterizing the idea of “Responsibility-Sensitive Safety”—it’s not about collision avoidance per se, it’s about not being responsible for accidents, in ways that greatly reduce the probability of such accidents. This is critical for understanding what it does and does not guarantee. More critically, for AI systems, this class of safety guarantee doesn’t work because you need a complete domain model as well as a complete failure mode model in order to implement a similar failsafe. I’ve even written about how RSS could be extended, and that explains why it’s not applicable to AGI back in 2018 - but found that many of my ideas were anticipated by Christiano’s 2016 work (which that post is one small part of,) and had been further refined in the context of AGI since then.

Davidmanheim 21 Sep 2023 7:53 UTC
4 points
1 ∶ 0
in reply to: Matthew_Barnett’s comment on: The Case for AI Safety Advocacy to the Public
“I mean a global, indefinite moratorium on the development of frontier models until it is safe to proceed.”
I think this is distinctly different from what you claim. In any actual system implementing a pause, the model developer is free “to prove their specific implementations are safe,” and go ahead. The question is what the default is—and you’ve implied elsewhere in this thread that you think that developers should be treated like pre-1938 drug manufacturers, with no rules.
If what you’re proposing is instead that there needs to be a regulatory body like the FDA to administer the rules and review cases when a company claims to have sufficient evidence of safety when planning to create a model, with a default rule that it’s illegal to develop the model until reviewed, instead of a ban with a review for exceptions when a company claims to have sufficient evidence of safety when planning to create a model, I think the gap between our positions is less blurry than it is primarily semantic.

Davidmanheim 21 Sep 2023 7:44 UTC
3 points
1 ∶ 0
in reply to: Matthew_Barnett’s comment on: The Case for AI Safety Advocacy to the Public
That’s not how modern risk assessment works. Risk registers and mitigation planning are based on proactively identifying risk. To the extent that this doesn’t occur before something is built and/or deployed, at the very least, it’s a failure of the engineering process. (It also seems somewhat perverse to argue that we need to protect innovation in a specific domain by sticking to the way regulation happened long in the past.)
And in the cases where engineering and scientific analysis has identified risks in advance, but no regulatory system is in place, the legal system has been clear that there is liability on the part of the producers. And given those widely acknowledged dangers, it seems clear that if model developers ignores a known or obvious risk, they are criminally liable for negligence. This isn’t the same as restricting by-default-unsafe technologies like drugs and buildings, but at the very least, I think you should agree that one needs to make an argument for why ML models should be treated differently than other technologies with widely acknowledged dangers.

Davidmanheim 20 Sep 2023 21:04 UTC
9 points
1 ∶ 0
in reply to: Matthew_Barnett’s comment on: The Case for AI Safety Advocacy to the Public
...and this risk isn’t predictable on priors?
(But if we had decades of experience with computer-based systems not reliably doing exactly what we wanted, you’d admit that this degree of caution on systems we expect to be powerful would be reasonable?)

Davidmanheim 20 Sep 2023 20:55 UTC
3 points
0 ∶ 0
in reply to: RobBensinger’s comment on: Comments on Manheim’s “What’s in a Pause?”
(Edit to add: The below is operating entirely in the world where we don’t get an indefinite moratorium initially. I strongly agree about the preferability of an idefinite governance regime, though I think that during a multi-year pause with review mechanisms we’ll get additional evidence that either safety is possible, and find a path, or conclude that we need a much longer time, or it’s not possible at all.)

If you grant that a pause increases danger by reducing the ability of society and safety researchers to respond, and you don’t think doom is very, very likely even with extreme effort, then it’s reasonable that we would prefer, say, a 50% probability of success controlling AI given 3 years over a 10% probability of success given a 2-year pause then only 18 months. Of course, if you’re 99.95% sure that we’re doomed given 3 years, it makes sense to me that the extra 6 months of survival moving the probability to 99.99% would seem more worth it. But I don’t understand how anyone gets that degree of confidence making predictions. (Superforecasters who have really fantastic predictive accuracy and calibration tend to laugh at claims like that.)
That said, I strongly agree that this isn’t an acceptable bet to make. We should not let anyone play Russian roulette with all of humanity, and even if you think it’s only a 0.05% probability of doom (again, people seem very obviously overconfident about their guesses about the future,) that seems like a reason to insist that other people get to check your work in saying the system is safe.
Finally, I don’t think that you can buy time to get governments to step in quite the way you’re suggesting, after a pause. That is, if we get a pause that then expires, we are going to need tons of marginal evidence after that point to get an even stronger response, even once it’s in the Overton window. But the evidence we’d need is presumably not showing up, or not showing up as quickly, because there isn’t as much progress. So either the pause is extended indefinitely without further evidence, or we’ll see a capabilities jump, and that increases risks.
And once we see the capabilities jump after a pause expires, it seems plausible that any stronger response will be far too slow. (It might be OK, they might re-implement the pause, but I don’t see a reason for confidence in their ability or willingness to do so.) And in general, unless there are already plans in place that they can just execute, governments react on timescales measured in years.
(Note for everyone reading that all of this assumes, as I do, that the risk is large and will become more obvious as time progresses and we see capabilities continue to outpace reliable safety. If safety gets solved during a pause, I guess it was worth it, or maybe even unnecessary. But I’m incredibly skeptical.)

Davidmanheim 20 Sep 2023 20:31 UTC
13 points
4 ∶ 0
in reply to: Matthew_Barnett’s comment on: The Case for AI Safety Advocacy to the Public
Just as a partial reply, it seems weird to me to claim that the groups both best able to demonstrate safety and most technically capable of doing so—the groups making the systems—should get a free pass to tell other people to prove what they are doing is unsafe. That’s a really bad incentive.

And I think basically everywhere in the western world, for the past half century or so, we require manufacturers and designers to ensure their products are safe, implicitly or explicitly. Houses, bridges, consumer electronics, and children’s toys all get certified for safety. Hell, we even license engineers in most countries and make it illegal for non-licensed engineers to do things like certify building safety. That isn’t a democratic control, but it’s clearly putting the burden of proof on the makers, not those claiming it might be unsafe.

Davidmanheim 20 Sep 2023 20:26 UTC
11 points
4 ∶ 0
in reply to: Vaipan’s comment on: The Case for AI Safety Advocacy to the Public
I don’t understand why doing outreach to EAs specifically to convince them of this would be an effective focus. It seems none of important, tractable, or neglected. The people in EA who you’re talking about, I think, are a small group compared to the general population, they aren’t in high-leverage positions to change things relevant to AI, and they are already aware of the topic and have not bought the arguments.

Davidmanheim

How could a mora­to­rium fail?

How could a moratorium fail?