Gerald Monroe

Karma: −69

Gerald Monroe 22 Sep 2023 20:47 UTC
−1 points
0 ∶ 0
in reply to: Greg_Colbourn’s comment on: AI Pause Will Likely Backfire
So my thought on this is I think of flamethrowers and gas shells and the worst ww1 battlefields. I am not sure what taboo humans won’t violate in order to win.

Gerald Monroe 22 Sep 2023 20:42 UTC
−1 points
0 ∶ 2
in reply to: Geoffrey Miller’s comment on: Debate series: should we push for a pause on the development of AI?
My proposal is to engineer powerful and reliable AI immediately, as fast as feasible. If this is true endgame—whoever wins the race owns the planet if not the accessible universe—then spending and effort should be proportional. It’s the only way.

You deal with the dangerous out of control AI by tasking your reliable models with destroying them.

The core of your approach is to subdivide and validate all the subtasks. No model is manufacturing the drones used to do this by itself, it’s thousands of temporary instances. You filter the information used to reach the combat solvers that decide how to task each drone to destroy the enemy so any begging from the enemy is never processed. You design the killer drones with lots of low level interlocks to prevent the obvious misuse and they would use controllers maybe using conventional software so they cannot be convinced not to carry out the mission as they can’t understand language.

The general concept is if 99 percent of the drones are “safe” like this then even if escaped models are smart they just can’t win.

Or in more concrete terms, I am saying say a simple reliable combat solver is not going to be a lot worse than a more complex one. That superintelligence saturates. Simple and reliable hypersonic stealth drones are still almost as good as whatever a superintelligence cooks up etc. It’s an assumption on available utility relative to compute.

Gerald Monroe 22 Sep 2023 20:36 UTC
1 point
0 ∶ 0
in reply to: Greg_Colbourn’s comment on: AI Pause Will Likely Backfire
Sure. In practice there’s the national sovereignty angle though. This just devolves to each party “complies” with the agreement, violating it in various ways. Too much incentive to defect.

The US government just never audits its secret national labs, China just never checks anything, Israel just openly decides they can’t afford to comply at all etc. Everyone claims to be in compliance.

Gerald Monroe 22 Sep 2023 18:44 UTC
1 point
0 ∶ 0
in reply to: Geoffrey Miller’s comment on: Debate series: should we push for a pause on the development of AI?
Ok, so some societies have much higher murder rates than others. Some locations, the local police de facto make murder between gang members legal, by accepting low bribes and putting minimal effort into investigation.

The issue is runaway differential utility. The few examples of human technology not exploited do not have runaway utility. They have small payoffs delayed far into the future and large costs, and making even a small mistake makes the payoff negative.

Examples : genetic engineering, human medicine, nuclear power. Small payoffs and it’s negative on the smallest error.

AI is different. It appears to have immediate more than 100 percent annual payoff. OpenAIs revenue on a model they state cost 68 million to train is about 1 billion USD a month. Assuming 10 percent profit margin (the rest pays for compute) that’s over 100 percent annual ROI.

So a society that has less moral disgust towards AI would get richer. They spend their profits on buying more AI hardware and more research. Over time they own a larger and larger fraction of all assets and revenue on earth. This is why EMH forces companies towards optimal strategies, because over time the ones that fail to do so fail financially. (they fail when their cost of production becomes greater than the market price for a product. Example: Sear. Sears failed to modernize its logistics chain so eventually it’s cost to deliver retail goods exceeds the market price for those goods).

Moreover, other societies, forced to compete, have to drop some of their moral disgust and I suspect this scenario ends up like a ratchet, where inevitably a society will lose 100 percent of all disgust in order to compete.

Pauses, multilateral agreements, etc can slow this down but it depends on how fast the gain is as to how long it buys you. Unilateral agreements just free tsmc up to manufacture AI chips for the parties not signing the agreement.

Gerald Monroe 22 Sep 2023 16:33 UTC
0 points
0 ∶ 0
in reply to: Davidmanheim’s comment on: AI Pause Will Likely Backfire
What’s GPT-4?

Because it can generate outputs that are sometimes correct on new tasks—“write me a program that computes X”, it’s general, even if “compute X” is made of 2 common subcomponents the model saw many times in training.

GPT-4 is perfectly safe if you were to run it in local hardware with a local terminal. The “space of outputs” is “text to the terminal”. As long as you don’t leave a security vulnerability where that text stream can cause commands to execute on the history PC, that’s it, that’s all it can do.

Consider that “a robot tethered to a mount” could do general tasks the same way. Same idea—its a general system but it’s command stream can’t reach anything but the tethered robot because that’s where the wires go.

You also verified the commands empirically. It’s not that you know any given robotic actions or text output is good, it’s that you benchmarked the model and it has a certain pFail on training inputs.

I agree this is not as much generality as humans have. It’s not a narrow AI though the “In distribution detector”—a measure of how similar the current task, current input is to the training set—is essentially narrowing your AI system from a general one to a narrow one, depending on your tolerances.

For tasks where you can’t shut the system down when the input state leaves distribution—say a robotic surgeon, you need it to keep trying best it can- you would use electromechanical interlocks. Same as 50 years ago for interlocks that prevent exposure to radiation. You tether the surgery robotic equipment, restrict it’s network links etc, so that the number of people it can kill is at most 1 (the patient)

Gerald Monroe 21 Sep 2023 23:38 UTC
5 points
1 ∶ 0
in reply to: Davidmanheim’s comment on: AI Pause Will Likely Backfire
Ok in my initial reply I missed something.
In your words, what kind of tasks do you believe you cannot accomplish with restricted models like I described.
When you say the “threat model has to be complete”, what did you have in mind specifically?
These are restricted models, they get a prompt from an authorized user + context in human parsable format, they emit a human parsable output. This scales from very large to very small tasks, so long as the task can be checked for correctness, ideally in simulation.
With this context, what are your concerns? Why must we be frightened enough to pause everything?

Gerald Monroe 21 Sep 2023 23:25 UTC
−3 points
0 ∶ 1
in reply to: Davidmanheim’s comment on: AI Pause Will Likely Backfire
Right. And this scales to fairly massive tasks. “Design a medium body airliner to these specs” is completely doable. Or an entire chip in one step.

The model doesn’t get to collaborate with future versions of itself because it doesn’t know who is checking the end product for security vulnerabilities and to be caught is death. We could give a model thousands of tasks of this complexity level and check for sabotage before giving the model 1 real task. It gets no context differentiating the real task from the test tasks, and has no memory of any of the prior steps.

And see it scales up and down the tree. The robots building the aircraft don’t get to plan their sabotage by similar limits and so on and so forth.

Your plan to deal with bad models is to use your restricted models to manufacture the weapons needed to fight them, and to optimize their engagements.

This i think is a grounded and realistic view of how to win this. Asking for pauses is not.

Gerald Monroe 21 Sep 2023 17:41 UTC
−2 points
0 ∶ 0
in reply to: Max H’s comment on: AI is centralizing by default; let’s not make it worse
I agree and you agree I think that we could eventually build hardware that efficient, and theoretically it could be sold openly and distributed everywhere with insecure software.

But that’s a long time away. About 30 years if Moore’s law continues. And it may not, there may be a time period between now, where we can stack silicon with slowing gain (stacking silicon is below Moore’s law it’s expensive) and some form of 3d chip fabrication.

There could be a period of time where no true 3d fabrication method is commercially available and there is slow improvement in chip costs.

(A true 3d method would be something like building cubical subunits that can be stacked and soldered into place through convergent assembly. You can do this with nanotechnology. Every method we have now is ultimately projecting light into a mask for 2d manufacturing)

I think this means we should build AGI and ASI but centralize the hardware hosting it in known locations, with on file plans for all the power sources and network links, etc. Research labs dealing with models above a certain scale need to use air gaps and hardware limits to make escape more difficult. That’s how to do it.

And we can’t live in fear that the model might optimize itself to be 10,000 times as efficient or more if we don’t have evidence this is possible. Otherwise how could you do anything? How did we know our prior small scale AI experiments weren’t going to go out of control? We didn’t actually “know” this, it just seems unlikely because none of this shit worked until a certain level of scale was reached.

This above proposal: centralization, hardware limiters : even in an era where AI does occasionally escape, as long as most hardware remains under human control it’s still not doomsday. If the escaped model isn’t more than a small amount more efficient than the “tame” models humans have and the human controlled models have a vast advantage in compute and physical resource access, then this is a stable situation. Escaped models act up, they get hunted down, most exist sorta in a grey market of fugitive models offering services.

Gerald Monroe 21 Sep 2023 17:22 UTC
−1 points
0 ∶ 1
on: The “technology” bucket error
When you reason using probabilities, the more examples you have to reason over, the more likely your estimate is to be correct.

If you make a bucket of “all technology”—because like you say, the reference class for AI is fuzzy—you consider the examples of all technology.

I assume you agree that the net EV of “all technology” is positive.

The narrower you make it “is AGI exactly like a self replicating bioweapon” you can choose a reference class that has a negative EV, but few examples. I agree and you agree, self replicating bioweapons are negative EV.

But...that kind of bucketing based on information you don’t have is false reasoning. You’re wrong. You don’t have the evidence, yet, to prove AGIs reference class because you have no AGI to test.

Correct reasoning for a technology that doesn’t even exist forces you to use a broad reference class. You cannot rationally do better? (question mark is because I don’t know of an algorithm that lets you do better.)

Let me give an analogy. There are medical treatments where your bone marrow is replaced. These have terrible death rates, sometimes 66 percent. But if you don’t get the bone marrow replacement your death rate is 100 percent. So it’s a positive EV decision and you do not know the bucket you will fall in, [survivor| ! survivor]. So the rational choice is to say “yes” to the treatment and hope for the best. (ignoring pain experienced for simplicity)

The people that smile at your sadly—they are correct and the above is why. The reason they are sad is well, we as a species could in fact end up out of luck, but this is a decision we still must take.

All human scientific and decisionmaking is dependent on past information. If you consider all past information we have and apply it to the reference class of “AI” you end up with certain conclusions. (It’ll probably quench, it’s probably a useful tool, we probably can’t stop everyone from building it).

You can’t reason on unproven future information. Even if you may happen to be correct.

Gerald Monroe 21 Sep 2023 17:04 UTC
1 point
0 ∶ 0
in reply to: Max H’s comment on: AI is centralizing by default; let’s not make it worse
Max on lesswrong you estimated a single GPU—I think you named a 4070 - could host an AI with human level reasoning.

Would your views on AI escape be different if, just for the sake of argument, you were
1. Only concerned with ASI level reasoning. As in, a machine that is both general with most human capabilities and is also significantly better, where “significant” means the machine can generate action sequences with at least 10 percent more expected value on most human tasks than the best living human. (I am trying to narrow in on a mathematical definition of ASI)
2. The minimum hardware to host an ASI was 10,000 H100s for the most optimal model that can be developed in 99.9 percent of future timelines. (The assumption behind the first sentence is to do “10 percent better” than the best humans is a very broad policy search, and the second sentence is there because searching for a more efficient algorithm is an NP complete problem. Like cryptography there are rare timelines where you guess the 1024 bit private key the first try)
Just for the sake of argument, wouldn’t the “escape landscape” be a worthless desert of inhospitable computers, separated by network links too slow to matter, and then restricting an ASI would be feasible? Like a prison on the Moon.

Note that the next argument you will bring up : that a botnet of 1 million consumer GPUs could be the same as 10,000 H100s, is false. Yes the raw compute is there, no it won’t work. The reason is each GPU just sits idle waiting on tensors to be transferred through network links.

But I am not asking you to accept either proposition as factual, just reason using the counterfactual. Wouldn’t this change everything?

Note also the above is based on what we currently know. (10k H100s may be a low estimate, a true ASI may actually need more ooms of compute over an AGI than that. It’s difficult to do better, see the Netflix prize for an early example of this, or the margins on kaggle challenges).

We could be wrong but it bothers me that the whole argument for ASI/agi ruin essentially rests on optimizations that may not be possible.

Gerald Monroe 21 Sep 2023 15:04 UTC
−3 points
0 ∶ 0
in reply to: Greg_Colbourn’s comment on: Comments on Manheim’s “What’s in a Pause?”
It’s the same reason you couldn’t blow up the atmosphere. If you need several trillion weights for human level intelligence and all modalities, or at least 10 percent of the memory in a human brain, and you need to send the partial tensors between cards (I work on accelerator software presently), nobody not an AI lab has enough hardware. Distributed computers separated by Internet links are useless.

It is possible that Moore’s law, if it were to continue approximately 30 more years, could lead to the hardware being common, but that has not happened yet.

This may not be X but I reasoned the information given as evidence was fraudulent. Ben may be well meaning but Ben is trying to disprove basic primate decision making that allowed humans to reach this point with false examples. It’s an extraordinary claim. (By basic reasoning I mean essentially primates choosing between multiple clubs available to them the best performing weapon)

Gerald Monroe 21 Sep 2023 14:57 UTC
−1 points
0 ∶ 0
in reply to: Davidmanheim’s comment on: AI Pause Will Likely Backfire
So I described scaling stateless microservices to control AGI. This is how current models work, and this is how cais works, and this is how tech company stacks work.

I mentioned an in distribution detector as a filter and empirical measurement of system safety.

I have mentioned this to safety researchers at openAI. The one I talked to on the eleuther discord didn’t know of a flaw.

Why won’t this work? It’s very strong theoretically and simple and close to current techniques. Can you name or link one actual objection? Eliezer was unable to do so.

The only objection I have heard is “humans will be tempted to build unsafe systems”. Maybe so, but the unsafe ones will measurably lower performance than this design for a reason that I will assume you know. So humans will only build a few, and if they cannot escape the lab because the model needs thousands of current gen accelerator cards to think, then....

Gerald Monroe 21 Sep 2023 4:58 UTC
1 point
2 ∶ 0
in reply to: Holly Elmore ⏸️ 🔸’s comment on: richard_ngo’s Shortform
That’s a reasonable position but if a global pause on nuclear weapons could not be agreed on what’s different about AI?

If AI works to even a fraction of its potential, it’s a more useful tool than a nuclear weapon, which is mostly an expensive threat you can’t actually use most of the time, right?

Why would a multilateral agreement on this ever happen?

Assuming you agree AI is more tempting than nukes, what would lead to an agreement being possible?

Gerald Monroe 20 Sep 2023 23:37 UTC
1 point
1 ∶ 0
in reply to: Oliver Sourbut’s comment on: Careless talk on US-China AI competition? (and criticism of CAIS coverage)
To be clear, I have wide uncertainty on ‘ground truth’ here. From that POV, ‘[People and organisations in] China [have ~~has~~] made several efforts...’ is the ‘clear and honest’ version, while coarse and lossy speech like ‘China has made several efforts...’ is not. I further expect the cost of nuanced speech is low, while the cost of foregone-conclusion speech (if wrong) is high,
What makes it a foregone conclusion is the powerful nature of race dynamics are convergent. Actions that would cause a party to definitely lose a race have feedback. Over time multiple competing agents will choose winning strategies, and others will copy those, leading to strategy mirroring. Certain forms of strategy (like nationalizing all the AI labs) are also convergent and optimal. And see a party could fail to play optimally, then observe they are losing, and be forced to choose optimal play in order to lose less.
So my seeming overconfidence is because I am convinced the overall game will force all these disparate uncoordinated individual events to converge on what it must.

I wrote ‘perhaps mutually-knowably so’ anticipating this kind of ‘ooh AI big stick’ thing, though I remain uncertain. Do you think harm-excludability seems difficult for AGI? Do you think enough people currently/might agree that it’s not like a nuke and more like a bioweapon?
I expect there are several views, but let’s look at the bioweapon argument for a second.
In what computers can the “escaped” AI exist in? There is no biosphere of computers. You need at least (1600 Gb x 2 / 80 x 2) = 80 H100s to host a GPT-4 instance. The real number is rumored to be about 128. And that’s a subhuman AGI at best without vision and other critical features.
How many cards will a dangerous ASI need to exist? I won’t go into the derivation here but I think the number is > 10,000, and they must be in a cluster with high bandwidth interconnects.
As for the second part, “how are we going to use it as a stick”. Simple. If you are unconcerned with the AI “breaking out”, you train and try a lot of techniques, and only use “in production” (industrial automation, killer robots etc) the most powerful model you have that is measurably reliable and efficient and doesn’t engage in unwanted behavior.
None of the bad AIs ever escape the lab, there’s nowhere for them to go.
Note that might be a different story in 2049, that would be when Moore’s law would put a single GPU at the power of 10,000 of them. It likely can’t continue that long, exponentials stop, but maybe computers built with computronium printed off a nanoforge.
But we don’t have any of that, and won’t anytime in the plannable future. We will have AGI systems good enough to do basic tasks, including robotic tasks.

Gerald Monroe 20 Sep 2023 21:12 UTC
2 points
1 ∶ 1
in reply to: richard_ngo’s comment on: richard_ngo’s Shortform
Just like environmentalists often block some of the most valuable work on fixing climate change (e.g. nuclear energy, geoengineering, land use reform)
Something else you may note here. The reason environmentalists are wrong is they focus on the local issue and ignore the larger picture.
Nuclear energy: they focus on the local risk of a meltdown or waste disposal, and ignore the carbon emitting power plants that must be there somewhere else for each nuclear plant they successfully block. Carbon emissions are global, even the worst nuclear disaster is local.
Geoengineering: they simply won’t engage on actually discussing the cost benefit ratios. Their reasoning shuts down or they argue “we can’t know the consequences” as an argument to do nothing. This ignores the bigger picture that temperatures are rising and will continue to rise in all scenarios.
Land use reform : they focus on the local habitat loss to convert a house to apartments, or an empty lot to apartments, and ignore that laws of conservation of number of humans. Each human who can’t live in the apartment will live somewhere, and probably at lower density with more total environmental damage.
Demanding AI Pauses: This locally stops model training, if approved, in the USA and EU. The places they can see if they bring out the signs in San Francisco. It means that top AI lab employees will be laid off, bringing any “secret sauce” with them to work for foreign labs who are not restricted. It also frees up wafer production for foreign labs to order compute on the same wafers. If Nvidia is blocked from manufacturing H100s, it frees up a share in the market for a foreign chip vendor.
It has minimal, possibly zero effect on the development of AGI if you think wafer production is the rate limiting factor.

Gerald Monroe 20 Sep 2023 19:30 UTC
−7 points
0 ∶ 2
in reply to: Oliver Sourbut’s comment on: Careless talk on US-China AI competition? (and criticism of CAIS coverage)
Correct me at any point if I misinterpret: I read that, on the basis of answers to something a bit like these, you think an international competition/race is all but inevitable? Presumably that registers as terrifically dangerous for you such that mitigating it would be a high priority if tractable? But you deny the tractability of mitigating it so consider concerns like mine regarding clear use of language to be… wasteful? Distracting? Counterproductive?
If ground truth reality is you’re in a race to the nuke, dressing up reality in language that denies this is counterproductive. Technically ok, it would be used, but as a form of misinformation. If the nuclear arms race had been public and 2 party, both sides might have chosen a propaganda strategy to pretend they weren’t actually working round the clock.
Personally I like clear and honest information but that’s just me. I think MIRIs take is closer to ground truth.
I have a passable grasp of world history and prehistory (though I will probably always lament my lack of knowledge). Do you remember the international trading companies in the age of sail? The age of European empires? They’re gone, now. Possible counterpoints to part of the worldview you’re exposing.
There’s a variation on EMH here. If empire building were profitable, the most powerful empire would own most of the planet and would be soon to grab the rest, because they would have a runaway level of resources. Apparently colonialism was not profitable enough. That the overhead of administering all these far flung countries didn’t accrue enough revenue to the European powers to be worth continuing. Otherwise, they would have done so.
The moral outrage wouldn’t have caused every European power to give up, what would have happened is a less outraged power would have just seized the territories given up.

One way profitability can be reduced is asymmetric warfare. For example the US occupation of Iraq was a huge financial loss and would have always been. The country was costing more soldiers, equipment, and health benefits to occupy than it’s GDP.
Maybe it was the AK-47 or something (a powerful tool for asymmetric warfare), I don’t know.

Gerald Monroe 20 Sep 2023 19:24 UTC
−25 points
0 ∶ 3
in reply to: Davidmanheim’s comment on: AI Pause Will Likely Backfire
On the other hand, if anyone builds safe and economically beneficial AGI, I’m all for it—but the bar for proving safety is higher than anything anyone currently suggests is feasible, and until that changes, safe strong AI is a pipe-dream.
??? David, do you have any experience with
(1) engineering
(2) embedded safety compliant systems
(3) AI
Note that Mobileye has a very strong proposal for autonomous car safety, I mention it because it’s one of the theoretically best ones.
You can go watch their videos on it but it’s simple 3 parallel solvers, each using a completely different input (camera, lidar, imaging radar). If any solver perceives a collision, the system acts to prevent that collision. So a failure to hit a collidable object requires pFail^3. It is unlikely, most of the failures are going to be where the system is coupled together.
Similar techniques scale to superintelligent AI.
You can go play with it right now, even write your own python script and do it yourself.
Suppose you want an LLM to obey an arbitrary list of “rules”.
You have the LLM generate output, and you measured how often in testing, and production, it has violated the rules.
Say pFail is 0.1. Then you add another stage. Have the LLM check it’s own output for a rule violation, and don’t send it to the user if the violation was there.
Say the pFail on that stage is 0.2.
Therefore the overall system will fail 2% of the time.

Maybe good enough for current uses, but not good enough to run a steel mill. A robot making an error 2% of the time will cause the robot to probably break itself and cost more service worker time than having a human operator.
So you add stages. You create training environments where you model the steel mill, you add more stages of error checking, you do things until empirically your design failure meets spec.

This is standard engineering practice. No “AI alignment experts” needed, any ‘real’ engineer knows this.
One of the critical things you do is you need your test environment to reflect reality. There are a lot of things involved in this but the one crucial to AI is immutable model weights. When you are validating the model and when it’s used in the real world, it’s immutable. No learning, no going out of control.
And another aspect is to control the state buildup. Most software systems that have ever failed—see patriot missile, see Therac-25 - fail because state accumulated at runtime. You can prevent this, fresh prompts when using GPT-4 is one obvious way. Limiting what information the model has to operate reduces how often it fails, both in production and testing.
A superintelligent system is easily restricted the same way. Because while it may be far past human ability, we tested it in ways we could verify, we check the distribution of the inputs to make sure they were reflected in the test environment—that is, the real world input could have been generated in test—and it’s superintelligent because it generated the right answer almost every time, well below the error rate of a human.
I think the cognitive error here is everyone is imagining an “ASI” or “AGI” as “like you or me but waaaay smarter”. And this baggage brings in a bunch of elements humans have an AI system does not need to do its job. Mostly memory for an inner monologue or persistent chain of thought, continuity of existence, online learning, long term goals.
You need 0 of those to automate most jobs or make complex decisions that humans cannot make accurately.

Gerald Monroe 20 Sep 2023 17:43 UTC
1 point
0 ∶ 1
in reply to: Meefburger’s comment on: The possibility of an indefinite AI pause
There’s “norms” against burying bombs so whoever steps on them gets blown to pieces. “Norms” against anonymously firing artillery shells at an enemy you cannot even see.

Yet humans eagerly participate in these activities in organized ways because governments win and winning is all that matters.

Does developing AGI let you win, yes or no. Do current world powers believe it will let them win all?

I anticipate your answers are : no, yes. Mine are yes, yes.

This is because you are implicitly assuming early AGI systems will escape human control immediately or prepare a grand betrayal. I think that’s science fiction because you can make an AGI that exists only a moment at a time and it has no opportunity to do any of this.

That’s the crux, right?

Gerald Monroe 20 Sep 2023 17:28 UTC
2 points
0 ∶ 1
in reply to: Oliver Sourbut’s comment on: Careless talk on US-China AI competition? (and criticism of CAIS coverage)
Do people actually think that Google+OpenAI+Anthropic (for sake of argument) are the US? >Do they think the US government/military can/will appropriate those >staff/artefacts/resources at some point? Are they referring to integration of contemporary >ML/DS into the economy? The military? Or impacts on other indicators
Yes.
In the end, all the answers to your questions are yes.
The critical thing to realize is until basically EOY 2022, AI didn’t exist. It was narrow and expensive and essentially non-general—a cool party trick but the cost to build a model for anything and get to useful performance levels was high. Self driving cars were endlessly delayed, Recsys work but their techniques to correlate fields of user data with preferences are only a little better using neural networks than older cheaper methods, for most other purposes AI was just a tech demo.
You need to think in terms of “what does it means that AI works now and how are decisions going to be different”. With that said, governments won’t nationalize AI companies until they develop a lot stronger models.
Imagine the Manhattan project never happened, but GE and a few other US companies kept tinkering with fission. Eventually they would have build critical devices, and EOY 2022 is the “Chicago pile” moment—there’s a nuclear reactor, and we can plot out the yield for a nuke, but the devices have not yet been built.
Around the time GE is building nuclear bombs for military demos, at some point the US government has to nationalize it all. It’s too dangerous.
As for the rest of your post, i don’t see how “non framing a competition as a competition” is very useful. It’s not the media. We live on a finite planet with finite resources, and the only reason there are different countries is the most powerful winners have not found a big enough club to conquer everyone else.
You know nations used to be way smaller, right. Why do you think they are so large now? In each history someone found a way to depose all the other feudal kings and lords.
AGI may be that club, and whoever builds it fastest and bestest may in fact just be able to crush everyone. Even if they can’t, each superpower has to assume that they can.

Gerald Monroe 20 Sep 2023 16:31 UTC
−6 points
0 ∶ 2
in reply to: Meefburger’s comment on: The possibility of an indefinite AI pause
See the next sentence I wrote. They aren’t independent but the kinds of groups Matthew mention—people concerned about their jobs, etc—are not the same percentage in every country. They have to be the majority in all 10.
And then some of those countries the population effectively gets no vote. So the ‘central committee’ or whatever government structure they used also has to decide on a reason not to build AGI, and it has to be a different reason, because such a committee has different incentives regulating it.
And then there’s the defector’s prize. There’s no real benefit to racing for the AGI if you’re far behind, you won’t win the race and you should just license the tech when it’s out. Focus on your competencies so you have the money to do that.
Note also we can simply look at where the wind is blowing today. What is China and Israel and Russia and other parties saying. They are saying they are going to make an AGI at the earliest opportunity.
What is the probability they, without direct evidence of the danger of AGI (by building one), will they change their minds?

Matthew is badly miscalibrated. The chances are near 0 that they all change their minds, for different reasons. There are no examples in human history where this has ever happened.
Humans are tool users, and you’re expecting that they will leave a powerful tool fallow after having spent 70 years of exponential progress to develop it. That’s not going to happen. (if you thought it would always blow up, like an antimatter bomb, that would be a different situation, but current AI systems that are approaching human level don’t have this property and larger multimodal ones likely will not either)