Remmelt

Karma: 1,411

See explainer on why AGI could not be controlled enough to stay safe:
https://www.lesswrong.com/posts/xp6n2MG5vQkPpFEBH/the-control-problem-unsolved-or-unsolvable

I post here about preventing unsafe AI.

Note that I’m no longer part of EA, because of overreaches I saw during my time in the community (core people leading technocratic projects with ruinous downside risks, a philosophy based around influencing consequences over enabling collective choice-making, and a culture that’s bent on proselytising both while not listening deeply enough to integrate other perspectives).

Remmelt Feb 27, 2025, 12:19 PM
4 points
2 votes
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Remmelt’s comment on: If AI is in a bubble and the bubble bursts, what would you do?
Update: back up to 50% chance.
Noting Microsoft’s cancelling of data center deals. And the fact the ‘AGI’ labs are still losing cash, and with DeepSeek are competing increasingly on a commodity product.

Remmelt Feb 12, 2025, 4:32 AM
3 points
3 votes
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Remmelt’s comment on: If AI is in a bubble and the bubble bursts, what would you do?
Update: 40% chance.
I very much underestimated/missed the speed of tech leaders influencing the US government through the Trump election/presidency. Got caught flat-footed by this.
I still think it’s not unlikely for there to be an AI crash as described above within the next 4 years and 8 months but it could be from levels of investment much higher than where we are now. A “large reduction in investment” at that level looks a lot different than a large reduction in investment from the level that markets were at 4 months ago.

Remmelt Jan 18, 2025, 6:03 AM
3 points
2 votes
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: gergo’s comment on: Funding Case: AI Safety Camp 11
We ended up having a private exchange about it.
Basically, organisers spend more than half of their time on general communications and logistics to support participants get to work.
And earmarking stipends to particular areas of work seems rather burdensome administratively, though I wouldn’t be entirely against it if it means we can cover more people’s stipends.
Overall, I think we tended not to allow differentiated fundraising before because it can promote internal conflicts, rather than having people come together to make the camp great.

Remmelt Jan 17, 2025, 4:40 PM
3 points
2 votes
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
on: What do you mean with ‘alignment is solvable in principle’?
Here’s how I specify terms in the claim:
- AGI is a set of artificial components, connected physically and/or by information signals over time, to in aggregate sense and act autonomously over many domains.
  - ‘artificial’ as configured out of a (hard) substrate that can be standardised to process inputs into outputs consistently (vs. what our organic parts can do).
  - ‘autonomously’ as continuing to operate without needing humans (or any other species that share a common ancestor with humans).
- Alignment is at the minimum the control of the AGI’s components (as modified over time) to not (with probability above some guaranteeable high floor) propagate effects that cause the extinction of humans.
- Control is the implementation of (a) feedback loop(s) through which the AGI’s effects are detected, modelled, simulated, compared to a reference, and corrected.

Remmelt Nov 11, 2024, 3:32 AM
2 points
1 vote
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Remmelt’s comment on: If AI is in a bubble and the bubble bursts, what would you do?
Update: reverting my forecast back to 80% chance likelihood for these reasons.

Remmelt Nov 11, 2024, 3:30 AM
3 points
2 votes
Overall karma indicates overall quality.
1 ∶ 0
Total points: 1
Agreement karma indicates agreement, separate from overall quality.
in reply to: Nicholas Kross’s comment on: An AI crash is our best bet for restricting AI
I’m also feeling less “optimistic” about an AI crash given:
1. The election result involving a bunch of tech investors and execs pushing for influence through Trump’s campaign (with a stated intention to deregulate tech).
2. A military veteran saying that the military could be holding up the AI industry like “Atlas holding the globe”, and an AI PhD saying that hyperscaled data centers, deep learning, etc, could be super useful for war.
I will revise my previous forecast back to 80%+ chance.
What links here?
- Remmelt's comment on If AI is in a bubble and the bubble bursts, what would you do? by Remmelt (Nov 11, 2024, 3:32 AM; 2 points)

Remmelt Oct 30, 2024, 1:21 PM
2 points
1 vote
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
on: OpenAI defected, but we can take honest actions
Just found a podcast on OpenAI’s bad financial situation.

It’s hosted by someone in AI Safety (Jacob Haimes) and an AI post-doc (Igor Krawzcuk).

https://kairos.fm/posts/muckraiker-episodes/muckraiker-episode-004/

Remmelt Oct 22, 2024, 4:55 AM
4 points
2 votes
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Vasco Grilo🔸’s comment on: Why Stop AI is barricading OpenAI
There are bunch of crucial considerations here. I’m afraid it would take too much time to unpack those.

Happy though to have had this chat!

Remmelt Oct 21, 2024, 10:37 AM
4 points
2 votes
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Vasco Grilo🔸’s comment on: Why Stop AI is barricading OpenAI
As a 1st approximation, I assume humans will be selecting AIs which benefit them, not AIs which maximally increase economic growth.
The problem here is that AI corporations are increasingly making decisions for us.
See this chapter.

Corporations produce and market products to increase profit (including by replacing their fussy expensive human parts with cheaper faster machines that do good-enough work.)

To do that they have to promise buyers some benefits, but they can also manage to sell products by hiding the negative externalities. See cases Big Tobacco, Big Oil, etc.

Remmelt Oct 21, 2024, 2:21 AM
10 points
2 votes
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Vasco Grilo🔸’s comment on: Why Stop AI is barricading OpenAI
I am open to a bet similar to this one.
I would bet on both, on your side.

Potentially relatedly, I think massive increases in unemployment are very unlikely.
I see you cite statistics of previous unemployment rates as an outside view, compensating against the inside view. Did you look into the underlying rate of job automation? I’d be curious about that. If that underlying rate has been trending up over time, then there is a concern that at some point the gap might not be filled with re-employment opportunities.

AI Safety inside views are wrong for various reasons in my opinion. I agree with many of Thorstad’s views you cited (eg. critiquing how fast take-off, orthogonality thesis and instrumental convergence relies on overly simplistic toy models, missing the hard parts about machinery coherently navigating an environment that’s more complex than just the machinery itself).
There are arguments that you are still unaware of, which mostly come from outside of the community. They’re less flashy, involving longer timelines. For example, it considers why the standardisation of hardware and code allows for extractive corporate-automation feedback loops.

To learn about why superintelligent AI disempowering humanity would be the lead up to the extinction of all current living species, I suggest digging into substrate-needs convergence.

I gave a short summary in this post:
- AGI is artificial. The reason why AGI would outperform humans at economically valuable work in the first place is because of how virtualisable its code is, which in turn derives from how standardisable its hardware is. Hardware parts can be standardised because their substrate stays relatively stable and compartmentalised. Hardware is made out of hard materials, like the silicon from rocks. Their molecular configurations are chemically inert and physically robust under human living temperatures and pressures. This allows hardware to keep operating the same way, and for interchangeable parts to be produced in different places. Meanwhile, human “wetware” operates much more messily. Inside each of us is a soup of bouncing and continuously reacting organic molecules. Our substrate is fundamentally different.
- The population of artificial components that constitutes AGI implicitly has different needs than us (for maintaining components, producing components, and/or potentiating newly connected functionality for both). Extreme temperature ranges, diverse chemicals – and many other unknown/subtler/more complex conditions – are needed that happen to be lethal to humans. These conditions are in conflict with our needs for survival as more physically fragile humans.
- These connected/nested components are in effect “variants” – varying code gets learned from inputs, that are copied over subtly varying hardware produced through noisy assembly processes (and redesigned using learned code).
- Variants get evolutionarily selected for how they function across the various contexts they encounter over time. They are selected to express environmental effects that are needed for their own survival and production. The variants that replicate more, exist more. Their existence is selected for.
- The artificial population therefore converges on fulfilling their own expanding needs. Since (by 4.) control mechanisms cannot contain this convergence on wide-ranging degrees and directivity in effects that are lethal to us, human extinction results.

Remmelt Oct 21, 2024, 1:40 AM
3 points
10 votes
Overall karma indicates overall quality.
0 ∶ 5
Total points: 5
Agreement karma indicates agreement, separate from overall quality.
on: Remmelt’s Quick takes
Donation opportunities for restricting AI companies:
- Pause AI: protests and lobbying
- Stop AI: is barricading OpenAI
- For Humanity: a podcast on risks.
- FoxGlove: a legal non-profit targeting big tech scaling.
- Disruption Network Institute: whistleblowers and researchers reporting on the AI military misuses.
- Distributed AI Research Institute: AI ethics researchers giving general AI advocates hell and advocating for specialised models serving communities.
- European Guild for AI Regulation: lobbying in the EU against the data scraping that makes large models possible.
- Concept Art Association: lobbying in the EU against the data scraping that makes large models possible.
In my pipeline:
- funding a ‘horror documentary’ against AI by an award-winning documentary maker (got a speculation grant of $50k)
- funding lawyers in the EU for some high-profile lawsuits and targeted consultations with EU AI Office.
If you’re a donor, I can give you details on their current activities. I worked with staff in each of these organisations. DM me.
What links here?
- OpenAI defected, but we can take honest actions by Remmelt (Oct 21, 2024, 8:41 AM; 19 points)
- OpenAI defected, but we can take honest actions by Remmelt (LessWrong; Oct 21, 2024, 8:41 AM; 17 points)

Remmelt Oct 19, 2024, 4:41 AM
2 points
1 vote
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: jackva’s comment on: Who looked into extreme nuclear meltdowns?
Hey, my apologies for taking even longer to reply (had family responsibilities this month).
I will read that article on why Chernobyl-style events are not possible with modern reactors. Respecting you for the amount of background research you must have done in this area, and would like to learn more.

Remmelt Oct 19, 2024, 2:47 AM
4 points
2 votes
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Vasco Grilo🔸’s comment on: Why Stop AI is barricading OpenAI
Although I think the probability of human extinction over the next 10 years is lower than 10^-6.
You and I actually agree on this with respect to AI developments. I don’t think the narratives I read of a large model recursively self-improving itself internally make sense.
I wrote a book for educated laypeople explaining how AI corporations would cause increasing harms leading eventually to machine destruction of our society and ecosystem.
Curious for your own thoughts here.

Remmelt Oct 15, 2024, 5:58 AM
1 point
2 votes
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: NickLaing’s comment on: Why Stop AI is barricading OpenAI
Basically I’m upvoting what you’re doing here, which I think is more important than the text itself.
Thanks for recognising the importance of doing the work itself. We are still scrappy so we’ll find ways to improve over time.

especially that you should have run this past a bunch of media savvy people before releasing
If you know anyone with media experience who might be interested to review future drafts, please let me know.

I agree we need to improve on our messaging.

Remmelt Oct 15, 2024, 1:02 AM
2 points
1 vote
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Jason’s comment on: Why Stop AI is barricading OpenAI
This is great! Appreciating your nitty-gritty considerations
(1) There’s a good chance the outcome will be some form of “catch and release”—it’s usually easier to deal with isolated protestors who do not cause violence or significantly damage property in this manner rather than by pursuing criminal charges to trial.
“Catch and release” is what’s happening right now. However, as we keep repeating the barricades, and as hopefully more and more protestors join us, it would highly surprise me if the police and court system just allow us to keep barricading OpenAI’s office entrances without pursuing criminal charges. If police would allow us to keep doing it with at most overnight time in jail, that would of course make doing the barricades easier.
(2) Generally, a trial court acquittal does not establish precedent binding as to future cases. And the California appellate courts are not generally known for being speedy.
This is insightful, thank you! I was personally skeptical that we could set a precedent that would give us a free pass every time. But I am not a lawyer.
Checking: Are you saying that the trial courts generally do not set a precedent (or more specifically do not set a precedent that allows “repeat offenses”)?
And would we only be able to take this up to the appellate court if the judge rules against us (the defendant), and we file an appeal?
[Asking for information only – whatever you share is not legal advice.]
(3) Even assuming there were some binding precedent establishing that you could not be criminally convicted for these actions, it doesn’t necessarily follow that the police or OpenAI could not physically remove you from the area
That is fine. Getting moved around by police is part of the process of doing the protests. As Sam said, when said the time comes, we’ll leave.
It also wouldn’t preclude OpenAI or others from filing a civil lawsuit against the protesters for damages or other relief.
I had not considered this! I’m not sure whether to see this as a serious personal risk to our organisers in the US (though Sam doesn’t have money to hand over anyway), and/or a great opportunity to publicly present arguments of OpenAI’s harmful/illegal activities.
many jurisdictions allow a property owner to use reasonable force to eject a trespasser—and it’s going to tough for a small group to blockade a company headquarters or data center without trespassing on the AI company’s property.
My impression is that the gates we are barricading of the office OpenAI co-rents are actually connected to the public side-walk. In that case we are sitting on the side-walk and not on private property (Sam can correct me here).
For the data centers, I’d expect us to be often be sitting on private property, and for private security to be present and ready to act.

Remmelt Oct 14, 2024, 5:22 PM
4 points
4 votes
Overall karma indicates overall quality.
0 ∶ 1
Total points: 1
Agreement karma indicates agreement, separate from overall quality.
in reply to: MaxRa’s comment on: Why Stop AI is barricading OpenAI
Thanks for the kind words!
I personally think it would be helpful to put more emphasis on how OpenAI’s reckless scaling and releases of models is already concretely harming ordinary folks (even though no major single accident has shown up yet).
Eg.
- training on personal/copyrighted data
- job losses because of the shoddy replacement of creative workers (and how badly OpenAI has treated workers it paid)
- school ‘plagiarism’, disinformation, and deepfakes
- environmental harms of scaling compute.

Remmelt Oct 14, 2024, 5:13 PM
17 points
9 votes
Overall karma indicates overall quality.
1 ∶ 0
Total points: 1
Agreement karma indicates agreement, separate from overall quality.
in reply to: titotal’s comment on: Why Stop AI is barricading OpenAI
Thank you for the specific feedback on the press release! I broadly agree with it, and I think it’s going to be useful for improving future texts.

Remmelt Oct 14, 2024, 5:10 PM
2 points
1 vote
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Oscar Sykes’s comment on: Why Stop AI is barricading OpenAI
Yes, thank you for mentioning this
I made a mistake in checking that number.
See also comment here.

Remmelt Oct 10, 2024, 2:08 PM
2 points
1 vote
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
in reply to: Remmelt’s comment on: If AI is in a bubble and the bubble bursts, what would you do?

To clarify for future reference, I do think it’s likely (80%+) that at some point over the next 5 years there will be a large reduction in investment in AI and a corresponding market crash in AI company stocks, etc, and that both will continue to be for at least three months.

Update: I now think this is 90%+ likely to happen (from original prediction date).

Remmelt Sep 8, 2024, 3:26 AM
2 points
1 vote
Overall karma indicates overall quality.
0 ∶ 0
Total points: 0
Agreement karma indicates agreement, separate from overall quality.
on: Who looked into extreme nuclear meltdowns?
But it’s weird that I cannot find even a good written summary of Bret’s argument online (I do see lots of political podcasts).
I found an earlier scenario written by Bret that covers just one nuclear power plant failing and that does not discuss the risk of a weakening magnetic field.