Remmelt

Karma: 1,177

See explainer on why AGI could not be controlled enough to stay safe:
https://www.lesswrong.com/posts/xp6n2MG5vQkPpFEBH/the-control-problem-unsolved-or-unsolvable

I post here about preventing unsafe AI.

Note that I’m no longer part of EA, because of overreaches I saw during my time in the community (core people leading technocratic projects with ruinous downside risks, a philosophy based around influencing consequences over enabling collective choice-making, and a culture that’s bent on proselytising both while not listening deeply enough to integrate other perspectives).

Remmelt Oct 15, 2024, 5:58 AM
1 point
0 ∶ 0
in reply to: NickLaing’s comment on: Why Stop AI is barricading OpenAI
Basically I’m upvoting what you’re doing here, which I think is more important than the text itself.
Thanks for recognising the importance of doing the work itself. We are still scrappy so we’ll find ways to improve over time.

especially that you should have run this past a bunch of media savvy people before releasing
If you know anyone with media experience who might be interested to review future drafts, please let me know.

I agree we need to improve on our messaging.

Remmelt Oct 15, 2024, 1:02 AM
2 points
0 ∶ 0
in reply to: Jason’s comment on: Why Stop AI is barricading OpenAI
This is great! Appreciating your nitty-gritty considerations
(1) There’s a good chance the outcome will be some form of “catch and release”—it’s usually easier to deal with isolated protestors who do not cause violence or significantly damage property in this manner rather than by pursuing criminal charges to trial.
“Catch and release” is what’s happening right now. However, as we keep repeating the barricades, and as hopefully more and more protestors join us, it would highly surprise me if the police and court system just allow us to keep barricading OpenAI’s office entrances without pursuing criminal charges. If police would allow us to keep doing it with at most overnight time in jail, that would of course make doing the barricades easier.
(2) Generally, a trial court acquittal does not establish precedent binding as to future cases. And the California appellate courts are not generally known for being speedy.
This is insightful, thank you! I was personally skeptical that we could set a precedent that would give us a free pass every time. But I am not a lawyer.
Checking: Are you saying that the trial courts generally do not set a precedent (or more specifically do not set a precedent that allows “repeat offenses”)?
And would we only be able to take this up to the appellate court if the judge rules against us (the defendant), and we file an appeal?
[Asking for information only – whatever you share is not legal advice.]
(3) Even assuming there were some binding precedent establishing that you could not be criminally convicted for these actions, it doesn’t necessarily follow that the police or OpenAI could not physically remove you from the area
That is fine. Getting moved around by police is part of the process of doing the protests. As Sam said, when said the time comes, we’ll leave.
It also wouldn’t preclude OpenAI or others from filing a civil lawsuit against the protesters for damages or other relief.
I had not considered this! I’m not sure whether to see this as a serious personal risk to our organisers in the US (though Sam doesn’t have money to hand over anyway), and/or a great opportunity to publicly present arguments of OpenAI’s harmful/illegal activities.
many jurisdictions allow a property owner to use reasonable force to eject a trespasser—and it’s going to tough for a small group to blockade a company headquarters or data center without trespassing on the AI company’s property.
My impression is that the gates we are barricading of the office OpenAI co-rents are actually connected to the public side-walk. In that case we are sitting on the side-walk and not on private property (Sam can correct me here).
For the data centers, I’d expect us to be often be sitting on private property, and for private security to be present and ready to act.

Remmelt Oct 14, 2024, 5:22 PM
4 points
0 ∶ 1
in reply to: MaxRa’s comment on: Why Stop AI is barricading OpenAI
Thanks for the kind words!
I personally think it would be helpful to put more emphasis on how OpenAI’s reckless scaling and releases of models is already concretely harming ordinary folks (even though no major single accident has shown up yet).
Eg.
- training on personal/copyrighted data
- job losses because of the shoddy replacement of creative workers (and how badly OpenAI has treated workers it paid)
- school ‘plagiarism’, disinformation, and deepfakes
- environmental harms of scaling compute.

Remmelt Oct 14, 2024, 5:13 PM
17 points
1 ∶ 0
in reply to: titotal’s comment on: Why Stop AI is barricading OpenAI
Thank you for the specific feedback on the press release! I broadly agree with it, and I think it’s going to be useful for improving future texts.

Remmelt Oct 14, 2024, 5:10 PM
2 points
0 ∶ 0
in reply to: Oscar Sykes’s comment on: Why Stop AI is barricading OpenAI
Yes, thank you for mentioning this
I made a mistake in checking that number.
See also comment here.

Remmelt Oct 10, 2024, 2:08 PM
2 points
0 ∶ 0
in reply to: Remmelt’s comment on: If AI is in a bubble and the bubble bursts, what would you do?

To clarify for future reference, I do think it’s likely (80%+) that at some point over the next 5 years there will be a large reduction in investment in AI and a corresponding market crash in AI company stocks, etc, and that both will continue to be for at least three months.

Update: I now think this is 90%+ likely to happen (from original prediction date).

Remmelt Sep 8, 2024, 3:26 AM
2 points
0 ∶ 0
on: Who looked into extreme nuclear meltdowns?
But it’s weird that I cannot find even a good written summary of Bret’s argument online (I do see lots of political podcasts).
I found an earlier scenario written by Bret that covers just one nuclear power plant failing and that does not discuss the risk of a weakening magnetic field.

Remmelt Sep 8, 2024, 2:15 AM
3 points
0 ∶ 0
in reply to: ABlank’s comment on: Who looked into extreme nuclear meltdowns?
The quotes from the OECD Nuclear Energy Agency’s report were interesting.

moving nuclear fuel stored in pools into dry casket storage The extent to which we can do this is limited because spent fuel must be stored for one to ten years in spent fuel pools while the shorter-lived isotopes decay before it’s ready to be moved to dry cask storage.

I did not know this. I added an edit to the post: “nuclear waste already stored in pools for 5 years”.

I don’t think an environmental radioisotope release can realistically give people across the world acute radiation syndrome.

Can you cite evidence and/or reasoning? Considering intake of the radioactive isotopes (through skin contact, breathing, and water/food consumption) as well? This feels like the kinda thing where modelling error can happen easily. I share some of your skepticism here, but also am an amateur in this area.

Remmelt Sep 8, 2024, 2:06 AM
2 points
0 ∶ 0
in reply to: jackva’s comment on: Who looked into extreme nuclear meltdowns?
Thanks, looking forward to reading your thoughts!

Remmelt Sep 6, 2024, 4:33 AM
2 points
0 ∶ 0
in reply to: jackva’s comment on: Who looked into extreme nuclear meltdowns?
Regarding 1., I would value someone who has researched this give more insight into:

A. How long diesel generators could be expected to be supplied with diesel when there is some continental electricity outage of a year (or longer). This is hard to judge. My intuition is that society would be in chaos and that maintaining diesel supplies would be extremely tough to manage.

B. What is minimally required in the long process of shutting down a nuclear power plant? Including but not limited to diesel or other backup generator supplies.

Regarding 2., I do not see how a meltdown as happened with Chernobyl is not thought to be possible with current reactor designs. Even if current reactor designs have more safeguards, the fundamental problem still seems the same. You have constantly heating fissile materials that require cooling water to not overheat. And there are some potentially flammable substances in the area of the (used up) fuel rods too. Could you clarify more specifically what you think are important differences here?

~

I felt a little confused about why the summary sounded conspiratorial to you. The way I meant to write it is “a chain of plausible-seeming but neglected events happen, and lead to catastrophe”. At most this seems a story of humans not being incentivised to act to prevent tail risks? Am I missing something?

Now, Bret Weinstein does seem to have a leaning toward potentially falsely identifying conspiracies. While I agree COVID vaccine negative side-effects seem underreported (I’ve seen it with family), I am also skeptical of Bret’s coverage. I also think there is another perspective on the Evergreen College situation, which is that the college had been slow to implement thoughtful reforms (whatever that means) to address structural discrimination and that tensions boiled over.

At the same time, I want to be careful about not dismissing explained reasoning outright because the person seems on the face of it unreasonable. There are people outside EA whom I talked with who initially seemed in some ways very unreasonable, until I probed their perspective enough, found a more comprehensive way of looking at the problem, and surprisingly changed my mind.

Would also want to encourage maintaining some openness around looking for conflicting perspectives. You have been advising a climate strategy of a developing a mix of energy solutions, including nuclear energy. You would adjust your recommendations if you ever found rigorous reasoning for there being an unacceptable risk of nuclear meltdown, right?

Remmelt Aug 26, 2024, 12:45 PM
4 points
1 ∶ 0
in reply to: Linda Linsefors’s comment on: Invitation to lead a project at AI Safety Camp (Virtual Edition, 2025)
Fixed it! You can use either link now to share with your friends.

Remmelt Aug 23, 2024, 12:35 AM
2 points
0 ∶ 0
in reply to: Remmelt’s comment on: If AI is in a bubble and the bubble bursts, what would you do?
Igor Krawzcuk, an AI PhD researcher, just shared more specific predictions:
“I agree with ed that the next months are critical, and that the biggest players need to deliver. I think it will need to be plausible progress towards reasoning, as in planning, as in the type of stuff Prolog, SAT/SMT solvers etc. do.
I’m 80% certain that this literally can’t be done efficiently with current LLM/RL techniques (last I looked at neural comb-opt vs solvers, it was _bad_), the only hope being the kitchen sink of scale, foundation models, solvers _and_ RL
…
If OpenAI/Anthropic/DeepMind can’t deliver on promises of reasoning and planning (Q*, Strawberry, AlphaCode/AlphaProof etc.) in the coming months, or if they try to polish more turds into gold (e.g., coming out with GPT-Reasoner, but only for specific business domains) over the next year, then I would be surprised to see the investments last to make it happen in this AI summer.”
https://x.com/TheGermanPole/status/1826179777452994657

Remmelt Aug 20, 2024, 5:58 AM
3 points
0 ∶ 2
on: If AI is in a bubble and the bubble bursts, what would you do?
To clarify for future reference, I do think it’s likely (80%+) that at some point over the next 5 years there will be a large reduction in investment in AI and a corresponding market crash in AI company stocks, etc, and that both will continue to be for at least three months.
Ie. I think we are heading for an AI winter.
It is not sustainable for the industry to invest 600+ billion dollars per year in infrastructure and teams in return for relatively little revenue and no resulting profit for major AI labs.
At the same time, I think that within the next 20 years tech companies could both develop robotics that self-navigate multiple domains and have automated major sectors of physical work. That would put society on a path to causing total extinction of current life on Earth. We should do everything we can to prevent it.

Remmelt Aug 20, 2024, 4:20 AM
2 points
0 ∶ 0
in reply to: huw’s comment on: If AI is in a bubble and the bubble bursts, what would you do?
What are you thinking about in terms of pre-harm enforcement?
I’m thinking about advising premarket approval – a requirement to scope model designs around prespecified uses and having independent auditors vet the safety tests and assessments.

Remmelt Aug 6, 2024, 3:10 AM
2 points
0 ∶ 0
in reply to: Habryka [Deactivated]’s comment on: Lessons from the FDA for AI
The report is focussed on preventing harms of technology to people using or affected by that tech.

It uses FDA’s mandate of premarket approval and other processes as examples of what could be used for AI.

Restrictions to economic productivity and innovation is a fair point of discussion. I have my own views on this – generally I think the negative assymetry around new scalable products being able to do massive harm gets neglected by the market. I’m glad the FDA exists to counteract that.

The FDA’s slow response to ramping up COVID vaccines during the pandemic is questionable though, as one example. Getting a sense there is a lot of problems with bureacracy and also industrial capture with FDA.

The report does not focus on that though.

Remmelt Jul 28, 2024, 5:59 AM
2 points
0 ∶ 0
in reply to: Rebecca’s comment on: 80,000 hours should remove OpenAI from the Job Board (and similar EA orgs should do similarly)
They mentioned that line at the top of the 80k Job board.

Still do I see.

“Handpicked to help you tackle the world’s most pressing problems with your career.”

https://jobs.80000hours.org/

Remmelt Jul 27, 2024, 2:47 AM
4 points
0 ∶ 0
in reply to: Vasco Grilo🔸’s comment on: I bet Greg Colbourn 10 k€ that AI will not kill us all by the end of 2027
Ah, I wasn’t clear. To bet that AI will not kill us all by the end of 2027.

I don’t think that makes sense, given the world-complexity “AI” would need to learn and evolve and get tinkered to be able to navigate. I’ve had some conversations with Greg about this.

Remmelt Jul 27, 2024, 2:45 AM
2 points
0 ∶ 0
in reply to: Linch’s comment on: Émile P. Torres’s history of dishonesty and harassment
Change your mind in what way? Could you elaborate a bit?

Remmelt Jul 27, 2024, 2:42 AM
2 points
0 ∶ 0
in reply to: Rebecca’s comment on: 80,000 hours should remove OpenAI from the Job Board (and similar EA orgs should do similarly)
Could you quote which line you mean? Then I can mention where you can find it back

Remmelt Jul 19, 2024, 2:44 PM
9 points
0 ∶ 0
in reply to: Conor Barnes 🔶’s comment on: 80,000 hours should remove OpenAI from the Job Board (and similar EA orgs should do similarly)
We used to list roles that seemed more tangentially safety-related, but because of our reduced confidence in OpenAI...

This misses aspects of what used to be 80k’s position:
❝ In fact, we think it can be the best career step for some of our readers to work in labs, even in non-safety roles. That’s the core reason why we list these roles on our job board.
– Benjamin Hilton, February 2024

❝ Top AI labs are high-performing, rapidly growing organisations. In general, one of the best ways to gain career capital is to go and work with any high-performing team — you can just learn a huge amount about getting stuff done. They also have excellent reputations more widely. So you get the credential of saying you’ve worked in a leading lab, and you’ll also gain lots of dynamic, impressive connections.
– Benjamin Hilton, June 2023 - still on website

80k was listing some non-safety related jobs:
– From my email on May 2023:
– From my comment on February 2024: