See explainer on why AGI could not be controlled enough to stay safe:
https://www.lesswrong.com/posts/xp6n2MG5vQkPpFEBH/the-control-problem-unsolved-or-unsolvable
Note: I am no longer part of EA because of the community’s/philosophy’s overreaches. I still post here about AI safety.
Is there an argument that it is impossible?
There is actually an impossibility argument. Even if you could robustly specify goals in AGI, there is another convergent phenonemon that would cause misaligned effects and eventually remove the goal structures.
You can find an intuitive summary here: https://www.lesswrong.com/posts/jFkEhqpsCRbKgLZrd/what-if-alignment-is-not-enough
Thanks! Also a good example of lots of complaints being prepared now by individuals
Actually, looks like there is a thirteenth lawsuit that was filed outside the US.
A class-action privacy lawsuit filed in Israel back in April 2023.
Wondering if this is still ongoing: https://www.einpresswire.com/article/630376275/first-class-action-lawsuit-against-openai-the-district-court-in-israel-approved-suing-openai-in-a-class-action-lawsuit
I agree that implies that those people are more inclined to spend the time to consider options. At least they like listening to other people give interesting opinions about the topic.
But we’re all just humans, interacting socially in a community. I think it’s good to stay humble about that.
If we’re not, then we make ourselves unable to identify and deal with any information cascades, peer proof, and/or peer group pressures that tend to form in communities.
Three reasons come to mind why OpenPhil has not funded us.
Their grant programs don’t match, and we have therefore not applied to them.They have fund individuals making early career decisions, our university-based courses, or programs that selectively support “highly talented” young people, or “high quality nuanced” communication. We don’t fit any of those categories.
We did sent in a brief application early 2023 though for a regrant covering our funds from FTX, which was not granted (same happened to at least one other field-building org I’m aware of).
AISC wasn’t contacted for bespoke grants – given OpenPhil’s fieldbuilding focuses shown above, and focus on technical research, academic programs, and governance organisations for the rest.
Also, even if we engage i with OpenPhil staff, I heard that another AIS field-building organisation had to make concessions and pick research focusses OpenPhil staff like, in order to ensure they get funding from OpenPhil. Linda and I are not prepared to do that.
I did not improve things by critiquing OpenPhil online for supporting AGI labs. I personally stand by the content of the critiques, but it was also quite in your face, and I can imagine they did not like that.
Whatever I critique about collaborations between longtermist orgs and AGI labs can be associated back to AI Safety Camp is or the area I run at AI Safety Camp. I want to be more mindful how I word my critiques in the future.
Does that raise any new questions?
They’re not quite doing a brand partnership.
But 80k has featured various safety researchers working at AGI labs over the years. Eg. see OpenAI.
So it’s more like 80k has created free promotional content, and given their stamp of approval of working at AGI labs (of course ‘if you weigh up your options, and think it through rationally’ like your friends).
Do you mean OP, as in Open Philanthropy?
Hi Conor,
Thank you.
I’m glad to see that you already linked to clarifications before. And that you gracefully took the feedback, and removed the prompt engineer role. I feel grateful for your openness here.
It makes me feel less like I’m hitting a brick wall. We can have more of a conversation.
~ ~ ~
The rest is addressed to people on the team, and not to you in particular:
There are grounded reasons why 80k’s approaches to recommending work at AGI labs – with the hope of steering their trajectory – has supported AI corporations to scale. While disabling efforts that may actually prevent AI-induced extinction.
This concerns work on your listed #1 most pressing problem. It is a crucial consideration that can flip your perceived total impact from positive to negative.
I noticed that 80k staff responses so far started by stating disagreement (with my view), or agreement (with a colleague’s view).
This doesn’t do discussion of it justice. It’s like responding to someone’s explicit reasons for concern that they must be “less optimistic about alignment”. This ends reasoned conversations, rather than opens them up.
Something I would like to see more of is individual 80k staff engaging with the reasoning.
If some employees actually have the guts to whistleblow on current engineering malpractices…
Plenty of concrete practices you can whistleblow on that will be effective in getting society to turn against these companies:
The copying of copyrighted and person-identifying information without permission (pass on evidence to publishers and they will have a lawsuit feast).
The exploitation and underpayment of data workers and coders from the Global South (inside information on how OpenAI staff hid that they instructed workers in Kenya to collect images of child sexual abuse, anyone?).
The unscoped misdesign and failure to test these systems for all the uses the AI company promotes.
The extent of AI hardware’s environmental pollution.
Pick what you’re in a position to whistleblow on.
Be very careful to prepare well. You’re exposing a multi-billion-dollar company. First meet in person with an attorney experienced in protecting whistleblowers.
Once you start collecting information, make photographs with your personal phone, rather than screenshots or USB copies that might be tracked by software. Make sure you’re not in line of sight of an office camera or webcam. Etc. Etc.
Preferably, before you start, talk with an experienced whistleblower about how to maintain anonymity. The more at ease you are there, the more you can bide your time, carefully collecting and storing information.
If you need information to get started, email me at remmelt.ellen[a/}protonmail<d0t>com.
~ ~ ~
But don’t wait it out until you can see some concrete dependable sign of “extinction risk”. By that time, it’s too late.
If labs do engage in behavior that is flagrantly reckless, employees can act as whistleblowers.
This is the crux for me.
If some employees actually have the guts to whistleblow on current engineering malpractices, I have some hope left that having AI safety researchers at these labs still turns out “net good”.
If this doesn’t happen, then they can keep having conversations about x-risks with their colleagues, but I don’t quite see when they will put up a resistance to dangerous tech scaling. If not now, when?
Internal politics might change
We’ve seen in which directions internal politics change, as under competitive pressures.
Nerdy intellectual researchers can wait that out as much as they like. That would confirm my concern here.
Another problem with the NIST approach is an overemphasis on solving for identified risks, rather than precautionary principle (just don’t use scaled tech that could destabilise society at scale), or on preventing and ensuring legal liability for designs that cause situationalised harms.
Safety-washing of AI is harmful as it gives people an out, a chance to repeat the line “well at least they are allegedly doing some safety stuff”, which is a convenient distraction from the fact that AI labs are knowingly developing a technology that can cause human extinction. This distraction causes otherwise safety-conscious people to invest in or work in an industry that they would reconsider if they had access to all the information.
Very much agreed.
It is an extreme claim to make in that context, IMO.
I think Benjamin made it to be nuanced. But the nuance in that article is rather one-sided.
If anything, the nuance should be on the side of identifying any ways you might accidentally support the development of dangerous auto-scaling technologies.
First do, no harm.
Do you think it would be better if no one who worked at OpenAI / Anthropic / Deepmind worked on safety?
It depends on what you mean with ‘work on safety’.
Standard practice for designing machine products to be safe in other established industries is to first narrowly scope the machinery’s uses, the context of use, and the user group.
If employees worked at OpenAI / Anthropic / Deepmind on narrowing their operational scopes, all power to them! That would certainly help. It seems that leadership, who aim to design unscoped automated machinery to be used everywhere for everyone, would not approve though.
If working on safety means in effect playing close to a ceremonial role, where even though you really want to help, you cannot hope to catch up with the scaling efforts, I would reconsider. In other industries, when conscientious employees notice engineering malpractices that are already causing harms across society, sometimes one of them has the guts to find an attorney and become a whistleblower.
Also, in that case, I would prefer the AGI labs to not hire for those close-to-ceremonial roles.
I’d prefer them to be bluntly transparent to people in society that they are recklessly scaling ahead, and that they are just adding local bandaids to the ‘Shoggoth’ machinery.
Not that that is going to happen anyway.
If those organizations devoted less of their budget to safety?
If AGI labs can devote their budget to constructing operational design domains, I’m all up.
Again, that’s counter to the leaders’ intentions. Their intention is to scale everywhere and rely on the long-term safety researchers to tell them that there must be some yet-undiscovered general safe control patch.
so individuals should avoid potentially increasing the pool of talent from which they can hire?
I think we should avoid promoting AGI labs as a place to work at, or a place that somehow will improve safety. One of the reasons is indeed that I want us to be clear to idealistic talented people that they should really reconsider investing their career into supporting such an organisation.
BTW, I’m not quite answering from your suggested perspective of what an AGI lab “should do”.
What feels relevant to me is what we can personally consider to do – as individuals connected into larger communities – so things won’t get even worse.
This was an insightful and concise overview. Thank you.
Consider Bridges v South Wales Police, where the court found in favour of Bridges on some elements not because the AI system was biased, but because a Data Protection Impact Assessment (DPIA) had not been carried out. Put simply, SWP hadn’t made sure it wasn’t biased. A DPIA is a foundation-level document in almost any compliance procedure.
This is an interesting anecdote.
It reminds me of how US medical companies having to go through FDA’s premarket approval process for software designed for prespecified uses holds them from launching medical software willy-nilly on the market. If they did release before FDA approval, they are quite likely to see regulatory action (and/or be held liable in court).
Thanks for clarifying.
Note that we are focussing here on decisions at the individual level.
There are limitations to that.
See my LessWrong comment.
I spent time digging into Uganda Community Farm’s plans last year, and ended up becoming a regular donor. From reading the write-ups and later asking Anthony about the sorghum training and grain-processing plant projects, I understood Anthony to be thoughtful and strategic about actually helping relieve poverty in the Kamuli & Buyende region.
Here are short explainers worth reading:
https://www.ugandafarm.org/sorghum/
https://www.ugandafarm.org/plant-details/
UCF focusses on training farmers and giving them the materials and tools needed to build up their own incomes, which is a much more targeted approach than just transferring money (though need to account for differences in local income levels too).
Personally, I think the EA community often focussed on measuring and mapping out consequences of global poverty interventions from afar and not as much on enabling charity entrepreneurs on the ground who have first-hand contextual knowledge on what’s holding their community back. My sense is that robust approaches will tend to consider both.