Meta: I count 25 questionmarks in this “quick” poll, and a lot of the questions appear to be seriously confused. A proper response here would take many hours.
Take your scenario number 5, for instance. Is there any serious literature examining this? Are there any reasons why anyone would assign that scenario >epsilon probability? Do any decisions hinge on this?
Søren Elverlin
Karma: 210
We should slow AI down
Otherwise I expect AI will kill us
>The mean estimate was [that bees suffer] around 15% as intensely as people.
To clarify. Does this mean that when comparing:A horrible prison keeping 10.000 humans
A beehive with 70.000 bees
The estimate implies that some people feel that the beehive is a worse moral problem? This strongly contradicts my moral intuitions.
This seems to be of questionable effectiveness. Brief answers/challenges:
Evaluations are key input to ineffective governance. The safety frameworks presented by the frontier labs are “safety-washing”, more appropriately considered roadmaps towards an unsurvivable future.
Disagreement on AI capabilities underpin performative disagreements on AI Risk. As far as I know, there’s no recent published substantial such disagreement—I’d like sources for your claim, please.
We don’t need more situational awareness of what current frontier models can and cannot do in order to respond appropriately. No decision-relevant conclusions can be drawn from evaluations in the style of Cybench and Re-Bench.
I’m also practicing how to give good presentations and introductions to AI Safety. You can see my YouTube channel here:
You might also be interested in one of my older presentations, number 293, which is closer to what you are working on.
Feel free to book a half-hour chat about this topic with me on this link:
The provided source doesn’t show PauseAI affiliated people calling Sam Altman and Dario Amodei evil.
I do in fact believe that delaying AI by 5 years reduce existential risk by something like 10 percentage points.
Probably this thread isn’t the best place to hash it out, however.
Another org in the same space, comprised of highly competent and experienced/plugged in people would certainly be welcome, and plausibly could be more effective.
>PauseAI suffers from the same shortcomings most lobbying outfits do...
I’m confused about this section: Yes, this kind of lobbying is hard, and the impact of a marginal dollar is very unclear. The acc-side also have far more resources (probably; we should be vary of this becoming a Bravery Debate).This doesn’t feel like a criticism of PauseAI. Limited tractability is easily outweighed by a very high potential impact.
I strongly agree. Almost all of the criticism in this thread seem to start from assumptions about AI that are very far from those held by PauseAI. This thread really needs to be split up to factor that out.
As an example: If you don’t think shrimp can suffer, then that’s a strong argument against the Shrimp Welfare Project. However, that criticism doesn’t belong in the same thread as a discussion about whether the organization is effective, because the two subjects are so different.
Your link is broken—it looks like it’s been pasted twice.
Without Delay: There is a 10% chance of catastrophe.
With a Cautious Delay of 70 Years: The risk of catastrophe is reduced to 5%.
Could you post a link to anyone who has something like these probabilities? I would be quite surprised to learn that this was not an extremely niche set of assumptions.
It’s a long post, and it starts by talking about consciousness.
Does it contain any response to the classic case for AI Risk, e.g. Bostrom’s Superintelligence or Yudkowsky’s List of Lethalities?
Ajeya Cotra posted an essay on schlep in the context of AI 2 weeks ago:
https://www.planned-obsolescence.org/scale-schlep-and-systems/I find that many of the topics she suggests as ‘schlep’ are actually very exciting and lots of fun to work on. This is plausible why we see so much open source effort in the space of LLM-hacking.
What would you think of as examples of schlep in other EA areas?
The 2016 Caplan-Yudkowsky debate (https://www.econlib.org/archives/2016/03/so_far_my_respo.html) fizzled out, with Bryan not answering Eliezers last question. I’d like to know his answer
The Budapest Memorandum provided security assurances, not security guarantees. And I believe this war has already caused enough damage to Russia that we can’t talk about Russia “getting away with” the invasion.
The destruction of the Russian military should be expected to make the world safer primarily because it will prevent future Russian agression.
The police is not bound by the “No drama” rule. If you steal money, you can expect the police to be “dramatic” about it.
A single data point: At a party at EAG, I met a developer who worked at Anthropic. I asked for his p(DOOM), and he said 50%. He told me he was working on AI capability.
I inquired politely about his views on AI safety, and he frankly did not seem to have given the subject much thought. I do not recall making any joke about “selling out”, but I may have asked what effect he thought his actions would have on X-risk.I don’t recall anyone listening, so this was probably not the situation OP is referring to.
I appreciate cultural works creating common knowledge that the AGI labs are behaving strongly unethically.
As for the specific scenario, point 17 seems to be contradicted by the orthogonality thesis / lack of moral realism.
=Confusion in What mildest scenario do you consider doom?=
My probability distribution looks like what you call the MIRI Torch, and what I call the MIRI Logo: Scenarios 3 to 9 aren’t well described in the literature because they are not in a stable equilibrium. In the real world, once you are powerless, worthless and an obstacle to those in power, you just end up dead.
=Confusion in Minimum P(doom) that is unacceptable to develop AGI?=
For non-extreme values, the concrete estimate and the most of the considerations you mention are irrelevant. The question is morally isomorphic to “What percentage of the worlds population am I willing to kill in expectation?”. Answers such as “10^6 humans” and “10^9 humans” are both monstrous, even though your poll would rate them very differently.
These possible answers don’t become moral even if you think that it’s really positive that humans don’t have to work any longer. You aren’t allowed to do something worse than the Holocaust in expectation, even if you really really like space travel or immortality, or ending factory farming, or whatever. You aren’t allowed to unilaterally decide to roll the dice on omnicide even if you personally believe that global warming is an existential risk, or that it would be good to fill the universe with machines of your creation.