LessWrong dev & admin as of July 5th, 2022.
RobertM
Karma: 609
He proposes instituting an international treaty, which seems to be aiming for the reference class of existing treaties around the proliferation of nuclear and biological weapons. He is not proposing that the United States issue unilateral threats of nuclear first strikes.
This advocates for risking nuclear war for the sake of preventing mere “AI training runs”. I find it highly unlikely that this risk-reward payoff is logical at a 10% x-risk estimate.
All else equal, this depends on what increase in risk of nuclear war you’re trading off against what decrease in x-risk from AI. We may have “increased” risk of nuclear war by providing aid to Ukraine in its war against Russia, but if it was indeed an increase it was probably small and worth the trade-off[1] against our other goals (such as disincentivizing the beginning of wars which might lead to nuclear escalation in the first place). I think approximately the only unusual part of Eliezer’s argument is the fact that he doesn’t beat around the bush in spelling out the implications.
- ^
Asserted for the sake of argument; I haven’t actually demonstrated that this is true but my point is more that there are many situations where we behave as if it is obviously a worthwhile trade-off to marginally increase the risk of nuclear war.
- ^
Many things about this comment seem wrong to me.
Yudkowsky’s suggestions seem entirely appropriate if you truly believe, like him, that AI x-risk is probability ~100%.
These proposals would plausibly be correct (to within an order of magnitude) in terms of the appropriate degree of response with much lower probabilities of doom (i.e. 10-20%). I think you need to actually run the math to say that this doesn’t make sense.
unproven and unlikely assumptions, like that an AI could build nanofactories by ordering proteins to be mixed over email
This is a deeply distorted understanding of Eliezer’s threat model, which is not any specific story that he can tell, but the brute fact that something smarter than you (and him, and everyone else) will come up with something better than that.
In the actual world, where the probability of extinction is signficantly less than 100%, are these proposals valuable?
I do not think it is ever particularly useful to ask “is someone else’s conclusion valid given my premises, which are importantly different from theirs”, if you are attempting to argue against someone’s premises. Obviously “A ⇒ B” & “C” does not imply “B”, and it especially does not imply “~A”.
It seems like they will just get everyone else labelled luddites and fearmongerers, especially if years and decades go by with no apocalypse in sight.
This is an empirical claim about PR, which:
does not seem obviously correct to me
has little to say about the object-level arguments
falls into pattern of suggesting that people should optimize for how others perceive us, rather than optimizing for communicating our true beliefs about the world.
Separate from object-level disagreements, my crux that people can have inside-view models which “rule out” other people’s models (as well as outside-view considerations) in a way that leads to assigning very high likelihoods (i.e. 99%+) to certain outcomes.
The fact that they haven’t successfully communicated their models to you is certainly a reason for you to not update strongly in their direction, but it doesn’t mean much for their internal epistemic stance.
I don’t think you should be optimizing to avoid extreme views, but in favor of those with the most robust models, who can also communicate them effectively to the desired audience. I agree that if we’re going to be trying anything resembling public outreach it’d be good to have multiple voices for a variety of reasons.
On the first half of the criteria I’d feel good about Paul, Buck, and Luke. On the second half I think Luke’s blog is a point of evidence in favor. I haven’t read Paul’s blog, and I don’t think that LessWrong comments are sufficiently representative for me to have a strong opinion on either Paul or Buck.
They often organize or attend events, meetups, or workshops that are focused on or friendly to polyamory, such as EA Polyamory, Poly Speed Dating, or Rationalist Polyamory. They also often join or contribute to online groups, forums, or platforms that are focused on or friendly to polyamory, such as Polyamory EA, Polyamory LessWrong, or Polyamory Rationality.
Of the six events/groups/etc listed here, the only one that seems to actually exist is “Poly Speed Dating”, which has several events/groups/etc bearing the same name in multiple places. That also happens to be the only one unrelated to the EA/rationality communities.
I suspect this was written using a modern LLM, given the structure, verbosity, and apparent hallucinations of non-existent groups.
When they asked a different Bay Area rationality organiser, they were told that their talk on diversity may have been “epistemically weak” and “not truth-seeking” enough.
So, to clarify, a guess from an unrelated party about why this talk might have resulted in a lack of an invitation pattern-matched to language used by other people in a way that has no (obvious to me) relationship to blacklists...?
I’m not sure what this was intended to demonstrate.
I am curious how you would distinguish a blacklist from the normal functioning of an organization when making hiring decisions. I guess maybe “a list of names with no details as to why you want to avoid hiring them” passed around between organizations would qualify as the first but not the second? I obviously can’t say with surety that no such thing exists elsewhere, but I would be pretty surprised to learn about any major organizations using one.
I haven’t included any names in any of my comments.
I’m not discussing naming the accuser, but the accused.
I do not think we have an obligation to avoid discussing object-level details of sexual assault claims when those claims have already been made publicly, if it seems like discussing them would otherwise be useful.
I’m pretty confused about what’s going on here. The person who made this accusation made it on Twitter under their real name using an unlocked account, and the accusation remains public to date. Is the concern here that the accused did not previously know of the accusation against them, but would be made aware of it by this discussion?
(I’m not sure whether I’d want them named in absence of a request to the contrary, but I don’t understand the implied threat model and think other explanations for the request are plausible, given the whole “public tweet” thing.)
Basically what Jason said, yes. The process described sounds reasonable but my prior, even given this post, was that it would sound reasonable.
Just noting that this reply seems to be, to me, very close to content-free, in terms of addressing object-level concerns. I think you could compress it to “I did due diligence” without losing very much.
If you’re constrained in your ability to discuss things on the object-level, i.e. due to promises to keep certain information secret, or other considerations like “discussing policy work in advance of it being done tends to backfire”, I would appreciate that being said explicitly. As it is, I can’t update very much on it.
ETA: to be clear, I’m not sure I how I feel about the broader norm of requesting costly explanations when something looks vaguely off. My first instinct is “against”, but if I were to adopt a policy of not engaging with such requests (unless they actually managed to surface something I’d consider a mistake I didn’t realize I’d made), I’d make that policy explicit.
No, sorry, I meant that at the time the feature was released (a few months ago), it didn’t have any karma requirement.
Clarifying a couple of points:
Crossposting used to be totally unrestricted; it now requires a user to have 100 karma on both LW and the EA Forum (regardless of which side they’re crossposting from) to use the feature
While historically most development was driven by the LW team, in the last year or so the EA Forum team has hired more engineers and is now larger than the LW team by headcount (and very likely by lines-of-code shipped, or whatever other metric you want to use to analyze “how much stuff did they do”).
I don’t think that the top-level comment is particularly responsive to the post, except insofar as it might have taken the title as a call to action (and then ignored the rest of the post). It’s also quite vague. But I agree that a ban seems like an unusually harsh response, absent additional context which supports the “provocation” interpretation.
Speak the truth, even if your voice trembles
Yes, I agree that there’s a non-trivial divide in attitude. I don’t think the difference in discussion is surprising, at least based on a similar pattern observed with the response to FTX. From a quick search and look at the tag, there were on the order of 10 top-level posts on the subject on LW. There are 151 posts under the FTX collapse tag on the EA forum, and possibly more untagged.
There’s definitely no censorship of the topic on LessWrong. Obviously I don’t know for sure why discussion is sparse, but my guess is that people mostly (and, in my opinion, correctly) don’t think it’s a particularly interesting or fruitful topic to discuss on LessWrong, or that the degree to which it’s an interesting subject is significantly outweighed by mindkilling effects.
Edit: with respect to the rest of the comment, I disagree that rationalists are especially interested in object-level discussion of the subjects, but probably are much more likely to disapprove of the idea that discussion of the subject should be verboten.I think the framing where Bostrom’s apology is a subject which has to be deliberately ignored is mistaken. Your prior for whether something sees active discussion on LessWrong is that it doesn’t, because most things don’t, unless there’s a specific reason you’d expect it to be of interest to the users there. I admit I haven’t seen a compelling argument for there being a teachable moment here, except the obvious “don’t do something like that in the first place”, and perhaps “have a few people read over your apology with a critical eye before posting it” (assuming that didn’t in fact happen). I’m sure you could find a way to tie those in to the practice of rationality, but it’s a bit of a stretch.
It’s possible I’ve flipped the sign on what you’re saying, but if I haven’t, I’m pretty sure most EAs are not moral realists, so I don’t know where you got the impression that it’s an underlying assumption of any serious EA efforts.
If I did flip the sign, then I don’t think it’s true that moral realism is “too unquestioned”. At this point it might be more fair to say that too much time & ink has been spilled on what’s frankly a pretty trivial question that only sees as much engagement as it does because people get caught up in arguing about definitions of words (and, of course, because some other people are deeply confused).
I think you are somewhat missing the point. The point of a treaty with an enforcement mechanism which includes bombing data centers is not to engage in implicit nuclear blackmail, which would indeed be dumb (from a game theory perspective). It is to actually stop AI training runs. You are not issuing a “threat” which you will escalate into greater and greater forms of blackmail if the first one is acceded to; the point is not to extract resources in non-cooperative ways. It is to ensure that the state of the world is one where there is no data center capable of performing AI training runs of a certain size.
The question of whether this would be correctly understood by the relevant actors is important but separate. I agree that in the world we currently live in, it doesn’t seem likely. But if you in fact lived in a world which had successfully passed a multilateral treaty like this, it seems much more possible that people in the relevant positions had updated far enough to understand that whatever was happening was at least not the typical realpolitik.
Obviously if you live in a world where you’ve passed such a treaty, the first step in response to a potential violation is not going to be “bombs away!”, and nothing Eliezer wrote suggests otherwise. But the fact that you have these options available ultimately bottoms out in the fact that your BATNA is still to bomb the data center.
I think conducting cutting edge AI capabilities research is pretty immoral, and in this counterfactual world that is a much more normalized position, even if consensus is that chances of x-risk absent a very strong plan for alignment is something like 10%. You can construct the least convenient possible world such that some poor country has decided, for perfectly innocent reasons, to build data centers that will predictably get bombed, but unless you think the probability mass on something like that happening is noticeable, I don’t think it should be a meaningful factor in your reasoning. Like, we do not let people involuntarily subject others to russian roulette, which is similar to the epistemic state of the world where 10% x-risk is a consensus position, and our response to someone actively preparing to go play roulette while declaring their intentions to do so in order to get some unrelated real benefit out of it would be to stop them.
I mean, no, in this world you’re already dead, and also nuclear exchange would in fact cost AI quite a lot so I expect many fewer nuclear wars in worlds where we’ve accidentally created an unaligned ASI.