I read your first paragraph and was like “disagree”, but when I got to the examples, I was like “well of I agree here, but that’s only because those analogies are stupid”.
At least one analogy I’d defend is the Sorcerer’s Apprentice one. (Some have argued that the underlying model has aged poorly, but I think that’s a red herring since it’s not the analogy’s fault.) I think it does share important features with the classical x-risk model.
Not my conclusion! I only linked to the post/copied and reformatted the text—the author is Ozy Brennan.
I agree, and I got permission from Ozy to include the full text, so now it’s here.
Ah, sorry! I had vaguely remembered that comment and it took me a while to find it, and I think I was so annoyed in that moment that I just assumed the context would fit without checking (which I realize makes no sense since you didn’t even write the original post). I’ll edit my comment.
If a company has 21 employees and someone says it has 7, I’m not going to call the situation ambiguous just because someone disagrees with me.
Sure, I agree with this particular case. There is a level of clarity above which you can say ambiguous without getting criticism. Mathematical statements or extremely simple factual statements are in that category. If that had been the only item in the section, I wouldn’t have an issue with it. But you’ve also picked the most objective one to make this argument. Habyrka also already admitted that he got the number of people in the company wrong, so this is not the problematic claim.
Edit: Apologies, I didn’t check the context. I endorse the general point about narrow claims, but the quote from Habryka isn’t relevant.
Speaking of unpleasant framing, I’m wholly unimpressed by the framing of me “trying to turn [your] objection into an object-level debate.” It matters whether, on the object level, things I said were unambiguous are.
I don’t get the point here. “The object-level question matters” isn’t a rebuttal to “you’re trying to make it about an object-level question”.
It’s inevitable that some people are going to dislike the way I chose to frame my reply, but I did so carefully and with an eye towards serious, charitable, direct engagement with the topic on its merits. “Epistemically toxic” is strong phrasing that I wholly reject.
I agree that it’s a strong phrasing, but I stand by it. I wouldn’t bring up style if I didn’t if I only had minor issues with it. I think the ability to differentiate between consensus-level claims and opinions is one of the main reasons why discussion on LW and EA is less bad than discussion everywhere else on the internet, and I perceive this post and the conversation as an step toward eroding that difference.
I have no doubts that you think the style was fine. I also have no doubts that you’d be able to justify the “charitable” and “direct” adjectives as you understand them. I expect the problem would again be that you’ll define them by subjective, things-people-can-disagree-about standards. By my standards, the post is not direct engagement since it begins by drawing an emotionally loaded analogy, and it’s not charitable because of the section I’ve been criticizing. Both of these are true regardless of the object-level merits (unless every object level claim were in the category of extremely straight-forward factual statements, which it isn’t).
Reasonable disagreement, I think, should be as in the legal sense for doubt: disagreement based on clear reasons, not disagreement from people who are generally reasonable.
With this definition, any and all ability to enforce norms objectively goes out the window. A follower of {insert crazy idea X} would be equally justified to talk about unambiguous delusions in doubters of X, and anyone disputing it would have to get into a debate about the merits of X rather than pointing out that plenty of people disagree with X so it doesn’t seem unambiguous.
We already have plenty of words to express personal opinions about a topic. Why would you want to define words that talk about consensus to also refer to personal opinions only? That just takes away our ability to differentiate between them. Why would we want that? Whether or not most people think something is useful information.
And there’s also a motte-and-bailey thing going on here. Because if you really only want to talk about what you personally think, then there wouldn’t be a reason to talk about unambiguous falsehoods. You’ve used the word unambiguous because it conveys a sense of objectivity, and when challenged, you defend by saying you that personally feel that way.
I’d see a lot more use to engaging with your point if instead of simply asserting that people could disagree, you explain precisely which you disagree with and why.
This is the second time you’ve tried to turn my objection into an object-level debate, and it completely misses the point. You don’t even know if I have any object-level disagreements with your post. I critiqued your style of of communication, which I believe to be epistemically toxic, not the merits of your argument.
The example there is rhetorically effective not because there is an analogy between what the New York Times does and what this post did, but because there isn’t.
I objected to the comparison because it’s emotionally loaded. “You’re worse than {bad thing}” isn’t any less emotionally loaded than “you’re like {bad thing}”.
People are still arguing about the falsehoods, but it’s unclear to me either that those arguments have any substance or that they’re germane to the core of my point.
Well yes, I would have been very surprised if you thought they had substance given your post. But the term “unambiguous” generally implies that reasonable people don’t disagree, not just that the author feels strongly about them. This is one of the many elements of your post that made me describe it as manipulative; you’re taking one side on an issue that people are still arguing about and call it unambiguous.
There are plenty of people who have strong feelings about the accusations but don’t talk like that. Nonlinear themselves didn’t talk like that! Throughout these discussions, there’s usually an understanding that we differentiate between how strongly we feel about something and whether it’s a settled matter, even from people who perceive the situation as incredibly unfair.
I find this post to be emotionally manipulative to a pretty substantial degree. It opens by explicitly asking readers to picture a scene involving the NYT, which most people will have a negative association with:
Picture a scene: the New York Times is releasing an article on Effective Altruism (EA) with an express goal to dig up every piece of negative information they can find. They contact Émile Torres, David Gerard, and Timnit Gebru, collect evidence about Sam Bankman-Fried, the OpenAI board blowup, and Pasek’s Doom, start calling Astral Codex Ten (ACX) readers to ask them about rumors they’d heard about affinity between Effective Altruists, neoreactionaries, and something called TESCREAL. They spend hundreds of hours over six months on interviews and evidence collection, paying Émile and Timnit for their time and effort. The phrase “HBD” is muttered, but it’s nobody’s birthday.
… and then proceeds to try to build an analogy between the NYT and Ben’s post.
When Nonlinear posted their reply, there was some controversy over whether the section “Sharing Information on Ben Pace” is okay or not. I defended it because, rightly or wrongly, Nonlinear had their reputation destroyed to a significant extent by Ben’s post, and I thought and still think that gives them some leeway when it comes to emotional appeals in their post. Nothing like that applies here, yet this post is written like a novel rather than an impartial analysis. (To be clear there’s much more to this than just the central analogy, but I’m not going to get into details.)
I think emotional manipulation is always bad because it’s orthogonal to truth, but it’s super extra bad when the topic under discussion is this controversial and emotionally loaded in the first place. I don’t expect many to agree with this, but I honestly think it may be because you don’t realize the extent to which you’re being manipulated because I guess this is more subtle than what Nonlinear did. I’d argue that makes it worse; Nonlinear’s section on Ben actually told the readers what it was doing. Again, don’t expect much agreement here, but I think incentivizing stuff like this is a bad idea, and it’s especially bad if we’re only incentivizing it if the author doesn’t admit what they’re doing.
The post itself also seems premature for the reasons habryka already mentioned. Both Ben’s post and Nonlinear’s reply widely swung pubic opinion; why do we expect that Ben’s reply won’t do the same? And if it does, then it seems unhelpful to make a post framing it as a settled matter. And it’s also not true that they’re unambiguous falsehoods since most of them leave plenty of room for argument, as demonstrated by people still arguing about them.
I don’t see the connection between this comment and the post.
Good reply. I’m back to feeling a lot of uncertainty about what to think.
This comment sounds very reasonable, but I think it really isn’t. Not because anything you said is false; I agree that the summary left out relevant sections, but because the standard is unreasonably high. This is a 134 page document. I expect that you could spend hours poking one legitimate hole after another into how they were arguing or paraphrasing.
Since I expect that you can do this, I don’t it makes sense to update based on you demonstrating it.
I feel the same way about what happened itself. It seems like Chloe really wanted to have a free day, but Emerson coerced her into working because it was convenient for him, that he probably wouldn’t have insisted if she had argued the point, but that she didn’t have the social courage to do so (which is super understandable, I don’t think I’d have argued in that sitaution). If so, that’s very much not cool from Emerson. It also is completely normal. I would expect that you can find anecdotes like this one from people who are more considerate than average. Not if you meet them for a day, but if you’re with them for several months.
Now if Chloe complained about this and the same thing kept happening, then we’re talking. I think that puts it into the territory of “so bad that it warrants sharing information about it publicly”. And who knows, maybe it did. I mean, here’s it’s just Kat’s word against Chloe’s. But then the problem isn’t quoting inaccurately, it’s that information contained in the doc isn’t true. If I take the doc at face value, I really don’t think the anecdote looks bad for Nonlinear, even with the full context from the quote.
This is also kind of how I feel about much of the comment section. A lot of it seems to apply the standard “did Nonlinear do something seriously wrong”. Yes, of course they did things seriously wrong. When regular people live together, everyone does things seriously wrong all the time. I think the standard we should apply instead is “did they do anything unusually wrong”, meaning unusual given that we’re picking from a several month window. And I’d say the same for this document. You shouldn’t ask “can I find serious errors with this document?” because the answer is bound to be yes, it should be “can I find really egregious errors?” This one doesn’t seem like an egregious error; it seems like one that most people would make many of in a document of this length. (I think that’s true even if they work on it for several months.)
If it is, in fact, based someone from Nonlinear, then I’d agree that the section is bad. At that point, it would no longer be a valid example of “look, you can do this to anyone”.
No, but I think she would be morally permitted to verbally insult him after that, especially if it’s the first time she gets to respond.
My point was you should make norms that ask realistic things of people. It’s not realistic to expect people to be completely emotionally detached toward someone who harmed them. But it is realistic to expect them to keep retaliation to a minimum, which again, I think is the norm that most people actually apply to situations most of the time. And yes, if you construct an example where the initial harm is extreme, then the 10% figure I postulated doesn’t work anymore.
I think the principle is something like, “if X socially harms Y, then Y is morally justified to pull analogous moves on X to make a point as long as this clearly causes only a fraction of the harm, maybe at most 10% something”. Which I recognize isn’t obvious; you could argue that X harming Y doesn’t give Y any permission to be less than maximally ethical. But that is not how most people assess things most of the time. People are generally not expected to be maximally nice to people who mistreated them. And given how humans work, I think that’s a norm that makes sense.
Kat framing the section as a negative example and explicitly telling people not to update reduces the reputational damage to a small fraction of what it would otherwise be (even though, as I said, I agree that it doesn’t remove it entirely). This looks to me like a high enough ethical standard given the context.
Yeah, I mean that would be an argument for why the section is worse than what Ben did. If you do conclude that, then I think your original comment becomes reasonable. It doesn’t strike me as obvious though, which might be the crux.
Since the anecdotes in the section are real rather than made-up, it seems nontrivial to me that you can write a section like that even if you have prior reason to dislike the person. I agree with your other comment that it’s non-crazy to do some amount of updating based on the section despite Kat saying you shouldn’t update. But I don’t agree that Kat is therefore not “morally allowed” to write it.
I strongly disagree. You logically have to either believe that the entire post of Ben was equally deranged, or that the section in this post is obviously worse than what Ben wrote, or both.
And yes, you could have used other examples to make the point. But it matters that you can do this with Ben in particular because people may have trusted the initial allegations because Ben wrote them. It seem to me to be a valid part of the argument, and one that Kat is morally justified in making.
Well, a computer model is “literally” transparent in the sense that you can see everything, which means the only difficulty is only in understanding what it means. So the part where you spend 5 million dollars on a PET scanner doesn’t exist for ANNs, and in that sense you can analyze them for “free”.
If the understanding part is sufficiently difficult… which it sure seems to be… then this doesn’t really help, but it is a coherent conceptual difference.
Not the OP, but a point I’ve made in past discussions when this argument comes up is that this is would probably actually not be all that odd without additional assumptions.
For any realist theory of consciousness, a question you could ask is, do there exist two systems that have the same external behavior, but one system is much less conscious than the other? (∃S1,S2:B(S1)=B(S2)∧C(S1)≈0≠C(S2)?)
Most theories answer “yes”. Functionalists tend to answer “yes” because lookup tables can theoretically simulate programs. Integrated Information Theory explicitly answers yes (see Fig 8, p.37 in the IIT4.0 paper). Attention Schema Theory I’m not familiar with, but I assume it has to answer yes because you could build a functionally identical system without an attention mechanism. Essentially any theory that looks inside a system rather than at input/output level only—any non-behaviorist theory—has to answer yes.
Well if the answer is yes, then a situation you describe has to be possible. You just take S1 and gradually rebuild it to S2 such that behavior also gets preserved along the way.
So it seems to me like the fact that you can alter a system such that its consciousness fades but its behavior remains unchanged is not itself all that odd, it seems like something that probably has to be possible. Where it does get odd is if also assume that S1 and S2 perform their computations in a similar fashion. One thing that the examples I’ve listed all have in common is that this additional assumption is false; replacing e.g. a human cognitive function with a lookup table would lead in dramatically different internal behavior.
Because of all this, I think the more damning question would not just be “can you replace the brain bit by bit and consciousness fades” but “can you replace the brain bit by bit such that the new components do similar things internally to the old components, and consciousness fades”?[1] If the answer to that question is yes, then a theory might have a serious problem.
Notably this is actually the thought experiment Eliezer proposed in the sequences (see start of the Socrates Dialogue).