Karl von Wendt

Karma: 322

Karl von Wendt 17 Sep 2025 17:39 UTC
1 point
0 ∶ 0
on: September ’25 EA Newsletter poll: Strong Longtermism
Far-future effects are the most important determinant of what we ought to do
“Far future” is an extremely fuzzy concept. We won’t get very far anyway if we don’t solve the near-term problems, like preventing ASI. For me, longtermism really means keeping the space of possible decisions for the next generation as large as possible, not determining the future in a way we today think is “good”. So it’s not much different from sustainability.

Karl von Wendt 6 Dec 2023 15:24 UTC
1 point
0 ∶ 1
in reply to: titotal’s comment on: The benefits and risks of optimism (about AI safety)
I agree that some “doomers” (you may count me as one) are “pessimistic”, being biased towards a negative outcome. I can’t rule out that I’m “overly cautious”. However, I’d argue that this is net positive for AI safety on the same grounds that I think optimism as I defined it is net positive under different circumstances, as described.
I agree that the word “optimism” can be used in different ways, that’s why I gave a definition of the way I usually use it. My post was a reaction to Pope and Belrose, but as I stated, not about their arguments but generally about being “optimistic” in the way I defined it. Nora Belrose said in a comment on LessWrong that my way of defining optimism is not how they meant it, and as long as I don’t analyze their texts, I have to accept that. But I think my definition of optimism fits in the range of common uses of the word (see Wikipedia, for example). All I did was trying to point out that this kind of “positive outcome bias” may be beneficial under certain circumstances, but not for thinking about AI safety.
I believe that if Pope and Belrose try to have a truly rational and unbiased stance, the term “AI Optimism” is at least misleading, as it can be understood in the way I have understood it. I hope this post is at least helpful in the sense that I have pointed that possible misunderstanding out.

Karl von Wendt 3 Dec 2023 15:21 UTC
1 point
0 ∶ 2
in reply to: titotal’s comment on: The benefits and risks of optimism (about AI safety)
I don’t think that there’s a huge difference. As long as there aren’t very strong fact-based arguments for exactly how likely it is that we will be able to control AGI, my definition of “optimists” will end up with a significantly higher probability of things going well. From what I read, I believe that Belrose and Pope have this basic bias towards “AGI is beneficial” and weigh the upside potential higher than the downside risks. They then present arguments in favor of that position. This is of course just an impression, I can’t prove it. In any case, even if they genuinely believe that everything they say is correct, they still should put in massive caveats and point out where exactly their arguments are weak or could be questioned. But that is not what a self-declared “optimist” does. So, instead they just present their beliefs. That’s okay, but it is clearly a sign of optimism the way I define it.

Karl von Wendt 1 Jul 2023 6:30 UTC
1 point
0 ∶ 0
in reply to: Robert_Wiblin’s comment on: Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?
I have thought something similar (without having read about it before), given the large percentage of people who were willing to change their minds. But I think the exact percentage of the shift, if there was one at all, isn’t really important. I think you could say that since there wasn’t a major shift towards x-risk, the debate wasn’t going very well from an x-risk perspective.
Imagine you’re telling people that the building you’re in is on fire, the alarm didn’t go off because of some technical problem, and they should leave the building immediately. If you then have a discussion and afterwards even just a small fraction of people decides to stay in the building, you have “lost” the debate.
In this case, though I was disappointed, I don’t think the outcome is “bad”, because it is an opportunity to learn. We’re just at the beginning of the “battle” about the public opinion on AI x-risk, so we should use this opportunity to fine-tune our communications. That’s why I wrote the post. There’s also this excellent piece by Steven Byrnes about the various arguments.

Karl von Wendt 28 Jun 2023 5:43 UTC
1 point
0 ∶ 0
in reply to: titotal’s comment on: Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?
I think it’s perfectly fine to (politely) call bullshit, if you think something is bullshit, as long as you follow it up with arguments as to why you think that
Agreed.
(which she did, even if you think the arguments were weak)
That’s where we disagree—strong claims (“Two Turing-award winners talk nonsense when they point out the dangerousness of the technology they developed”) require strong evidence.

Karl von Wendt 27 Jun 2023 15:19 UTC
2 points
1 ∶ 0
in reply to: titotal’s comment on: Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?
I think calling their opinions “ungrounded speculation” is an entirely valid opinion, although I would personally use the more diplomatic term “insufficiently grounded speculation”.
I disagree on that. Whether politely said or not, it disqualifies another’s views without any arguments at all. It’s like saying “you’re talking bullshit”. Now, if you do that and then follow up with “because, as I can demonstrate, facts A and B clearly contradict your claim”, then that may be okay. But she didn’t do that.
She could have said things like “I don’t understand your argument”, or “I don’t see evidence for claim X”, or “I don’t believe Y is possible, because …”. Even better would be to ask: “Can you explain to me why you think an AI could become uncontrollable within the next 20 years?”, and then answer to the arguments.

Karl von Wendt 27 Jun 2023 15:09 UTC
1 point
0 ∶ 0
in reply to: titotal’s comment on: Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?
I don’t think your “heuristic” vs “argument” distinction is sufficiently coherent to be useful. I prefer to think of it all as evidence, and talk about the strength of that evidence.
I agree in principle. However, I still think that there’s a difference between a heuristic and a logical conclusion. But not all heuristics are bad arguments. If I get an email from someone who wants to donate $10,000,000 to me, I use the heuristic that this is likely a scam without looking for further evidence. So yeah, heuristics can be very helpful. They’re just not very reliable in highly unusual situations. In German comments, I often read “Sam Altman wants to hype OpenAI by presenting it as potentially dangerous, so this open letter he signed must be hype”. That’s an example of how a heuristic can be misleading. It is ignoring the fact, for example, that Yoshua Bengio and Geoffrey Hinton also signed that letter.
You talk about Tegmark citing recent advances in AI as ” concrete evidence” that a future AGI will be world domination capable.
No. Tegmark cites this as concrete evidence that a future uncontrollable AGI is possible and that we shouldn’t carelessly dismiss this threat. He readily admits that there may be unforeseen obstacles, and so do I.
Who is right? You can’t figure that out by the semantics of “heuristics”. To get an actual answer, you have to dig into actual research on capabilities and limitations, which was not done by anybody in this debate (mainly because it would have been too technical for a public-facing debate).
I fully agree.

Karl von Wendt 27 Jun 2023 10:04 UTC
1 point
0 ∶ 0
in reply to: titotal’s comment on: Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?
I definitely disagree with the OP that Mitchell was being “dismissive” for stating her honest belief that near-term AGI is unlikely. This is a completely valid position held by a significant portion of AI researchers.
I didn’t state that. I think Mitchell was “dismissive” (even aggressively so) by calling the view of Tegmark, Bengio, and indirectly Hinton and others “ungrounded speculations”. I have no problem with someone stating that AGI is unlikely within a specific timeframe, even if I think that’s wrong.
I agree with most of what you wrote about the debate, although I don’t think that Mitchell presented any “good” arguments.
intelligent AI would be able to figure out what we wanted
It probably would, but that’s not the point of the alignment problem. The problem is that even if it knows what we “really” want, it won’t care about it unless we find a way to align it with our values, needs, and wishes, which is a very hard problem (if you doubt that, I recommend watching this introduction). We understand pretty well what chickens, pigs, and cows want, but we still treat them very badly.

Karl von Wendt 27 Jun 2023 9:50 UTC
1 point
0 ∶ 3
in reply to: titotal’s comment on: Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?
Thank you for clarifying your view!
Take the heuristic that Tegmark employed in this debate: that the damage potential of human weapons has increased over time. He talks about how we went from sticks, to guns, to bombs, killing dozens, to hundreds, to millions.
This is undeniably a heuristic, but it’s used to prime people for his later logical arguments as to why AI is also dangerous, like these earlier technologies.
This is not a heuristic. It would be a heuristic if he had argued “Because weapons have increased in power over time, we can expect that AI will be even more dangerous in the future”. But that’s not what he did if I remember it correctly (unfortunately, I don’t have access to my notes on the debate right now, I may edit this comment later). However, he may have used this example as priming, which is not the same in my opinion.
Mitchell in particular seemed to argue that AI x-risk is unlikely and talking about it is just “ungrounded speculation” because fears have been overblown in the past, which would count as a heuristic, but I don’t think LeCun used it in the same way. But I admit that telling it apart isn’t easy.
The important point here is not so much whether using historical trends or other unrelated data in arguments is good or bad, it’s more whether the argument is built mainly on these. As I see it:
Tegmark and Bengio argued that we need to take x-risk from AI seriously because we can’t rule it out. They gave concrete evidence for that, e.g. the fast development of AI capabilities in the past years. Bengio mentioned how that had surprised him, so he had updated his probability. Both admitted that they didn’t know with certainty whether AI x-risk was real, but gave it a high enough probability to be concerned. Tegmark explicitly asked for “humbleness”: Because we don’t know, we need to be cautious.
LeCun mainly argued that we don’t need to be worried because nobody would be stupid enough to build a dangerous ASI without knowing how to control it. So in principle, he admitted that there would indeed be a risk if there was a probability that someone could be stupid enough to do just that. I think he was closer to Bengio’s and Tegmark’s viewpoints on this than to Mitchell’s.
Mitchell mainly argued that we shouldn’t take AI x-risk seriously because a) it is extremely unlikely that we’ll be able to build uncontrollable AI in the foreseeable future and b) talking about x-risk is dangerous because it takes away energy from “real” problems. a) was in direct contradiction to what LeCun said. The evidence she provided for it was mainly a heuristic (“people in the 60′s thought we were close to AGI, and it turned out they were wrong, so people are wrong now”) and an anthropomorphic view (“computers aren’t even alive, they can’t make their own decisions”), which I would also count as a heuristic (“humans are the only intelligent species, computers will never be like humans, therefore computers are very unlikely to ever be more intelligent than humans”), but this may be a misrepresentation of her views. In my opinion, she gave no evidence at all to justify her claim that two of the world’s leading AI experts (Hinton and Bengio) were doing “ungrounded speculation”. b) is irrelevant to the question debated and also a very bad argument IMO.
I admit that I’m biased and my analysis may be clouded by emotions. I’m concerned about the future of my three adult sons, and I think people arguing like LeCun and, even more so, Mitchell are carelessly endangering that future. That is true for your own future as well, of course.

Karl von Wendt 26 Jun 2023 8:11 UTC
3 points
1 ∶ 2
in reply to: Joe M’s comment on: Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?
I agree that we should always be cautious when dismissing another’s arguments. I also agree that some pro-x-risk arguments may be heuristics. But I think the distinction is quite important.
A heuristic is a rule of thumb, often based on past experience. If you claim “tomorrow the sun will rise in the east because that’s what it has always done”, that’s a heuristic. If you instead say “Earth is a ball circling the sun while rotating on its axis; reversing this rotation would require enormous forces and is highly unlikely, therefore we can expect that it will look like the sun is rising in the east tomorrow”, that’s not a heuristic, but a logical argument.
Heuristics work well in situations that are more or less stable, but they can be misleading when the situation is unusual or highly volatile. We’re living in very uncommon times, therefore heuristics are not good arguments in a discussion about topics like AI safety.
The claim “people have always been afraid of technology, and it was always unfounded, so there is no need to be afraid of AI x-risk now” is a typical heuristic, and it is highly misleading (and also wrong IMO—the fear of a nuclear war is certainly not unfounded, for instance, and some of the bad consequences of technology, like climate change and environmental destruciton, are only now becoming apparent).
The problem with heuristics is that they are unreliable. They prove nothing. At the same time, they are very easy to understand, and therefore very convincing. They have a high potential of misleading people and clouding their judgements. Therefore, we should avoid and fight them wherever we can, or at least make it transparent that they are just rules of thumb based on past experience, not logical conclusions based on a model of reality.
Edit: I’d be interested to hear what the people who disagree with this think. Seriously. This may be an important discussion.

Karl von Wendt 26 Jun 2023 5:24 UTC
1 point
1 ∶ 0
in reply to: JWS 🔸’s comment on: Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?
I think the value of these debates lies in normalising this issue as one that is valid to have a debate about in the public sphere. These debates aren’t confined to LessWrong or in-the-know Twitter sniping anymore, and I think that’s unironically a good thing.
I agree.
I think you’re a bit too hasty to extrapolate what the ‘AI Safety Side’ strategy should be.
That may well be true.
Personally think that Stuart Russell would be a great spokesman for the AI-risk-is-serious side, impeccable credentials, has debated the issue before (see here vs Melanie), and I think his persepective on AI risk lends itself to a “slow-takeoff” framing rather than a “hard-takeoff” framing which Bengio/Hinton/Tegmark etc. seem to be pushing more.
Yes, he definitely would have been a good choice.

Karl von Wendt 26 Jun 2023 5:20 UTC
2 points
1 ∶ 0
in reply to: Wim’s comment on: Did Bengio and Tegmark lose a debate about AI x-risk against LeCun and Mitchell?
Thank you for this insight! I agree, as I wrote, that the result is inconclusive. I still think the debate could have had much “better” results in the sense that it could have made people more concerned about AI x-risks. I think that Bengio and Tegmark failed at this at least in part, though I’m not sure whether my explanation for this is correct.

Karl von Wendt 22 Jun 2023 12:41 UTC
10 points
3 ∶ 2
in reply to: Habryka [Deactivated]’s comment on: My tentative best guess on how EAs and Rationalists sometimes turn crazy
I think a major contributing factor in almost all extremely unhealthy group dynamics is narcissism. Pathological narcissists have a high incentive to use the dynamics you describe to their advantage and are (as I have heard and have some anecdotical evidence, but can’t currently back up with facts) often found in leading positions in charities, religious cults, and all kinds of organizations. Since they hunger for being admired and are very good at manipulating people, they are attracted to charities and altruistic movements. Of course, a pathological narcissist is selfish by definition and completely unable to feel anything like compassion, so it’s the antithesis of an altruist.
The problem is made worse because there are two types of narcissists: “grandiose” types who are easily recognizable, like Trump, and “covert” ones who are almost impossible to detect because they are often extremely skilled at gaslighting. My mother-in-law is of the latter type; it took me more than 40 years to figure that out and even realize that my wife had a traumatic childhood (she didn’t realize this herself all that time, she just thought something was wrong with her). About 1-6% of adults are assumed to be narcisissts, with the real dangerous types probably making up only a small fraction of that. Still, it means that in any group of people there’s on average one narcissist for every 100 members, and I have personally met at least one of them in the EA community. They often end up in leading positions, which of course doesn’t mean that all leaders are narcissists. But if something goes badly wrong with an altruistic movement, narcissism is one of the possible reasons I’d look into first.
It’s difficult to tell from a distance, but for me, it looks like SBF may be a pathological narcissist as well. It would at least explain why he was so “generous” and supported EA so much with the money he embezzled.

Karl von Wendt 21 Jun 2023 12:14 UTC
3 points
0 ∶ 0
in reply to: kokotajlod’s comment on: A Friendly Face (Another Failure Story)
As replied on LW, we’ll discuss this. Thanks!

Karl von Wendt 21 Jun 2023 12:13 UTC
1 point
0 ∶ 0
in reply to: Stephen McAleese’s comment on: A Friendly Face (Another Failure Story)
Thank you! I hope this story will have a similar effect outside of the EA community.

Karl von Wendt 21 Jun 2023 12:12 UTC
1 point
0 ∶ 0
in reply to: Sam Robinson 🔸’s comment on: A Friendly Face (Another Failure Story)
Thanks! I should point out that this isn’t my work alone, we were a team of six developing the story. Unfortunately, the co-authors Artem, Daniel, Ishan, Peter, and Sofia are not visible in this cross-post from LessWrong.

Karl von Wendt 21 Jun 2023 12:10 UTC
3 points
1 ∶ 0
in reply to: Geoffrey Miller’s comment on: A Friendly Face (Another Failure Story)
Thank you! Yes, stories about how the movies “War Games” and “The Day After” changed Ronald Reagan’s mind about cyberattacks and the risk of a global nuclear war were in part inspiration for this AI Safety Camp project. Stories can indeed be powerful.

Karl von Wendt 8 Jun 2023 8:40 UTC
5 points
1 ∶ 0
in reply to: Geoffrey Miller’s comment on: We don’t need AGI for an amazing future
Thank you very much! The cell comparison is very interesting.

Karl von Wendt 7 Jun 2023 9:51 UTC
4 points
2 ∶ 0
in reply to: rime’s comment on: Agentic Mess (A Failure Story)
OK, I updated: risk is less straightforward than I thought. While the AIs do call copies of themselves, rLLMs can’t really undergo a runaway replication cascade unless they can call themselves as “daemons” in separate threads (so that the control loop doesn’t have to wait for the output before continuing). And I currently don’t see an obvious profit motive to do so.
I’m not sure if I understand your point correctly. The LLMs wouldn’t have to be replicated, because different copies of the self-replicating agent could access the same LLM in parallel, just like many human users can access ChatGPT at the same time. At the later stage, when the LLM operators try to block access to their LLMs or even take them offline, the agent would have to find a way to replicate at least one LLM just once and run it on sufficiently powerful hardware to use it as (part of) its “brain”.

Karl von Wendt 15 May 2023 15:59 UTC
1 point
0 ∶ 0
on: EA Hamburg Meetup
I’ll bring a simple board game about AI safety that I’ve developed recently in case anyone wants to do an initial test (not the one that was tested on the AISER, which was way too complex and slow ;).