Lukas_Gloor

Karma: 6,773

Lukas_Gloor Apr 14, 2025, 9:06 AM
4 points
0 ∶ 0
in reply to: Jason’s comment on: Anthropic is not being consistently candid about their connection to EA
(I know I’m late again replying to this thread.)
What surprises me about this whole situation is that people seem surprised at the executive leadership at a corporation worth an estimated $61.5B would engage in big-corporation PR-speak. The base rate for big-corporation execs engaging in such conduct in their official capacities seems awfully close to 100%.
Hm, good point. This gives me pause, but I’m not sure what direction to update in. Like, maybe I should update “corporate speak is just what these large orgs do and it’s more like a fashion thing than a signal of their (lack of) integrity on things that matter most.” Or maybe I should update in the direction you suggest, namely “if an org grows too much, it’s unlikely to stay aligned with its founding character principles.”
I’m getting the sense that a decent number of people assume that being “EA aligned” is somehow a strong inoculant against the temptations of money and power.
I would have certainly thought so. If anything can be an inoculant against those temptations, surely a strong adherence to a cause greater than oneself packaged in lots warnings against biases and other ways humans can go wrong (as is the common message in EA and rationalist circles) seems like the best hope for it? If you don’t think it can be a strong inoculant, that makes you pretty cynical, no? (I think cynicism is often right, so this isn’t automatically a rejection of your position. I just want to flag that yours is a claim with quite strong implications on its own.)
Arguably the FTX scandal—which after all involved multiple EAs, not just SBF—should have already caused people to update on how effective said inoculant is, at least when billions of dollars were floating around.
If you were just talking about SBF, then I’d say your point is weak because he probably wasn’t low on dark triad traits to start out with. But you emphasizing how other EAs around him were also involved (the direct co-conspirators at Alameda and FTX) is a strong point.
Still, in my mind this would probably have gone very differently with the same group of people minus SBF and with a leader with a stronger commitment and psychological disposition towards honesty. (I should flag that parts of Caroline Ellison’s blog also gave me vibes of “seems to like having power too much”—but at least it’s more common for young people to later change/grow.) That’s why I don’t consider it a huge update for “power corrupts”. To me, it’s a reinforcement of “it matters to have good leadership.”
My worldview(?) is that “power corrupts” doesn’t apply equally to every leader and that we’d be admitting defeat straight away if we stopped trying to do ambitious things. There doesn’t seem to be a great way to do targeted ambitious things without some individual acquiring high amounts of power in the process.(?) We urgently need to do a better job at preventing that those who end up with a lot of power are almost always those with kind of shady character. The fact that we’re so bad at this suggests that these people are advantaged at some aspects of ambitious leadership, which makes the whole thing a lot harder. But that doesn’t mean it’s impossible.
I concede that there’s a sense in which this worldview of mine is not grounded in empiricism—I haven’t even looked into the matter from that perspective. Instead, it’s more like a commitment to a wager: “If this doesn’t work, what else are we supposed to do?”
I’m not interested in concluding that the best we can do is criticise the powers that be from the sidelines.
Of course, if leaders exhibit signs of low integrity, like in this example of Anthropic’s communications, it’s important not to let this slide. The thing I want to push back against is an attitude of “person x or org y has acquired so much power, surely that means that they’re now corrupted,” and this leading to no longer giving them the benefit of the doubt/not trying to see the complexities of their situation when they do something that looks surprising/disappointing/suboptimal. With great power comes great responsiblity, including a responsibility to not mess up your potential for doing even more good later on. Naturally, this does come with lots of tradeoffs and it’s not always easy to infer from publicly visible actions and statements whether an org is still culturally on track. (That said, I concede that you can often tell quite a lot about someone’s character/an org’s culture based on how/whether they communicate nuances, which is sadly why I’ve had some repeated negative updates about Anthropic lately.)

Lukas_Gloor Apr 3, 2025, 12:36 PM
20 points
4 ∶ 0
in reply to: Lorenzo Buonanno🔸’s comment on: Anthropic is not being consistently candid about their connection to EA
I think the people in the article you quote are being honest about not identifying with the EA social community, and the EA community on X is being weird about this.
I never interpreted that to be the crux/problem here. (I know I’m late replying to this.)

People can change what they identify as. For me, what looks shady in their responses is the clusmy attempts at downplaying their past association with EA.
I don’t care about it because I still identify with EA; instead, I care because it goes under “not being consistently candid.” (I quite like that expression despite its unfortunate history). I’d be equally annoyed if they downplayed some significant other thing unrelated to EA.
Sure, you might say it’s fine not being consistently candid with journalists. They may quote you out of context. Pretty common advice for talking to journalists is to keep your statements as short and general as possible, esp. when they ask you things that aren’t “on message.” Probably they were just trying to avoid actually-unfair bad press here? Still, it’s clumsy and ineffective. It backfired. Being candid would probably have been better here even from the perspective of preventing journalists from spinning this against them. Also, they could just decide not to talk to untrusted journalists?
More generally, I feel like we really need leaders who can build trust and talk openly about difficult tradeoffs and realities.

Lukas_Gloor Apr 1, 2025, 12:01 AM
25 points
8 ∶ 1
in reply to: NickLaing’s comment on: Anthropic is not being consistently candid about their connection to EA
I agree that these statements are not defensible. I’m sad to see it. There’s maybe some hope that the person making these statements was just caught off guard and it’s not a common pattern at Antrhopic to obfuscate things with that sort of misdirection. (Edit: Or maybe the journalist was fishing for quotes and made it seem like they were being more evasive than they actually were.)
I don’t get why they can’t just admit that Anthropic’s history is pretty intertwined with EA history. They could still distance themselves from “EA as the general public perceives it” or even “EA-as-it-is-now.”
For instance, they could flag that EA maybe has a bit of a problem with “purism”—like, some vocal EAs in this comment section and elsewhere seem to think it is super obvious that Anthropic has been selling out/became too much of a typical for-profit corporation. I didn’t myself think that this was necessarily the case because I see a lot of valid tradeoffs that Anthropic leadership is having to navigate, and the armchair quarterbacks EAs seem to be failing to take that into account? However, the communications highlighted in the OP made me update that Anthropic leadership probably does lack the integrity needed to do complicated power-seeking stuff that has the potential to corrupt. (If someone can handle the temptions from power, they should at the very least be able to handle the comparatively easy dynamics of don’t willingly distort the truth as you know it.)

Lukas_Gloor Mar 17, 2025, 6:35 PM
6 points
0 ∶ 0
in reply to: William_MacAskill’s comment on: Existential Choices Symposium with Will MacAskill and other special guests (3-5pm GMT Monday)
As you say you can block the obligation to gamble and risk Common-sense Eutopia for something better in different ways/for different reasons.

For me, Common-sense Eutopia sounds pretty appealing because it ensures continuity for existing people. Considering many people don’t have particularly resource-hungry life goals, Common-sense Eutopia would score pretty high on a perspective where it matters what existing people want for the future of themselves and their loved ones.

Even if we say that other considerations besides existing people also matter morally, we may not want those other considerations to just totally swamp/outweigh how good Common-sense Eutopia is from the perspective of existing people.

Lukas_Gloor Mar 17, 2025, 5:31 PM
4 points
0 ∶ 0
in reply to: Andreas_Mogensen’s comment on: Existential Choices Symposium with Will MacAskill and other special guests (3-5pm GMT Monday)
Now, if you accept utilitarianism for a fixed population, you should think that D is better than C

If we imagine that world C already exists, then yeah, we should try to change C into D.(Similarly, if world D already exists, we’d want to prevent changes from D to C.)
So, if either of the two worlds already exists, D>C.
Where the way you’re setting up this argument turns controversial, though, is when you suggest that “D>C” is valid in some absolute sense, as opposed to just being valid (in virtue of how it better fulfills the preferences of existing people) under the stipulation of starting out in one of the worlds (that already contains all the relevant people).
Let’s think about the case where no one exists so far, where we’re the population planners for a new planet that can either shape into C or D. (In that scenario, there’s no relevant difference between B and C, btw.) I’d argue that both options are now equally defensible because the interests of possible people are under-defined* and there are defensible personal stances on population ethics for justifying either.**
*The interests of possible people are underdefined not just because it’s open how many people we might create. In addiiton, it’s also open who we might create: Some human psychological profiles are such that when someone’s born into a happy/priviledged life, they adopt a Buddhist stance towards existence and think of themselves as not having benefitted from being born. Other psychological profiles are such that people do think of themselves as grateful and lucky for having been born. (In fact, others yet even claim that they’d consider themselves lucky/grateful even if their lives consisted of nothing but torture). These varying intuitions towards existence can inspire people’s population-ethical leanings. But there’s no fact of the matter of “which intuitions are more true.” These are just difference interpretations for the same sets of facts. There’s no uniquely correct way to approach population ethics.

**Namely, C is better on anti-natalist harm reduction grounds (at least depending on how we interpret the scale/negative numbers on the scale), whereas D is better on totalist grounds.

All of that was assuming that C or D are the only options. If we add a third alternative, say, “create no one,” the ranking between C and D (previously they were equally defensible) can change.
At this point, the moral realist proponents of an objective “theory of the good” might shriek in agony and think I have gone mad. But hear me out. It’s not crazy at all to think that choices depend on the alternatives we have available. If we also get the option, “create no one,” then I’d say C becomes worse than the two other options because there’s no approach to population ethics according to which C is optimal from the three options. My person-affecting stance on population ethics says that we’re free to do a bunch of things, but the one thing we cannot do is do things that reflect a negligent disregard for the interests of potential people/beings.
Why? Essentially for similar reasons why common-sense morality says that struggling lower-class families are permitted to have children that they raise under harship with little means (assuming their lives are still worth living in expectation), but if a millionaire were to do the same to their child, he’d be an asshole. The fact that the millionaire has the option “give my child enough resources to have a high chance at happiness” makes it worse if he then proceeds to give his child hardly any resources at all. Bringing people into existence makes you responsible for them. If you have the option to make your children really well off, but you decide not to do that, you’re not taking into consideration the interests of your child, which is bad. (Of course, if the millionaire donates all their money to effective causes and then raises a child in relative poverty, that’s acceptable again.)

I think where the proponents of an objective theory of the good go wrong is the idea that you keep track, on the same objective scoreboard, no matter whether it concerns existing people or potential people. But those are not commensurate perspectives. This whole idea of an “objective axiology/theory of the good” is dubious to me. It also has pretty counterintuitive implications to try to squeeze these perspectives under one umbrella. As I wrote elsewhere:
There’s a tension between the beliefs “there’s an objective axiology” and “people are free to choose their life goals.”
Many effective altruists hesitate to say, “One of you must be wrong!” when one person cares greatly about living forever while the other doesn’t. By contrast, when two people disagree on population ethics “One of you must be wrong!” seems to be the standard (implicit) opinion. I think these two attitudes are in tension. To the degree people are confident that life goals are up to the individual to decide/pursue, I suggest they lean in on this belief. I expect that resolving the tension in that way – leaning in on the belief “people are free to choose their life goals;” giving up on “there’s an axiology that applies to everyone” – makes my framework more intuitive and gives a better sense of what the framework is for, what it’s trying to accomplish.
Here’s a framework for doing population ethics without an objective axiology. In this framework, person-affecting views seem quite intuitive because we can motivate them as follows:
- “Preference utilitarianism for existing (and sure-to-exist) people/beings, but subject to also giving some consideration—in ways that aren’t highly demanding—to the (underdefined) interests of possible people/beings.
- That’s a common approach in situations where there are two possible things to value and care about, but someone primilarly chooses one of them, as opposed to crafting a theory that unifies both of these things and stipulates tradeoffs for all situations. For instance, on the topic of “do you care about yourself or everyone?” a self-oriented individual will choose to mostly benefit themselves, but they might feel like they should take low-demanding effective altruist actions.
- So, when figuring out how to do “systematized altruism,” someone can decide that for their interpretation of the concept (note that there is no uniquely correct answer here!) “systematized altruism” wants to incorporate the slogan Michael mentioned earlier, “make people happy, not make happy people.” So, the person focuses on doing what existing people want. However, when it comes to possible people/beings, instead of going “anything goes,” they still want to follow low-effort ways of doing good according to the interests of possible people/beings, at least in the sense of “don’t take actions that violate what possible people/beings could agree on as an interest group.” (And it’s not the case that they’d all want to be born, because like I said, there are psychological profiles of possible people/beings that prefer no chance of being born if there’s a chance of suffering. But maybe you could argue that if you have the chance to create happy people at no cost to yourself and no risk of suffering, it would be bad not to take that chance, since at least some possible people/beings would stake their weight into that.)

Lukas_Gloor Mar 2, 2025, 10:01 PM
2 points
0 ∶ 0
in reply to: Nunik’s comment on: An experiment for those who think the intrinsic value of pleasure is beyond doubt
When I said earlier that some people form non-hedonistic life goals, I didn’t mean that they commit to the claim that there are things that everyone else should value. I meant that there are non-hedonistic things that the person in question values personally/subjectively.
You might say that subjective (dis)value is trumped by objective (dis)value—then we’d get into the discussion of whether objective (dis)value is a meaningful concept. I argue against that in my above-linked post on hedonist axiology. Here’s a shorter attempt at making some of the key points from that post:
Earlier, when I agreed with you that we can, in a sense, view “suffering is bad” as moral fact, I would still maintain that this way of speaking makes sense only as a shorthand pointing towards the universality and uncontroversialness of “suffering is bad,” rather than it pointing to some kind objectivity-that-through-its-nature-trumps-everything-else that suffering is supposed to have (again, I don’t believe in that sort of objectivity). By definition, when there’s suffering, there’s a felt sense (by the sufferer) of wanting the experience to end or change, so there’s dissatisfaction and a will towards change. The definition of suffering means it’s a motivational force. But whether it is the only impetus/motivational force that matters to someone, or whether there are other pulls and pushes that they deem equally worthy (or even more worthy, in many cases), depends on the person. So, that’s where your question about the non-hedonistic life goals comes in.
But why do they say so? Because they have a feeling that something or other has value?
People choosing life goals is a personal thing, more existentialism than morality. I wouldn’t even use the word “value” here. People adopt life goals that motivate them to get up in the morning and go beyond the path of least resistance (avoiding short-term suffering). If I had tto sum it up in one word, I’d say it’s about meaning rather than value. See my post on life goals, which also discusses my theory of why/how people adopt them.
If you feel that we’re talking past each other, it’s likely because we’re thinking in different conceptual frameworks.

Let’s take a step back. I see morality as having two separate parts:
- Morality as systematized altruism: the attempt to figure out the most altruistic life goal. (Not everyone cares about this, but even people who don’t themselves want to do altruistic things could still reason about it as an intellectual exercise.)
  - Most likely, there’s no single correct “most altruistic life goal.” Even so, some answers to “What’s systematized altruism?” are clearly worse than others! So, even if there’s no single correct morality, we can still make progress in finding the most defensible options. And among the mulitple good options, it becomes a question of whether we are, for one reason or another, drawn to one of them, or whether we are indifferent between them—see my long post on how moral uncertainty within moral anti-realism is sometimes better thought of as moral indecisiveness (and on other, related topics).
- Morality as pondering obligations from the fact that other people may not share my life goals: contractualism, cooperation, respecting others’ autonomy; etc. All of that seems really important, so much so that answers to the bullet point above (“most altruistic life goal”) that would prompt us to completely thwart other people’s life goals don’t seem to be good answers.
Separately, there are non-moral life goals (and it’s possible for people to have no life goals, if there’s nothing that makes them go beyond the path of least resistance). Personally, I have a non-moral life goal (being a good husband to my wife) and a moral one (reducing suffering subject to low-effort cooperation with other people’s life goals).
That’s pretty much it. As I say in my post on life goals, I subscribe to the Wittgensteinian view of philosophy (summarized in the Stanford Encyclopedia of Philosophy):
[...] that philosophers do not—or should not—supply a theory, neither do they provide explanations. “Philosophy just puts everything before us, and neither explains nor deduces anything. Since everything lies open to view there is nothing to explain (PI 126).”
Per this perspective, I see the aim of moral philosophy as to accurately and usefully describe our option space – the different questions worth asking and how we can reason about them.
I feel like my framework lays out the option space and lets us reason about (the different parts of) morality in a satisfying way, so that we don’t also need the elusive concept of “objective value”. I wouldn’t understand how that concept works and I don’t see where it would fit in. On the contrary, I think thinking in terms of that concept loses us clarity.
Some people might claim that they can’t imagine doing without it or would consider everything meaningless if they had to do without it (see “Why realists and anti-realists disagree”). I argued against that here, here and here. (In those posts, I directly discuss the concept of “irreducible normativity” instead of “objective value,” but those are very closely linked, such that objections against one also apply against the other, mostly.)

Lukas_Gloor Mar 2, 2025, 5:18 PM
2 points
0 ∶ 0
in reply to: Nunik’s comment on: An experiment for those who think the intrinsic value of pleasure is beyond doubt
Depends what you mean by “moral realism.”
I consider myself a moral anti-realist, but I would flag that my anti-realism is not the same as saying “anything goes.” Maybe the best way to describe my anti-realism to a person who thinks about morality in a realist way is something like this:
“Okay, if you want to talk that way, we can say there is a moral reality, in a sense. But it’s not a very far-reaching one, at least as far as the widely-compelling features of the reality are concerned. Aside from a small number of uncontroversial moral statements like ‘all else equal, more suffering is worse than less suffering,’ much of morality is under-defined. That means that several positions on morality are equally defensible. That’s why I personally call it anti-realism: because there’s not one single correct answer.”
See section 2 of my post here for more thoughts on that way of defining moral realism. And here’s Luke Muehlhauser saying a similar thing.

Lukas_Gloor Mar 2, 2025, 2:20 PM
3 points
0 ∶ 0
on: An experiment for those who think the intrinsic value of pleasure is beyond doubt
I agree that hedonically “neutral” experiences often seem perfectly fine.
I suspect that there’s a sleight of hand going on where moral realist proponents of hedonist axiology try to imply that “pleasure has intrinsic value” is the same claim as “pleasure is good.” But the only sense in which “pleasure is good” is obviously uncontroversial is merely the sense of “pleasure is unobjectionable.” Admittedly, pleasure also often is something we desire, or something we come to desire if we keep experiencing it—but this clearly isn’t always the case for all people, as any personal hedonist would notice if they stopped falling into the typical mind fallacy and took seriously that many other people sincerely and philosophically-unconfusedly adopt non-hedonistic life goals.
See also this short form or this longer post.

Lukas_Gloor Feb 18, 2025, 10:29 PM
6 points
1 ∶ 2
in reply to: Tim Hua’s comment on: Teaching AI to reason: this year’s most important story
I agree it is somewhat misleading, but I feel like using the internet is itself a highly useful skill in the modern world and insofar as the other models couldn’t do it, that is too bad for them.

Lukas_Gloor Feb 13, 2025, 2:22 PM
15 points
2 ∶ 0
in reply to: Matthew_Barnett’s comment on: Matthew_Barnett’s Shortform
I haven’t read your other recent comments on this, but here’s a question on the topic of pausing AI progress. (The point I’m making is similar to what Brad West already commented.)
Let’s say we grant your assumptions (that AIs will have values that matter the same as or more than human values and that an AI-filled future would be just as or more morally important than one with humans in control). Wouldn’t it still make sense to pause AI progress at this important junction to make sure we study what we’re doing so we can set up future AIs to do as well as (reasonably) possible?
You say that we shouldn’t be confident that AI values will be worse than human values. We can put a pin in that. But values are just one feature here. We should also think about agent psychologies and character traits and infrastructure beneficial for forming peaceful coalitions. On those dimensions, some traits or setups seem (somewhat robustly?) worse than others?

We’re growing an alien species that might take over from humans. Even if you think that’s possibly okay or good, wouldn’t you agree that we can envision factors about how AIs are built/trained and about what sort of world they are placed in that affect whether the future will likely be a lot better or a lot worse?
I’m thinking about things like:
- pro-social insctincts (or at least absence of anti-social ones)
- more general agent character traits that do well/poorly at forming peaceful coalitions
- agent infrastructure to help with coordinating (e.g., having better lie detectors, having a reliable information environment or starting out under the chaos of information warfare, etc.)
- initial strategic setup (being born into AI-vs-AI competition vs. being born in a situation where the first TAI can take to proceed slowly and deliberately)
- maybe: decision-theoretic setup to do well in acausal interactions with other parts of the multiverse (or at least not do particularly poorly)
If (some of) these things are really important, wouldn’t it make sense to pause and study this stuff until we know whether some of these traits are tractable to influence?
(And, if we do that, we might as well try to make AIs have the inclination to be nice to humans, because humans already exist, so anything that kills humans who don’t want to die frustrates already-existing life goals, which seems worse than frustrating the goals of merely possible beings.)
I know you don’t talk about pausing in your above comment—but I think I vaguely remember you being skeptical of it in other comments. Maybe that was for different reasons, or maybe you just wanted to voice disagreement with the types of arguments people typically give in favor of pausing?
FWIW, I totally agree with the position that we should respect the goals of AIs (assuming they’re not just roleplayed stated goals but deeply held ones—of course, this distinction shouldn’t be uncharitably weaponized against AIs ever being considered to have meaningful goals). I’m just concerned because whether the AIs respect ours in turn, especially when they find themselves in a position where they could easily crush us, will probably depend on how we build them.

Lukas_Gloor Feb 5, 2025, 12:51 PM
5 points
0 ∶ 0
on: Issues in (Meta) Normative Ethics
Cool post!
From the structure of your writing (moslty the high number of subtitles), I often wasn’t sure where you’re endorsing a specific approach versus just laying out what the options are and what people could do. (That’s probably fine because I anyway see the point of good philosophy as “clearly laying out the option space.”)
In any case, I think you hit on the things I also find relevant. E.g., even as a self-identifying moral anti-realist, I place a great deal of importance on “aim for simplicity (if possible/sensible)” in practice.
Some thoughts where I either disagree or have something important to add:
- Another objection to 4. moral ambiguity, next to what you already listed under 4. i, is that sometimes the extension of an intuitive principle is itself ambiguous. For instance, consider the intuitive principle, “what we want to do to others is what is in their interests.” How do we extend that principle to situations where the number of others isn’t fixed? We now face multiple levels of underdefinedness (no wonder, then, that population ethics is considered difficult or controversial):
  (1) It’s under-defined how many new people with interests/goals there will be.
  (2) It’s under-defined which interests/goals a new person will have.
  (See here for an exploration of what this could imply.)
- I endorse what you call “embracing ‘biases’” in some circumstances, but I would phrase that in a more appealing way. :) I know you put “biases” in quotation marks, but it still sounds a bit unappealing that way. The way I would put it:
  Morality is inherently underdefined (see, e.g., my previous bullet point), so we are faced with the option to either embrace that underdefinedness or go with a particular axiology not because it’s objectively justified, but sbecause we happen to care deeply about it. Instead of “embracing ‘biases,’” I’d call the latter “filling out moral underdefinedness by embracing strongly held intuitions.” (See also the subsection Anticipating objections (dialogue) in my post on moral uncertainty from an anti-realist perspective.)
- What you describe as the particularist solution to moral uncertainty, is that really any different from the following:
  Imagine you have a “moral parliament” in your head filled with advocates for moral views and intuitions that you find overall very appealing and didn’t want to distill down any further. (Those advocates might be represented at different weights.) Whenever a tough decision comes up, you mentally simulate bargaining among those advocates where the ones who have a strong opininion on the matter in question will speak up the loudest and throw in a higher portion of their total bargaining allowance.
  This approach will tend to give you the same answer as the particularist one in practice(?), but it seems maybe a bit more principled in the way it’s described?
  Also, I want to flag that this isn’t just an approach to moral uncertainty—you can also view it as a full-blown normative theory in the light of undecidedness between theories.
- “If we think moral realism is true, we’d expect the best theories of morality to be simple as simplicity is an epistemic virtue.” This is just a tangential point, but I’ve seen other people use this sort of reasoning as an argument for hedonist utilitarianism (because that view is particularly simple). I just want to flag that this line of argument doesn’t work because confident belief in moral realism and moral uncertainty don’t go together. In other words, the only worlds in which you’re justified to be a confident moral realist are worlds where you already know the complete moral reality. Basically, if you’re morally uncertain, you’re by necessity also metaethically uncertain, which means that you can’t just bet on pure simplicity with all your weight (to the point that you would bite large bullets that you otherwise—under anti-realism—wouldn’t bite). (Also, if someone wanted to bet on pure simplicity, I’d wager that tranquilism is simpler than hedonism—but again, I don’t think we should take aim-for-simplicity reasoning quite that far.)

Lukas_Gloor Dec 14, 2024, 3:17 PM
2 points
0 ∶ 0
in reply to: David T’s comment on: Probabilities might be off by one percentage point
Thanks for the reply, and sorry for the wall of text I’m posting now (no need to reply further, this is probably too much text for this sort of discussion)...
I agree that uncertainty is in someone’s mind rather than out there in the world. Still, granting the accuracy of probability estimates feels no different from granting the accuracy of factual assumptions. Say I was interested in eliciting people’s welfare tradeoffs between chicken sentience and cow sentience in the context of eating meat (how that translates into suffering caused per calorie of meat). Even if we lived in a world where false-labelling of meat was super common (such that, say, when you buy things labelled as ‘cow’, you might half the time get tuna, and when you buy chicken, you might half the time get ostrich), if I’m asking specifically for people’s estimates on the moral disvalue from chicken calories vs cow calories, it would be strange if survey respondees factored in information about tunas and ostriches. Surely, if I was also interested in how people thought about calories from tunas and ostriches, I’d be asking about those animals too!
Also, circumstances about the labelling of meat products can change over time, so that previously elicited estimates on “chicken/cow-labelled things” would now be off. Survey results will be more timeless if we don’t contaminate straightforward thought experiments with confounding empirical considerations that weren’t part of the question.
A respondee might mention Kant and how all our knowledge about the world is indirect, how there’s trust involved in taking assumptions for granted. That’s accurate, but let’s just take them for granted anyway and move on?
On whether “1.5%” is too precise of an estimate for contexts where we don’t have extensive data: If we grant that thought experiments can be arbitrarily outlandish, then it doesn’t really matter.
Still, I could imagine that you’d change your mind about never using these estimates if you thought more about situations where they might become relevant. For instance, I used estimates in that area (roughly around 1.5% chance of something happening) several times within the last two years:
My wife developed lupus a few years ago, which is the illness that often makes it onto the whiteboard in the show Dr House because it can throw up symptoms that mimic tons of other diseases, sometimes serious ones. We had a bunch of health scares where we were thinking “this is most likely just some weird lupus-related symptom that isn’t actually dangerous, but it also resembles that other thing (which is also a common secondary complication from lupus or its medications), which would be a true emergency. In these situations, should we go to the ER for a check-up or not? With a 4-5h average A&E waiting time and the chance to catch viral illnesses while there (which are extra bad when you already have lupus), it probably doesn’t make sense to go in if we think the chance of a true emergency is only <0.5%. However, at 2% or higher, we’d for sure want to go in. (In between those two, we’d probably continue to feel stressed and undecided, and maybe go in primarily for peace of mind, lol). Narrowing things down from “most likely it’s nothing, but some small chance that it’s bad!” to either “I’m confident this is <0.5%” or “I’m confident this is at least 2%” is not easy, but it worked in some instances. This suggests some usefulness (as a matter of practical necessity of making medical decisions in a context of long A&E waiting times) to making decisions based on a fairly narrowed down low-probability estimate. Sure, the process I described is still a bit more fuzzy than just pulling a 1.5% point estimate from somewhere, but I feel like it approaches similar levels of precision needed to narrow things down that much, and I think many other people would have similar decision thresholds in a situation like ours.
Admittedly, medical contexts are better studied than charity contexts, and especially influencing-the-distant-future charity contexts. So, it makes sense if you’re especially skeptical of that level of precision in charitable contexts. (And I indeed agree with this; I’m not defending that level of precision in practice for EA charities!) Still, like habryka pointed out in another comment, I don’t think there’s a red line were fundamental changes happen as probabilities get lower and lower. The world isn’t inherently frequentist, but we can often find plausibly-relevant base rates. Admittedly, there’s always some subjectivity, some art, in choosing relevant base rates, assessing additional risk factors, making judgment calls about “how much is this symptom a match?.” But if you find the right context for it (meaning: a context where you’re justifiably anchoring to some very low-probability base rate), you can get well below the 0.5% level for practically-relevant decisions (and maybe make proportional upwards or downwards adjustments from there). For these reasons, it doesn’t strike me as totally outlandish that some group will at some point come up with ranged very-low-probability estimate of averting some risk (like asteroid risk or whatever), while being well-calibrated. I’m not saying I have a concrete example in mind, but I wouldn’t rule it out.

Lukas_Gloor Dec 13, 2024, 10:20 AM
2 points
0 ∶ 0
in reply to: Will Howard🔹’s comment on: Probabilities might be off by one percentage point
That makes sense; I understand that concern.

I wonder if, next time, the survey makers could write something to reassure us that they’re not going to be using any results out of context or with an unwarranted spin (esp. in cases like the one here, where the question is related to a big ‘divide’ within EA, but worded as an abstract thought experiment.)

Lukas_Gloor Dec 13, 2024, 1:45 AM
2 points
0 ∶ 0
in reply to: David T’s comment on: Probabilities might be off by one percentage point
If we’re considering realistic scenarios instead of staying with the spirit of the thought experiment (which I think we should not, partly precisely because it introduces lots of possible ambiguities in how people interpret the question, and partly because this probably isn’t what the surveyors intended, given the way EA culture has handled thought experiments thus far – see for instance the links in Lizka’s answer, or the way EA draws heavily from analytic philosophy, where straightforwardly engaging with unrealistic thought experiments is a standard component of the toolkit), then I agree that an advertized 1.5% chance of having a huge impact could be more likely upwards-biased than the other way around. (But it depends on who’s doing the estimate – some people are actually well-calibrated or prone to be extra modest.)
[...] is very much in keeping with the spirit of the question (which presumably is about gauging attitudes towards uncertainty, not testing basic EV calculation skills
(1) what you described seems to me best characterized as being about trust. Trust in other’s risk estimates. That would be separate from attitudes about uncertainty (and if that’s what the surveyors wanted to elicit, they’d probably have asked the question very differently).
(Or maybe what you’re thinking about could be someone having radical doubts about the entire epistemology behind “low probabilities”? I’m picturing a position that goes something like, “it’s philosophically impossible to reason sanely about low probabilities; besides, when we make mistakes, we’ll almost always overestimate rather than underestimate our ability to have effects on the world.” Maybe that’s what you think people are thinking – but as an absolute, this would seem weirdly detailed and radical to me, and I feel like there’s a prudential wager against believing that our reasoning is doomed from the start in a way that would prohibit everyone from pursuing ambitious plans.)
(2) What I meant wasn’t about basic EV calculation skills (obviously) – I didn’t mean to suggest that just because the EV of the low-probability intervention is greater than the EV of the certain intervention, it’s a no-brainer that it should be taken. I was just saying that the OP’s point about probabilities maybe being off by one percentage point, by itself, without some allegation of systematic bias in the measurement, doesn’t change the nature of the question. There’s still the further question of whether we want to bring in other considerations besides EV. (I think “attitudes towards uncertainty” fits well here as a title, but again, I would reserve it for the thing I’m describing, which is clearly different from “do you think other people/orgs within EA are going to be optimistically biased?.”)
(Note that it’s one question whether people would go by EV for cases that are well within the bounds of numbers of people that exist currently on earth. I think it becomes a separate question when you go further to extremes, like whether people would continue gambling in the St Petersburg paradox or how they relate to claims about vastly larger realms than anything we understand to be in current physics, the way Pascal’s mugging postulates.)
Finally, I realize that maybe the other people here in the thread have so little trust in the survey designers that they’re worried that, if they answer with the low-probability, higher-EV option, the survey designers will write takeaways like “more EAs are in favor of donating to speculative AI risk interventions.” I agree that, if you think survey designers will make too strong of an update from your answers to a thought experiment, you should point out all the ways that you’re not automatically endorsing their preferred option. But I feel like the EA survey already has lots of practical questions along the lines of “Where do you actually donate to?” So, it feels unlikely that this question is trying to trick respondees or that the survey designers will just generally draw takeaways from this that aren’t warranted?

Lukas_Gloor Dec 12, 2024, 4:02 PM
4 points
2 ∶ 2
in reply to: titotal’s comment on: Probabilities might be off by one percentage point
My intuitive reaction to this is “Way to screw up a survey.”
Considering that three people agree-voted your post, I realize I should probably come away with this with a very different takeaway, more like “oops, survey designers need to put in extra effort if they want to get accurate results, and I would’ve totally fallen for this pitfall myself.”
Still, I struggle with understanding your and the OP’s point of view. My reaction to the original post was something like:
Why would this matter? If the estimate could be off by 1 percentage point, it could be down to 0.5% or up to 2.5%, which is still 1.5% in expectation. Also, if this question’s intention were about the likelihood of EA orgs being biased, surely they would’ve asked much more directly about how much respondees trust an estimate of some example EA org.
We seem to disagree on use of thought experiments. The OP writes:
When designing thought experiments, keep them as realistic as possible, so that they elicit better answers. This reduces misunderstandings, pitfalls, and potentially compounding errors. It produces better communication overall.
I don’t think this is necessary and I could even see it backfiring. If someone goes out of their way to make a thought experiment particularly realistic, maybe respondees might get the impression that it is asking about a real-world situation where they are invited to bring in all kinds of potentially confounding considerations. But that would defeat the point of the thought experiment (e.g., people might answer based on how much they trust the modesty of EA orgs, as opposed to giving you their personal tolerance of risk of the feeling of having had no effect/wasted money in hindsight). The way I see it, the whole point of thought experiments is to get ourselves to think very carefully and cleanly about the principles we find most important. We do this by getting rid of all the potentially confounding variables. See here for a longer explanation of this view.
Maybe future surveys should have a test to figure out how people understand the use of thought experiments. Then, we could split responses between people who were trying to play the thought experiment game the intended way, and people who were refusing to play (i.e., questioning premises and adding further assumptions).
*On some occasions, it makes sense to question the applicability of a thought experiment. For instance, in the classical “what if you’re a doctor who has the opportunity to kill a healthy patient during routine chek-up so that we could save the lives of 4 people needing urgent organ transplants,” it makes little sense to just go “all else is equa! Let’s abstract away all other societal considerations or the effect on the doctor’s moral character.”
So, if I were to write a post on thought experiments today, I would add something about the importance of re-contextualizing lessons learned within a thought experiments to the nuances of real-world situations. In short, I think my formula would be something like, “decouple within thought experiments, but make sure add an extra thinking step from ‘answers inside a thought experiment’ to ‘what can we draw from this in terms of real-life applications.’” (Credit to Kaj Sotala, who once articulated a similar point in probably a better way.)

Lukas_Gloor Dec 2, 2024, 7:23 PM
6 points
0 ∶ 0
on: How committed to AGI safety are the current OpenAI nonprofit board members?
Probably most people are “committed to safety” in the sense that they wouldn’t actively approve conduct at their organization where executives got developers to do things that they themselves presented as reckless. To paint an exaggerated picture, imagine if some executive said the following:
“I might be killing millions of people here if something goes wrong, and I’m not super sure if this will work as intended because the developers flagged significant uncertainties and admitted that they’re just trying things out essentially flying blind; still, we won’t get anywhere if we don’t take risks, so I’m going to give everyone the go-ahead here!”
In that scenario, I think most people would probably object quite strongly.
But as you already suggest, the more relevant question is one of willingness to go the extra mile, and of qualifications: Will the board members care about and be able to gain an informed understanding of the risks of certain training runs, product releases, or model security measures? Alternatively, will they care about (and be good at) figuring out whose opinions and judgment they can trust and defer to on these matters?
With the exodus of many of the people who were concerned about mitigating risks of AI going badly wrong, the remaining culture there will likely be more focused on upsides, on moving fast, on beating competitors, etc. There will likely be fewer alarming disagreements among high-ranking members of the organization (because the ones who had alarming disagreements already left). The new narrative on safety will likely be something like, “We have to address the fact that there was this ideological club of doomer folks who used to work at OpenAI. I think they were well-intentioned (yada yada), but they were wrong because of their ideological biases, and it’s quite tragic because the technology isn’t actually that risky the way we’re currently building it.” (This is just my guess; I don’t have any direct info on what the culture is now like, so I might be wrong.)
So, my guess is that the rationales the leadership will present to board members on any given issue will often seem very reasonable ASSUMING that you go into this with the prior of “I trust that leadership has good judgment here.” The challenging task for board members will be that they might have to go beyond just looking at things the way they get presented to them and ask questions that the leadership wouldn’t even put on the meeting agenda. For instance, they could ask for costly signals of things the organization could do to create a healthy culture for assessing risks and for discussing acceptable risk tradeoffs. (Past events suggest that this sort of culture is less likely to exist than it was before the exodus of people who did safety-themed work.)
To summarize, my hope is for board members to take seriously the following three possibilities: (1) That there might be big risks in the AGI tech tree. (2) That org leadership might not believe in these risks or might downplay them because it’s convenient for them that way. (3) That org-internal discussions on risks from AI might appear one-sided because of “evaporative cooling” (most of the people who were more particularly concerned having already left for reasons that weren’t related to a person’s judgment/forecasting abilities).

Lukas_Gloor Jul 17, 2024, 1:25 PM
10 points
4 ∶ 0
on: Rethink Priorities’ Moral Parliament Tool
This seems cool!

I could imagine that many people will gravitate towards moral parliament approaches even when all the moral considerations are known. If moral anti-realism is true, there may not come a point in moral reflection under idealized circumstances where it suddenly feels like “ah, now the answer is obvious.” So, we can also think of moral parliament approaches as a possible answer to undecidedness when all the considerations are laid open.
I feel like only seeing it as an approach to moral uncertainty (so that, if we knew more about moral considerations, we’d just pick one of the first-order normative theories) is underselling the potential scope of applications of this approach.

Lukas_Gloor Jul 16, 2024, 5:39 PM
3 points
4 ∶ 2
on: Towards more cooperative AI safety strategies
Secondly, prioritizing competence. Ultimately, humanity is mostly in the same boat: we’re the incumbents who face displacement by AGI. Right now, many people are making predictable mistakes because they don’t yet take AGI very seriously. We should expect this effect to decrease over time, as AGI capabilities and risks become less speculative. This consideration makes it less important that decision-makers are currently concerned about AI risk, and more important that they’re broadly competent, and capable of responding sensibly to confusing and stressful situations, which will become increasingly common as the AI revolution speeds up.
I think this is a good point.
At the same time, I think you can infer that people who don’t take AI risk seriously are somewhat likely to lack important forms of competence. This is inference is only probabilistic, but it’s IMO pretty strong already (it’s a lot stronger now than it used to be four years ago) and it’ll get stronger still.
It also depends how much a specific person has been interacting with the technology; meaning, it probably applies a lot less to DC policy people, but applies more to ML scientists or people at AI labs.

Lukas_Gloor Jun 27, 2024, 10:22 AM
11 points
1 ∶ 0
in reply to: Nathan Young’s comment on: Truth-seeking vs Influence-seeking—a narrower discussion
Okay, what does not tolerating actual racism look like to you? What is the specific thing you’re asking for here?
Up until recently, whenever someone criticized rationality or EA for being racist or for supporting racists, I could say something like the following:
“I don’t actually know of anyone in these communities who is racist or supports racism. From what I hear, some people in the rationality community occasionally discuss group differences in intelligence, because this was discussed in writings by Scott Alexander, which a lot of people have read and so it gives them shared context. But I think this doesn’t come from a bad place. I’m pretty sure people who are central to these communities (EA and rationality) would pretty much without exception speak up strongly against actual racists.”
It would be nice if I could still say something like that, but it no longer seems like I can, because a surprising number of people have said things like “person x is quite racist, but [...] interesting ideas.”

Lukas_Gloor Jun 24, 2024, 10:04 PM

2 points

0 ∶ 0

Error

The value
  NIL
is not of type
  SIMPLE-STRING
when binding #:USER-ID163