All opinions are my own unless otherwise stated. Geophysics and math graduate with some web media and IT skills.
Noah Scales
Karma: 501
EAs should read more deep critiques of EA, especially external ones
Yes, I gave David my wish list of stuff he could discuss in a comment when he announced his blog. So far he hasn’t done that, but he’s busy with his chosen topics, I expect. I wrote quite a lot in those comments, but he did see the list.
In an answer to Elliot Temple’s question “Does EA Have An Alternative To Rational Written Debate”, I proposed a few ideas, including one on voting and tracking of an EA canon of arguments. Nobody dunked on me for it, though Elliot’s question wasn’t that popular, so I suppose few people actually read it. I appreciated Elliot’s focus on argumentation and procedure. Procedural tools to systematize debates are useful.
I’m not at all familiar with literature on impacts of diversity on decision-making. I’ll follow up on your suggestions of what to read, as much as I can. There are different kinds of diversity (worldview, race, ideology, background, expertise, …), but from what classes I took in communications studies and informal argumentation, I know that models are available and helpful to improve group discussion, and that best practices exist in several areas relevant to group communications and epistemics.
I was watching Cremer discuss ideas and read her Vox article about distributing power and changing group decision strategies. Her proposals seem serious, exciting, and somewhat technical, as do yours, ConcernedEA’s. That implies a learning curve to follow but with results that I expect are typically worth it for EA folks. Any proposal that combines serious + exciting + technical is one that I expect will be worth it for those involved, if the proposal is accepted. However, that is as seen through your perspective, one intending to preserve the community.
As someone on the outside observing your community grapple with its issues, I still hope for a positive outcome for you all. Your community pulls together many threads in different areas, and does have an impact on the rest of the world.
I’ve already identified elsewhere just what I think EA should do, and still believe the same. EA can preserve its value as a research community and supporter of charitable works without many aspects of the “community-building” it now does. Any support of personal connections outside research conferences and knowledge-sharing could end. Research would translate to support of charitable work or nonprofits explicitly tied to obviously charitable missions. I suppose that could include work on existential risk, but in limited contexts.
I have tried to make the point that vices (the traditional ones, ok? Like drugs, alcohol, betting, …) and the more general problem of selfishness are what to focus on. I’m not singling out your community as particularly vice-filled (well, betting is plausibly a strong vice in your community) but just that vices are in the background everywhere, and if you’re looking for change, make positive changes there.
And what do I mean by the “general problem of selfishness”? Not what you could expect, that I think you’re all too selfish. No. Selfishness matters because self-interest matters if altruism is your goal. Every altruistic effort is intended to serve someone else’s self-interest. Meanwhile, selfishness vs altruism is the classic conflict in most ethical decisions. Not the only one, but the typical one. The one to check for first, like, when you’re being self-serving, or when your “ethical goals” aren’t ethical at all. Yet your community has not grappled with the implications. Furthermore, no one here seems to think it matters. In your minds, you put these old-fashioned ways of thinking behind you.
You seem to have put Peter Singer’s work behind you as well, or some of you have, I think that is a mistake as well. I don’t know what kind of personal embarrassing statements or whatever that Peter Singer might have ever made, everyone seems hyper-alert to that kind of thing. But his work in ethics is foundational and should have a prominent place in your thinking and debates.
Furthermore, if you stick with your work on AGI, Bostrom’s work in Superintelligence showed insight and creativity in understanding and assessing AGI and ASI. I can’t say I agree with his thinking in further work that he’s produced, but if I were in your shoes, I wouldn’t stop mentioning his professional work just because he wrote some shameful stuff on-line, once, 20 years ago, and recently acknowledged it. Like Peter Singer, MacAskill, and many others associated with EA, Bostrom’s done impressive and foundational work(in Bostrom’s case, in AI), and it deserves consideration on its merits.
But back to writing about what I think, which has a much less impressive source.
Me.
Problems that plague humanity don’t really change. Vices are always going to be vices if they’re practiced. And selfishness? It plays such a large role in everything that we do, if you ignore it, or focus solely on how to serve others’ self-interests, you won’t grapple with selfishness well when its role is primary, for example, in contexts of existential harm. This will have two results:
your ostensible altruistic goals in those contexts will be abandoned
your further goals won’t be altruistic at all
My heuristics about a positive community are totally satisfied if your EA community focuses on giving what you can, saving the lives that you can, effective charity, effective altruism. That EA is inspiring, even inspiring guilt, but in a good way. Sure, vices are typically in the background, and corruption, plausibly, but that’s not the point. Are your goals self-contradicting? Are you co-opted by special interests already? Are you structurally incapable of providing effective charity? No, well, with caveats, but no. Overall, the mission and approach of the giving side of EA is and has been awesome and inspiring.
When EA folks go further, with your second and third waves, first existential risk prevention, now longtermism, you make me think hard about your effectiveness. You need to rock selfishness well just to do charity well (that’s my hunch). But existential risk and longtermism and community-building.… The demands on you are much much higher, and you aren’t meeting them. You need to stop all your vices, rid your community of them, prohibition-style. You need to intensively study selfishness and perform original academic research about it. I’m not joking. You really need think past current work in evolutionary psychology and utilitarianism and cognitive science. You could need to look into the past at failed research efforts and pick them up again, with new tools or ideas. Not so that you succeed with all your goals, but just so that you can stop yourself from being a significant net harm. Scout mindset was a step in the right direction and not an endpoint in improving your epistemics. Meanwhile, with your vices intact, your epistemics will suffer. Or so I believe.
If I had all the answers about selfishness vs altruism, and how to understand and navigate one’s own, I would share them. It’s a century’s research project, a multidisciplinary one with plausibly unexpected results, involving many people, experiments, different directions, and some good luck.
I don’t want to associate Singer, Cremer, Bostrom, Galef, MacAskill, or any other EA person or person who I might have referenced with my admittedly extreme and alienating beliefs about betting and other vices or with my personal declarations about what the EA community needs to do. I imagine most folks beliefs about vices and selfishness reflect modern norms and that none would not take the position that I am taking. And that’s OK with me.
However, register my standards for the EA community as extreme given the goals you have chosen for yourself. The EA community’s trifecta of ambitions is extreme. So are the standards that should be set for your behavior in your everyday life.
What about testing code for quality, that is, verifying code correctness, thereby reducing bugs?
Newcomb’s problem, honesty, evidence, and hidden agendas
Thought experiments are usually intended to stimulate thinking, rather than be true to life. Newcomb’s problem seems important to me in that it leads to a certain response to a certain kind of manipulation, if it is taken too literally. But let’s assume we’re all too mature for that.
In Newcomb’s problem, a person is given a context, and a suggestion, that their behavior has been predicted beforehand, and that the person with that predictive knowledge is telling them about it . There are hypothetical situations in which that knowledge is correct, but Newcomb’s problem doesn’t appear to be one of them.
But to address the particulars I will focus on testing the scientist’s honesty and accuracy. Let’s recap quickly:
-
the scientist claims to make a prediction, and that the prediction determines one of two possible behavioral options. You take two boxes from the scientist, or take the opaque one only.
-
the scientist claims to make a decision about whether to put $1,000,000 in an opaque box before interacting with a person(you) who enters the scientist’s tent using a brain scan machine posted at the tent entrance. The brain scan machine gives the scientist a signal about what you’re likely to do, and the scientist either puts a million in the opaque box, or not. In addition, there’s a clear box in the tent containing $1000.
-
you can’t see what’s in the opaque box the whole time you’re in the tent. You can see the $1000 the entire time.
-
if the scientist believes what they claim, then the scientist thinks that interaction with you will have no affect on what you do once you walk in the tent. It was decided when you walked through the door. In other words, in the scientist’s mind, no matter what the scientist or you would otherwise do, only one of two outcomes will occur. You will take both boxes or just the opaque box.
So here’s what I think. There are far more situations in life where someone tells you a limited set of your options from a larger set than there are situations in which someone tells you your full set of options. The scientist claimed only two outcomes would occur (put differently, you would do one of two things). The scientist supposedly has this brain scan technology that tells them what your two options are, and the scientist is confident that the technology works. Your willingness to believe the scientist at all depends on the scientist’s claims being believed in their entirety. That means the scientist’s claims about the reliability of the machine as well. Once some claims show as false, you have reason to question the rest. At that point, the thought experiment’s setup fails. Let’s test the scientist’s claims.
So, don’t take either box. Instead, walk out of the tent. If you make it out without taking any boxes, then you know that the scientist was wrong or lying about what you would do. You did not take any boxes. You just left both boxes on the table. Now, think this over. If the scientist was sincere, then there’s a mad scientist with a $1,001,000 in the tent you just walked out of who either thought you would follow their instructions or thought that they had predicted you so well that they could just tell you what you would do. If the scientist was not sincere, then there’s a lying and manipulative scientist in the tent with a $1,000 and an opaque mystery box that they’re hoping you’ll take from them.
BTW: If someone gives me free money, even a $1000, to take a mystery package from them, I decline.
But, you say, “I think it’s understood that you could walk out of the tent, or start a conversation, maybe even ask the scientist about the opaque box’s contents, or do other things instead.” However, if that’s so, why couldn’t you just take the $1000, say thanks, and leave rather than take the opaque box with you? What constrained your freedom of choice?
Was it the mad scientist? Did the mad scientist zipper the tent entrance behind you and booby-trap the boxes so you either take both boxes or just the opaque one? Is the scientist going to threaten you if you don’t take either box? If so, then you’ve got a mad scientist who’s not only interested in predicting what you do, but also interested in controlling what you do, by constraining it as much as they can. And that’s not the thought experiment at all. No, the thought experiment is about the scientist predicting you, not controlling you, right? And you’re an ethical person, because otherwise you would shake the scientist down for the million still in the tent, so we’ll ignore that option.
However, in case the thought experiment is about the scientist controlling you, well, I would leave the tent immediately and be grateful that the scientist didn’t choose to keep you there longer. That is, leave if you can. Basically, it seems that if you do anything too creative in response to the scientist, you could be in for a fight. I would go with trying to leave.
But lets assume you don’t believe that the scientist is controlling you in any way, something about controlling you seems like a different thought experiment. Lets just go with you walking out of the tent without any boxes. Catch your breath, think over what happened, and don’t go back in the tent and try to interact with the scientist anymore. Remember, anyone willing to do that sort of thing to strangers like you is plausibly a desperate criminal wanting you to take a mysterious package from them. Or a distraught (and plausibly delusional) scientist who you just proved has a worthless brain scan machine that they wasted millions of dollars testing.
EDIT: ok, so in case it’s not obvious, you disproved that the scientist’s brain scanner works. It predicted two behavioral outcomes, and you chose a third from several, including:
trying to take the $1000 out of the clear box and leaving the opaque box behind
shaking down the scientist for the million presumably in the tent somewhere, if it’s not all in the two boxes
starting a conversation with the scientist, maybe to make a case that you really need a million dollars no matter what kind of decision-maker you are
leaving the tent asap
and plausibly others
By disproving that the brain scanner works reliably, you made a key claim of the scientist’s false: “my brain scanner will predict whether you take both boxes or only one”. Other claims from the scientist, like “I always put a million in the opaque box if my brain scanner tells me to” and “So far, my brain scanner has always been right” are now suspect. That means that the scientist’s behavior and the entire thought experiment can be seen differently, perhaps as a scam, or as evidence of a mad scientist’s delusional belief in a worthless machine.
You could reply:
-
“What if the brain scanning machine only works for those situations where you take both boxes or only the opaque box and then just leave?”: Well, that would mean that loads of people could come in the tent, do all kinds of things, like ransack it, or take the clear box, or just leave the tent while taking nothing, and the machine gives the scientist a bogus signal for all of those cases. The machine has, then, been wrong, and frequently.
-
“What if the brain scanner gives no signal if you won’t do one of the two things that the scientist expects?”: Interesting, but then why is the scientist telling you their whole speal (“here are two boxes, I scanned your brain when you came through the door, blah blah blah...”) after finding out that you won’t just take one of the two options that the scientist offers? After all, as a rational actor you can still do all the things you want to do after listening the scientist’s speal.
-
“Maybe the scientist changes their speal, adds a caveat that you follow their instructions in order for the predictions to work.” OK, then. Let’s come back to that.
-
“What if there are guards in the tent, and you’re warned that you must take either the opaque box or both boxes or the guards will fatally harm you?”: Well, once again, it’s clear that the scientist is interested in controlling and limiting your behavior after you enter the tent, which means that the brain scanner machine is far from reliable at predicting your behavior in general.
-
“Hah! But you will choose the opaque box or both boxes, under duress. This proves that some people are one-boxers and others are two-boxers. I got you!”: Well, some people would follow the scientist’s instructions, you’re right. Other people would have a panic attack, or ask the scientist which choice the scientist would prefer, or just run for their lives from the tent, or even offer the guards a chance to split the scientist’s money if the guards change sides. Pretty soon, that brain scanning machine is looking a lot less relevant to what the tent’s visitors do than the guards and the scientist are. From what I understand, attempting to give someone calm and reassuring instructions while also threatening their lives (“Look, just take the $1000 and the opaque box, everything will be fine”) doesn’t tend to work very well
-
“Wait a minute. What if the scientist has a brain scanning device that predicts 100′s of different behaviors you could do by scanning you as you walk in the tent, and …”: Let me stop you there. If the scientist needs that kind of predictive power, and develops it, it’s _ to know what to do_ when you walk in the tent, not just to know what you will do when you walk in the tent. And just because the scientist knows what you will do if you’re confronted with a situation, doesn’t mean that the scientist has a useful response to what you will do. At this point, whose decision-making is really under the microscope, the tent’s visitor, or the scientist’s?
-
“Let’s back this up. All we’re really thinking about is someone who willingly participates in the scientist’s game, trusts the scientist, and follows the scientist’s instructions. Aren’t you just distorting the experiment’s context?” If someone claims to be able to predict your behavior, and the only way for their predictions to ever seem accurate is for you to play along with the options they provide, then don’t you see that dishonesty is already present? You are the one being dishonest, or you both are. You’re playing along with the mad scientist, or the mad scientist isn’t mad at all, but has some ulterior motive for wanting you take an opaque box with you, or otherwise participate in their bizarre game. The predictions aren’t really about what you would do if confronted with two boxes in such a situation. The predictions are make-believe that you play with someone with boxes in a tent, and only if you’re that kind of person. Not everyone is.
-
No, you just said that the visitor to the tent is ‘playing along’. But the thought experiment is about someone who trusts the scientist , and playing along is not trusting the scientist .” Yes, exactly the kind of thing that I’ve been cautioning you about. Don’t be one of those people. There are people who trust you and select among the options you give them for whatever reason you offer, no matter how contrary to existing evidence (e.g., of their own free will) the option selection is. Their decision strategies do not include acting on good evidence or understanding causality very well. And such people would likely leave with just the opaque box, and, if the scientist is to be believed, will be rewarded for it with a million dollars. However, they fall for every magic trick, and do not gather evidence carefully.
-
No, no, it’s not a magic trick. The thought experiment says that the scientist is really checking the brain scanning machine and putting the money in the opaque box, or not, according to what the machine says, and then making the same claims to every visitor about how the whole experiment works, and asking the visitors to participate according to the scientist’s simple instructions. All along you’ve been distorting this every which way. The machine could fail, but we know it succeeds. It succeeds with everybody, and the point of the thought experiment is just to think through what you ought to do in that situation, to get the most money, if you agree to the scientist’s terms. The only way to prove the scientist is wrong as a single visitor is to do everything right, leave with the opaque box only, but then find nothing inside. But we know that never happens. I see. Yeah. OK! I think you’ve changed the experiment a little though. Before, it was just, walk in, and get predicted. Now, it’s walk in and choose to cooperate, and the scientist is telling the truth, and the brain scanning machine appears to work, and then get predicted. And you can’t just play along, a visitor has to believe the scientist, and for good reason, in order to for people to draw any conclusions about what the experiment means.
-
“What? No, you don’t have to believe the scientist. You can play along, get some money, just choose one or two boxes. That’s what everyone should do, and the experiment shows it.” Some people would do that. We might as well flip a coin, or just pretend that we have reason to believe the scientist’s claim for causal reasons, and make up a causal reason. How about something like, “Hey, that million in the opaque box is like Schrodinger’s cat.” Maybe we make up a causal reason in hindsight after we find that million in the opaque box and leave the clear box behind. However, “rational” people would only follow the instructions if they believed the evidence warranted it, then those “rational” people would explore the reasons why. As far as I know, this thought experiment is supposed to mean that evidential and causal decision theory can conflict, but in fact, I think it only means that causal decisions can be revised based on new evidence. For example, brain scanner prediction, mind control, subtle influence by the scientist, money teleportation, time travel by someone observing you and taking the money back in time, or an unlikely string of random predictive success by a totally useless brain scanner, all potential explanations of the reason that the scientist’s machine would appear to work, if you decided to test if it works by taking the opaque box.
-
So what? Then the thought experiment only applies to people who follow instructions and trust the scientist and have good reason to trust the scientist’s claims, if you accept the idea that it’s supposed to distinguish evidential and causal decision theory. All your discussion of it managed to do was convince me that the thought experiment is well-designed, but also plausible. I think brain scanners like that, that work specific to a context where you choose to follow instructions, are plausible. If they were built, then setting something like this up in real life would be easy.” Yeah, and expensive. Plenty of people would take the opaque box only. I think this makes me want to revise the definition of “plausible” a little bit, for myself. I would just leave the tent. Julia Galef also thinks that such devices as brain scanners are plausible, or she claimed that, in her old video. So you’re in good company.
And thanks!
-
Regarding decision theory: I responded to you on substack. I’ll stand by my thought that real-world decisions don’t allow accurate probabilities to be stated, particularly in some life-or-death decision. Even if some person offered to play a high-stakes dice game with me, I’d wonder if the dice are rigged, if someone were watching us play and helping the other player cheat, etc.
Separately, it occurred to me yesterday that a procedure to decide how many chances to take depends on how many will meet a pre-existing need of mine, and what costs are associated with not fulfilling that need. For example, if I only need to live 10 more years, or else something bad happens (other than my death), and the chances that I don’t live that extra 10 years are high unless I play your game (as you say, I only have 1 year to live and your game is the only game in town that could extend my life), then I will choose an N of 1. This argument can be extended to however many years. There are plausible futures in which I myself would need to live an extra 100 years, but not an extra 1000, etc. That allows me to select a particular N to play for in accordance with my needs.
Let’s say, though, for the sake of argument, that I need to live the entire 10^50000 years or something bad will happen. In that case, I’m committed to playing your game to the bitter end if I want to play at all. In which case, if I choose to play, it will only be because my current one year of life is essentially worthless to me. All that matters is preventing that bad thing from happening by living an extra 10^50000 years.
Alternatively, my only need for life extension is to live longer than 10^50,000. If I don’t, something bad will happen, or a need of mine won’t be met. In that case, I will reject your game, since it won’t meet my need at all.
This sort of thinking might be clearer in the case of money, instead of years of life. If I owed a debt of 10^50000 dollars, and I only have 1 dollar now, and if in fact, the collector will do something bad to me unless I pay the entire debt, well, then, the question becomes whether your game is the only game in town. If so, then the final question is whether I would rather risk a large chance of dying to try to pay my debt, or whether I would rather live through the collector’s punishment because the money I have already (or earned from playing your game to less than the end) is short of the full repayment amount. If death were preferable to the collector’s punishment and your game was the only game in town, then I would go with playing your game even though my chances of winning are so pitifully small.
Similar thinking applies to smaller amounts of debt owed, and commensurate choice of N sufficient to pay the debt if I win. I will only play your game out to high enough N to earn enough to pay the debt I owe.
Regardless of the payoff amount’s astronomically large size as the game progresses, there is either a size past which the amount is greater than my needs, or my needs are so great that I cannot play to meet my needs (so why play at all), or only playing to the end meets my needs. Then the decision comes down to comparing the cost of losing your game to the cost of not meeting my pre-existing needs.
Oh, you could say, “Well, what if you don’t have a need to fulfill with your earnings from playing the game? What if you just want the earnings but would be ok without them?” My response to that is, “In that case, what am I trying to accomplish by acquiring those earnings? What want (need) would those earnings fulfill, in what amount of earnings, and what is the cost of not acquiring them?”
Whether it’s money or years of life or something else, there’s some purpose(s) to its use that you have in mind, no? And that purpose requires a specific amount of money or years. There’s not an infinitude of purposes, or if there are, then you need to claim that as part of proposing your game. I think most people would disagree with that presumption.
What do you all think? Agree? Disagree?
On policy, there’s Annie Duke’s idea of “resulting”, that just because a policy leads to success or failure doesn’t necessarily speak to whether it was the strategically best choice. Causes of policy failure go beyond the policy specifics. For example, bad luck is a cause of policy failure. Accordingly, then, you can be certain your policy choice is the best but still be doubtful of the intended outcome’s occurrence.
There’s a bit of irony in that we should also realize our ignorance of what others want from policy, stated goals are not necessarily shared goals.
[Question] What do EA’s think about Bayesian inferences versus other types?
There’s no agreement that there is a meta-crisis. Yes, there are multiple sources of danger, and they can interact synergistically and strongly (or so I believe), but that’s not the same as saying that there must be root causes for those (global, existential) dangers that humanity can address.
If you asked a different question, like: “What are the underlying drivers of the multiple anthropogenic existential threats that we all face, like nuclear war, engineered pandemics, climate destruction, etc?”
You could get some interesting answers from people who think in those terms. I’m curious what others here think.
But the answer to why underlying drivers are not addressed is easy to deduce: lack of belief, interest, or education in the matter.
There’s this thing, “the repugnant conclusion”. It’s about how, if you use aggregate measures of utility for people in a population, and consider it important that more people each getting the same utility means more total utility, and you think it’s good to maximize total utility, then you ought to favor giant populations of people living lives barely worth living.
Yes, it’s a paradox. I don’t care about it because there’s no reason to want to maximize total utility by increasing a population’s size that I can see. However, by thinking so, I’m led down a different path. I’m not a utilitarian, but I check in with the utilitarian perspective to understand some things better.
The form of utilitarianism that I introduce below is my best utilitarian perspective. I created it as part of rejecting the repugnant conclusion. I’ll let you ask the interested questions, if you have any, lol. Here it is.
Imagine an accounting system that, for each person, measures the utility, positive and negative, of that person’s actions for other people. Your own personal utilitarian ledger, but lets assume someone else keeps it for you. That other person knows every action you take and what positive or negative utility that your actions create.
If the term “utility” confuses you, think of other terms, like:
benefit or harm
happiness or suffering
gain or loss
pleasure or pain
improvement or decline
For example, positive utility that you create for someone could be an improvement in their health.
Your ledger holds information about what you cause people everywhere, millions, billions, even trillions of people, now and in the future. Well, OK, that’s only if you consider individuals from various other species as deserving a page in your ledger.
How would I make this ledger work? Here’s what I would do:
-
Put aside the mathematical convenience of aggregate measures in favor of an individual accounting of utility. If you can track the utility you cause for even two other people, your ledger keeper should be able to do it for two hundred billion, right? Sure.
-
Set up a few rules to handle when people cease to exist. Those rules should include:
-
Once a person’s existence ends, you can no longer create utility for that person. Accordingly, there should be no new entries onto your ledger about that person. Prior utility accounting associated with a person from when they were alive can be kept but not altered unless to better reflect utility that you created for the person when the person was still living.
-
Ledger entries associated with people who were expected to be conceived but are no longer expected to be conceived must be deleted entirely from the ledger, because those entries apply to a never-existent person. They are bogus.
-
Entries about the utility of termination of existence (death) that you (inadvertently) cause others should be full and complete, applying to all those affected by a death who are not the dead person, including everyone still living and who will be conceived that get positive or negative utility from the person’s death.
-
The suffering or happiness involved in the person’s going through the process of dying should also be considered negative or positive utility and accounted for accordingly. A painful, slow death is a large negative harm to inflict on someone, whereas a quick, painless death in the presence of loving family is an improvement over a painful slow death, all other things equal.
-
Do not record death itself as a change in utility. The fact of death itself should not be recorded as a negative (or positive) utility applying to the now nonexistent person. There are still all the harms of death noted previously. Aside from those however, the only change recorded on the ledger to the dead person’s utility is that there are no longer events generating new utility for the person because the person no longer exists.[1]
Do not record intended consequences as creating utility just because they were intended. That is a different form of morality tracking, to do with keeping a record of a person’s character. On the utilitarian ledger, only actual utility gets recorded in an entry.
Other than those changes, I think you can go ahead and practice utilitarianism as you otherwise would, that is, doing the the greatest good for the greatest number, and considering all people as equally deserving of consideration.
Utilitarianism developed in that way does not offer the typical problems of:
-
aggregate measures (average, total, variance) screwing up determination of utility maximization for many individuals
-
bogus accounting of utility intended for nonexistent or never-existent people.
-
bogus accounting of utility intended to be created for existent people but not actually created.
This personal utilitarian ledger only tells you about actual utility created in a single shared timeline for a population of individuals. Intentions and alternatives are irrelevant. Disliking death or liking for having children are similarly irrelevant unless contradiction of those values is considered a negative utility created for existent people. Of course there’s still the harms to existent others associated with death or absence of conception that are recorded in the ledger. And, the welfare of the population as a whole is never actually considered.
An extension to the accounting ledger, one that tracks consequences of actions for your utility, would record your actions including such interesting ones as actions to make hypothetical people real or to extend the lives of existing people. The extension would record actual consequences for you even if those actions create no utility for other existing people. You might find this extension useful if, as someone with a ledger, you want to treat your own interests as deserving equal consideration compared to other’s interests.
For me, a utilitarian ledger of this sort, or a character ledger that tracks my intentions and faithfully records evidence of my character, would provide a reference point for me to make moral judgments about me. Not a big deal, but when you look at something like the repugnant conclusion, you could ask yourself, “Who does this apply to and how?” I don’t require that I practice utilitarianism, but in a context where utilitarian considerations apply, for example, public policy, I would use this approach to it. Of course, I’m no policy-maker, so this ledger is little more than a thought experiment.
[1] The only exception would be error-correction events to revise old utility information from when the person was living. Error-correction events only occur when the ledger keeper corrects a mistake.
Directly address the substance of all criticisms of EA.
if a criticism contains a faulty premise, identify it and rebut it.
if a criticism uses poor reasoning, identify it and reject it.
if a criticism contains valid elements, identify and acknowledge them all.
Use the source’s language as much as you can, rather than add your own jargon. Using your jargon and writing for other EA’s makes you less credible and legitimate. It looks like obfuscation to the source of the criticism and to other outsiders reviewing your response.
Avoid going meta. Going meta to a criticism is not impressive to outsiders. Such meta-comments as:
“The number of errors in this criticism is alarming!”
“Geez, these bullies just won’t stop.”
“Oh, another boring, wrong criticism.”
“We should get a PR firm to handle these kinds of reputation attacks!”
and other typical options are useless and counterproductive.
By the way, if you actually want to use a PR firm to handle criticisms, don’t keep writing about it but go get one, because constantly discussing it is embarrassing, given your preferred reputation as rational people, as opposed to people who would hire a PR firm. You post those thoughts in a public forum. Your critics think you look weak every time you do that, and their criticisms, justified or not, seem validated to them and to others who judge you by your defensiveness and lack of a “high quality” response.
Otherwise, act on the reputation you aspire to, skip the meta talk, and address the criticism with a clear, rational analysis that the source of the criticism and interested observers can understand and appreciate. Don’t expect that they will then agree with your response or give EA more respect, but do expect that anyone who cares about the truth will recognize the integrity behind your response to the criticism. Whether the criticism is “high quality” or not, whether you were bored by it or not.
You can always ignore criticisms as well, for whatever reason. Maybe your busy schedule.
But if you do respond to criticism, it doesn’t matter that you don’t “speak for EA”, but rather, that EA contains community members who can and do practice the rationality they profess. You want that reputation. Go after it with your responses to criticism. You’re doing your community a favor that way.
Well, I’ve been noodling that human physiology defines our senses, our senses limit our ability to represent information to ourselves, and correction for differences of sensory representation of different sets of information from the same class allows for better comparisons and other reasoning about each (for example, interpreting) . A classic example is television pharmaceutical drug ads. The ads present verbal information about the dangers of a medication in tandem with visual information showing happy people benefiting from the same medication. Typically.
Does “intuition” have a specific, carefully-guarded meaning in moral philosophy? Intuition as I understand it is vague. The term “intuition” captures examples of lots of opinions and preferences and conclusions that share the attribute of having a feeling or partial representation to the person holding them. For example, some moral intuitions could develop through or depend on personal experience but have this property of having a vague representation. For someone using my definition of “intuition”, a discussion of whether all moral intuitions are evolutionarily-driven seems clearly wrong.
I made a critique of EA that I think qualifies as “deep” in the sense that it challenges basic mechanisms established for bayesianism as EA’s practice it, what you call IBT, but also epistemic motives or attitude. This was not my red-team, but something a bit different.
The Scout Mindset offers a partitioning of attitudes relevant to epistemics if its categories of “scout” and “soldier” are interpreted broadly. If I have an objection to Julia Galef’s book “The Scout Mindset”, it is in its discussion of odds. Simply the mention of “odds.” I see it as a minor flaw in an otherwise wonderful and helpful book. But it is a flaw. Well, it goes further, I know, but that’s an aside.
A current of betting addiction running through EA could qualify as a cause for acceptance of FTX money. These crypto-currency markets are known financial risks and also known purveyors to corrupt financial interests. Their lack of regulation has been noted by the SEC and for years, crypto has been associated with scams. For the last couple years, the addition of obviously worthless financial instruments via “web3” was an even bigger sign of trouble. However, to someone who sees betting as a fun, normal, or necessary activity, an investment or placement of faith in FTX makes more sense. It’s just another bet.
The vice of betting, one of the possibilities that explains IBT results, is in my view obvious, and has been known for 1000′s of years, to have bad results. While you EA folks associate betting with many types of outcomes other than earnings for yourselves, and many scenarios of use of money (for example, investments in charitable efforts), overall, betting should have the same implications to you as it has had to human communities for 1000′s of years. It leads away from positive intentions and outcomes, and corrupts its practitioners. The human mind distorts betting odds in the pursuit of the positive outcome of a bet. Far from improving your epistemics, betting hinders your epistemics. On this one point, folks like Julia Galef and Annie Duke are wrong.
When did EA folks decide that old, generations-tested ideas of vices, were irrelevant? I think, if there’s a failure in the “smartest people in the room” mentality that EA fosters, it’s in the rejection of common knowledge about human failings. Consequences of vices identify themselves easily. However you consider their presence in common-sense morality, common knowledge is there for you.
Meanwhile, I don’t know the etiology of the “easy going” approach to vices common now. While I can see that many people’s behaviors in life remain stable despite their vices, many others fall, and perhaps it’s just a question of when. In a group, vices are corrosive. They can harm everyone else too, eventually, somehow. You built EA on the metaphor of betting. That will come back to bite you, over and over.
Your many suggestions are worthwhile, and Scout Mindset is a necessary part of them, but Galef didn’t address vices, and you folks didn’t either, even though vices wreck individual epistemics and thereby group epistemics. They’re an undercurrent in EA, just like in many other groups. Structural changes that ignore relevant vices are not enough here.
You folks lost billions of dollars promised by a crypto guy. Consider the vice of betting as a cause, for your choice to trust in him and his actions in response and in general. Regardless of whether it was corrupt or sanctioned betting, it was still betting, the same movie, the typical ending. Well, actually, since betting is now a sport and skilled bettors are now heroes, I guess common knowledge isn’t so common anymore, at least if you watch the movies.
EDIT: Oh! It was rockstrom, but the actual quote is: “The richest one percent must reduce emissions by a factor [of] 30, while the poorest 50% can actually increase emissions by a factor [of] 3” from Johan Rockström at #COP26: 10 New Insights in Climate Science | UN Climate Change. There he is talking about fair and just carbon emissions adjustments. The other insights he listed have economic implications as well, if you’re interested. The accompanying report is available here.
The quote is:
“Action on climate change is a matter of intra- and intergenerational justice, because climate change impacts already have affected and continue to affect vulnerable people and countries who have least contributed to the problem (Taconet et al., Reference Taconet, Méjean and Guivarch2020). Contribution to climate change is vastly skewed in terms of wealth: the richest 10% of the world population was responsible for 52% of cumulative carbon emissions based on all of the goods and services they consumed through the 1990–2015 period, while the poorest 50% accounted only for 7% (Gore, Reference Gore2020; Oswald et al., Reference Oswald, Owen, Steinberger, Yannick, Owen and Steinberger2020).
A just distribution of the global carbon budget (a conceptual tool used to guide policy) (Matthews et al., Reference Matthews, Tokarska, Nicholls, Rogelj, Canadell, Friedlingstein, Thomas, Frölicher, Forster, Gillett, Ilyina, Jackson, Jones, Koven, Knutti, MacDougall, Meinshausen, Mengis, Séférian and Zickfeld2020) would require the richest 1% to reduce their current emissions by at least a factor of 30, while per capita emissions of the poorest 50% could increase by around three times their current levels on average (UNEP, 2020). Rich countries’ current and promised action does not adequately respond to the climate crisis in general, and, in particular, does not take responsibility for the disparity of emissions and impacts (Zimm & Nakicenovic, Reference Zimm and Nakicenovic2020). For instance, commitments based on Nationally Determined Contributions under the Paris Agreement are insufficient for achieving net-zero reduction targets (United Nations Environment Programme, 2020).”
Whether 1.5 is really in reach anymore is debatable. We’re approaching an El Nino year, it could be a big one, we could see more heat in the atmosphere then, let’s see how close we get to 1.5 GAST then. It won’t be a true GAST value, I suppose, but there’s no way we’re stopping at 1.5 according to Peter Carter:
“This provides more conclusive evidence that limiting to 1.5C is impossible, and only immediate global emissions decline can possibly prevent a warming of 2C by 2050”
and goes on from there.… He prefers CO2e and radiative forcing rather than the carbon budget approach as mitigation assessment measures. It’s worth a viewing as well.
There’s quite a lot to unpack in just these two sources, if you’re interested.
Then there’s Al Gore at the World Economic Forum, who drops some truth bombs: “Are we going to be able to discuss… or putting the oil industry in charge of the COP … we’re not going to disguise it anymore”
OLD:I believe it was Rockstrom, though I’m looking for the reference, who said that citizens of developed countries needed to cut their per capita carbon production by 30X, while in developing countries people could increase it by 3X. That’s not a quote, but I think the numbers are right.
That is a counterpoint to the analysis made by some climate economists.
When I find the reference I’ll share it, because I think he was quoting an analysis from somewhere else, and that could be useful to your analysis given the sources you favor, even if you discount Rockstrom.
Great fun post!
I read the whole post. Thanks for your work. It is extensive. I will revisit it. More than once. You cite a comment of mine, a listing of my cringy ideas. That’s fine, but my last name is spelled “Scales” not “Scale”. :)
About scout mindset and group epistemics in EA
No. Scout mindset is not an EA problem. Scout and soldier mindset partition mindset and prioritize truth-seeking differently. To reject scout mindset is to accept soldier mindset.
Scout mindset is intellectual honesty. Soldier mindset is not. Intellectual honesty aids epistemic rationality. Individual epistemic rationality remains valuable. Whether in service of group epistemics or not. Scout mindset is a keeper. EA suffers soldier mindset, as you repeatedly identified but not by name. Soldier mindset hinders group epistemics.
We are lucky. Julia Galef has a “grab them by the lapel and shake them” interest in intellectual honesty. EA needs scout mindset.
Focus on scout mindset supports individual epistemics. Yes.
scout mindset
critical thinking skills
information access
research training
domain expertise
epistemic challenges
All those remain desirable.
Epistemic status
EA’s support epistemic status announcements to serve group epistemics. Any thoughts on epistemic status? Did I miss that in your post?
Moral uncertainty
Moral uncertainty is not an everyday problem. Or remove selfish rationalizations. Then it won’t be. Or revisit the revised uncertainty, I suppose.
Integrity
Integrity combines:
intellectual honesty
introspective efficacy
interpersonal honesty
behavioral self-correction
assess->plan->act looping efficacy
Personal abilities bound those behaviors. So do situations. For example, constantly changing preconditions of actions bound integrity. Another bound is your interest in interpersonal honesty. It’s quite a lever to move yourself through life, but it can cost you.
Common-sense morality is deceptively simple
Common-sense morality? Not much eventually qualifies. Situations complicate action options. Beliefs complicate altruistic goals. Ignorance complicates option selection. Internal moral conflicts reveal selfish and altruistic values. Selfishness vs altruism is common-sense moral uncertainty.
Forum karma changes
Yes. Lets see that work.
Allow alternate karma scoring. One person one vote. As a default setting.
Allow karma-ignoring display. On homepage. Of Posts. And latest comments. As a setting.
Allow hide all karma. As a setting.
Leave current settings as an alternate.
Diversifying funding sources and broader considerations
Tech could face lost profits in the near future. “Subprime Attention Crisis” by Tim Hwang suggests why. An unregulated ad bubble will gut Silicon Valley. KTLO will cost more, percentage-wise. Money will flow to productivity growth without employment growth.′
Explore income, savings, credit, bankruptcy and unemployment trends. Understand the implications. Consumer information will be increasingly worthless. The consumer class is shrinking. Covid’s UBI bumped up Tech and US consumer earnings temporarily. US poverty worsened. Economic figures now mute reality. Nevertheless, the US economic future trends negatively for the majority.
“Opportunity zones” will be a predictive indicator despite distorted economic data, if they ever become reality. There are earlier indicators. Discover some.
Financial bubbles will pop, plausibly simultaneously. Many projects will evaporate. Tech’s ad bubble will cost the industry a lot.
Conclusion
Thanks again for the post. I will explore the external links you gave.
I offered one suggestion (among others) in a red team last year: to prefer beliefs to credences. Bayesianism has a context alongside other inference methods. IBT seems unhelpful, however. It is what I advocate against, but I didn’t have a name for it.
Would improved appetite regulation, drug aversion, and kinesthetic homeostasis please our plausible ASI overlords? I wonder. How do you all feel about being averse to alcohol, disliking of pot, and indifferent to chocolate? The book “Sodium Hunger: The Search for a Salty Taste” reminds me that cravings can have a benefit, in some contexts. However, drugs like alcohol, pot, and chocolate would plausibly get no ASI sympathy. Would the threat of intelligent, benevolent ASI that take away interest in popular drugs (e.g ,through bodily control of us) be enough to halt AI development? Such a genuine threat might defeat the billionaire-aligned incentives behind AI development.
By the way, would EA’s enjoy installing sewage and drinking water systems in small US towns 20-30 years from now? I am reminded of “The End Of Work” by Jeremy Rifkin. Effective altruism will be needed from NGO’s working in the US, I suspect.
It says something about accountability and the importance of feedback, that is, as consequences accumulate, feedback about them is fairly important. People recognize ideologies that do not depend on feedback for their claims of good intentions knowing that such ideologies are trojan horses for counterproductive plans, as longtermism appears to be.
You don’t know yet how Shell’s ownership affects what Sonnen does in the marketplace. If you think home batteries are a net positive morally then it’s just a matter of comparing the impact of Sonnen with the impact of other companies where you could work.
Home batteries are part of the energy transition at small scale but I don’t believe they matter at large scale in terms of reducing climate destruction. However, home batteries are great for buffering against blackouts and if I were a homeowner, I would be grateful to have a battery technology like Sonnen’s.
Oh, I see. So by “benign” you mean shaming from folks holding common-sense but wrong conclusions, while by “deserved” you mean shaming from folks holding correct conclusions about consequences of EA actions. And “compromise” is in this sense, about being a source of harm.
I have read the Democratizing Risk paper that got EA criticism and think it was spot on. Not having ever been very popular anywhere (I get by on being “helpful” or “ignorable”), I use my time here to develop knowledge.
Your work and contributions could have good timing right now. You also have credentials and academic papers, all useful to establish your legitimacy for this audience. It might be useful to check to what extent TUA had to do with the FTX crisis, and whether a partitioning of EA ideologies combines or separates the two.
I believe that appetite for risk and attraction to betting is part and parcel of EA, as is a view informed more by wealth than by poverty. This speaks to appetite for financial risk and dissonance about charitable funding.
Critiques of EA bureaucracy could have more impact than critiques of EA ideology. Certainly your work with Luke Kemp on TUA seems like a hard sell for this audience, but I would welcome another round, there’s a silent group of forum readers who could take notice of your effort.
Arguments against TUA visions of AGI just get an ignoring shrug here. Climate change is about as interesting to these folks as the threat of super-fungi. Not very interesting. Maybe a few 100 points on one post, if the author speaks “EA” or is popular. I do think the reasons are ideological rather than epistemic, though ideologies do act as an epistemic filter (as in soldier mindset).
It could be that EA folks:
risk criticism for all actions. Any organization risks criticism for public actions.
deserve criticism for any immoral actions. Immoral actions deserve criticism.
risk criticism with risky actions whose failure has unethical consequences and public attention. EA has drawn criticism for using expected value calculations to make moral judgments.
Is that the compromise you’re alluding to when you write:
But the greater part of it being normal is that all action incurs risk, including moral risk. We do our best to avoid them (and in my experience grantmakers are vigilant about negative EV things), but you can’t avoid it entirely. (Again: total inaction also does not avoid it.) Empirically, this risk level is high enough that nearly everyone eventually bites it.
SBF claimed that, if events had gone differently, FTX would have recovered enough funds to carry on. In that hypothetical scenario, FTX’s illegal dealing with Alameda would have gone unnoticed and would have had no adverse financial consequences. Then the risk-taking is still unethical but does not inspire criticism.
There is a difference between maximizing potential benefits and minimizing potential harms. It’s not correct to say that minimizing unavoidable harms from one’s actions has negative consequences for others and therefore those actions are immoral options, unless all one means by an immoral action is that the action had negative consequences for others.
I don’t think there’s unanimity about whether actions should be taken to minimize harms, maximize benefits, or some combination.
If all it means to “bite it” is that one takes actions with harmful consequences, then sure, everyone bites the bullet. However, that doesn’t speak to intention or morality or decision-making. There’s no relief from the angst of limited altruistic options in my knowing that I’ve caused harm before. If anything, honest appraisal of that harm yields the opposite result. I have more to dislike about my own attempts at altruism. In that way, I am compromised. But that’s hardly a motive for successful altruism. Is that your point?
Hmm. I’ve watched the scoring of topics on the forum, and have not seen much interest in topics that I thought were important for you, either because the perspective, the topic, or the users, were unpopular. The forum appears to be functioning in accordance with the voting of users, for the most part,because you folks don’t care to read about certain things or hear from certain people. It comes across in the voting.
I filter your content, but only for myself. I wouldn’t want my peers, no matter how well informed, deciding what I shouldn’t read, though I don’t mind them recommending information sources and I don’t mind recommending sources of my own, on a per source basis. I try to follow the rule that I read anything I recommend before I recommend it. By “source” here I mean a specific body of content, not a specific producer of content.
I actually hesitate to strong vote, btw, it’s ironic. I don’t like being part of a trust system, in a way. It’s pressure on me without a solution.
I prefer “friends” to reveal things that I couldn’t find on my own, rather than, for their lack of “trust”, hide things from me. More likely, their lack of trust will prove to be a mistake in deciding what I’d like to read. No one assumes I will accept everything I read, as far as I know, so why should they be protecting me from genuine content? I understand spam, AI spam would be a real pain, all of it leading to how I need viagra to improve my epistemics.
If this were about peer review and scientific accuracy, I would want to allow that system to continue to work, but still be able to hear minority views, particularly as my background knowledge of the science deepens. Then I fear incorrect inferences (and even incorrect data) a bit less. I still prefer scientific research to be as correct as possible, but scientific research is not what you folks do. You folks do shallow dives into various topics and offer lots of opinions. Once in a while there’s some serious research but it’s not peer-reviewed.
You referred to AI gaming the system, etc, and voting rings or citation rings, or whathaveyou. It all sounds bad, and there should be ways of screening out such things, but I don’t think the problem should be handled with a trust system.
An even stronger trust system that will just soft-censor some people or some topics more effectively. You folks have a low tolerance for being filters on your own behalf, I’ve noticed. You continue to rely on systems, like karma or your self-reported epistemic statuses, to try to qualify content before you’ve read it. You absolutely indulge false reasons to reject content out of hand. You must be very busy and so you make that systematic mistake.
Implementing an even stronger trust system will just make you folks even more marginalized in some areas, since EA folks are mistaken in a number of ways. With respect to studies of inference methods, forecasting, and climate change, for example, the posting majority’s view here appears to be wrong.
I think it’s baffling that anyone would ever risk a voting system for deciding the importance of controversial topics open to argument. I can see voting working on Stack Overflow, where answers are easy to test, and give “yes, works well” or “no, doesn’t work well” feedback about, at least in the software sections. There, expertise does filter up via the voting system.
Implementing a more reliable trust system here will just make you folks more insular from folks like me. I’m aware that you mostly ignore me. Well, I develop knowledge for myself by using you folks as a silent, absorbing, disinterested, sounding board. However, if I do post or comment, I offer my best. I suppose you have no way to recognize that though.
I’ve read a lot of funky stuff from well-intentioned people, and I’m usually ok with it. It’s not my job, but there’s usually something to gain from reading weird things even if I continue to disagree with it’s content. At the very least, I develop pattern recognition useful to better understand and disagree with arguments: false premises, bogus inferences, poor information-gathering, unreliable sources, etc, etc. A trust system will deprive you folks of experiencing your own reading that way.
What is in fact a feature must seem like a bug:
“Hey, this thing I’m reading doesn’t fit what I’d like to read and I don’t agree with it. It is probably wrong! How can I filter this out so I never read it again. Can my friends help me avoid such things in future?”
Such an approach is good for conversation. Conversation is about what people find entertaining and reaffirming to discuss, and it does involve developing trust. If that’s what this forum should be about, your stronger trust system will fragment it into tiny conversations, like a party in a big house with different rooms for every little group. Going from room to room would be hard,though. A person like me could adapt by simply offering affirmations and polite questions, and develop an excellent model of every way that you’re mistaken, without ever offering any correction or alternative point of view, all while you trust that I think just like you. That would have actually served me very well in the last several months. So, hey, I have changed my mind. Go ahead. Use your trust system. I’ll adapt.
Or ignore you ignoring me. I suppose that’s my alternative.