Some simple observations.
To perform such a QALY estimate you need
A credible model for predicting the consequences of possible responses
An estimate of how likely your advocacy is to effect policy
1 is something you need to even know what the best response it (and I’m currently not sure whether you have it).
2 sounds like something that should have been researched by many people by now, but I’m far from an expert so no specific suggestions.
I think that most people here will tell you that we already know specific examples of such wrongdoing e.g. factory farming.
I have some evidence that there are many software engineers who would gladly volunteer to code for EA causes (and some access to such engineers). What volunteering opportunities like that are available? EA organizations that need coders? Open source projects that can be classified as EA causes? Anything else?
What do you mean by “accept what it really means to be Human”? To what end is it “more productive”?
Not any “human” thing is a good thing. Being susceptible to disease, old age and death is part of “being human” but it is a part I would rather part with. On the other hand, being Human also means having the ability to use reason to find a better strategy than the one suggested by the initial emotional response.
A rational thinker should factor the limitations of their own brain into their decision making. Also, sometimes we do care about some people more than about other people (e.g. friends and family). However, certain behaviors are simply bugs (in more scientific language, cognitive biases). There is no rational reason to “accept” them if we can find a way to work around them.
This strikes me as a strange choice of words since e.g. I think it is good to occasionally experience sadness. But arguing over words is not very fruitful.
I’m not sure this interpretation is consistent with “filling the universe with tiny beings whose minds are specifically geared toward feeling as much pure happiness as possible.”
First, “pure happiness” sounds like a raw pleasure signal rather than “things conscious beings experience that are good” but ok, maybe it’s just about wording.
Second, “specifically geared” sounds like wireheading. That is, it sounds like these beings would be happy even if they witnessed the holocaust which again contradicts my understanding of “things conscious beings experience that are good.” However I guess it’s possible to read it charitably (from my perspective) as minds that have superior ability to have truly valuable experiences i.e. some kind of post-humans.
Third, “tiny beings” sounds like some kind of primitive minds rather than superhuman minds as I would expect. But maybe you actually mean physical size in which case I might agree: it seems much more efficient to do something like running lots of post-humans on computronium than allocating for each the material resources of a modern biological human (although at the moment I have no idea what volume of computronium is optimal for running a single post-human: on the one hand, running a modern-like human is probably possible in a very small volume, on the other hand a post-human might be much more computationally expensive).
So, for a sufficiently charitable (from my perspective) reading I agree, but I’m not sure to which extent this reading is aligned with your actual intentions.
Upvoted, because although I disagree with much of this on object level, I think the post is totally legit and I think we should encourage original thinking.
Perhaps we need to find a time and place to start a serious discussion of ethics. I think hedonistic utilitarianism is wrong already on the level of meta-ethics. It seems to assume the existence of universal morals which from my point of view is a non-sequitur. Basically all discussions of universal morals are games with meaningless words, maps of non-existing territories. The only sensible meta-ethics I know is equating ethics with preferences. It seems that there is such a thing as intelligent agents with preferences (although we have no satisfactory mathematical definition yet). Of course each agent has its own preferences and the space of possible preferences is quite big (orthogonality thesis). Hence ethical subjectivism. Human preferences don’t seem to differ much from human to human once you take into account that much of the differences in instrumental goals are explained by different beliefs rather than different terminal goals (=preferences). Therefore it makes sense in certain situations to use approximate models of ethics that don’t explicitly mention the reference human, like utilitarianism. On the other hand, there is no reason the precise ethics should have a simple description (complexity of value). It is a philosophical error to think ethics should be low complexity like physical law since ethics (=preferences) is a property of the agent and has quite a bit of complexity put in by evolution. In other words, ethics is in the same category as the shape of Africa rather than Einstein’s equations. Taking simplified models which take only one value into account (e.g. pleasure) to the extreme is bound to lead to abhorrent conclusions as all other values as sacrificed.
I agree with Tom. I think the core values of EA have to include:
Always keep looking for new creative ways to do better.
Maintain an open, honest and respectful discussion with your peers.
In particular exploring new interventions and causes should always be in the EA spotlight. When you think something is an effective charity but most EAs wouldn’t agree with you, in my book it’s a reason to state your case loud and clear rather than self-censor.
“Hold until future orders” is one approach but it might turn out to be much more difficult than actually creating an AI with correct values. This is because the formal specification of metaethics (that is a mathematical procedure that takes humans as input and produces a utility function as output) should be of much lower complexity than specifying what it means to “protect from other AI but do nothing else.”
I completely agree that many conceivable post-human future have low value. See also “unhumanity” scenario in my analysis. I think that term “existential risk” might be somewhat misleading since what we’re really aiming it as “existence of beings and experiences that we value” rather than just existence of “something.” That is, I view your reasoning not as an argument for caring less about existential risk but as an argument for working towards a valuable far future.
Regarding MIRI, I think their position is completely adequate since once we create a singleton which endorses our values it will guard us from all sorts of bad futures, not only from extinction.
Regarding “consciousness as similarity”, I think it’s a useful heuristic but it’s not necessarily universally applicable. I consider certain futures in which I gradually evolve into something much more complex than my current self as positive, but one must be very careful about which trajectories to endorse. Building an FAI will save us from doing irreversible mistakes, but if for some reason constructing a singleton turns out to be intractable we will have to think of other solutions.
Interesting. One way to solve the replaceability problem is to force the government to announce a preliminary budget before the ear-marking “bids” and pledge to treat the bids as differentials with respect to the preliminary budget.
The question is what is the mechanism of value spreading.
If the mechanism is having rational discussions then it is not necessarily urgent to have these discussions right now. Once we create a future in which there is no death and no economic pressures to self-modify in ways that are value destructive, we’ll have plenty of time for rational discussions. Things like “experience machine” also fit into this framework, as long as the experiences are in some sense non-destructive (this rules out experiences that create addiction, for example).
If the mechanism is anything but rational discussion then
It’s not clear in what sense the values you’re spreading are “correct” if it’s impossible to convince other people through rational discussion.
I would definitely consider this sort of intervention as evil and would fight rather than cooperate with it (at least assuming the effect cannot be reversed by rational discussion; I also consider hedonistic utilitarianism abhorrent except as an approximate model in very restricted contexts).
Regarding MIRI in particular, I don’t think the result of their work depends on the personal opinions of its director in the way you suggest. I think that any reasonable solution to the FAI problem will be on the meta-level (defining what does it mean for values to be “correct”) rather than the object level (hard-coding specific values like animal suffering).
I think the main comparative advantage (= irreplaceability) of the typical EA comes not from superior technical skill but from motivation to improve the world (rather than make money, advance one’s career or feel happy). This means researching questions which are ethically important but not grant-sexy, donating to charities which are high-impact but don’t yield a lot of warm-fuzzies, promoting policy which goes against tribal canon etc.
In the Arab Spring many of the revolutionary groups were radical Islamists rather than champions of liberal democracy. Also, I didn’t say anything about revolution: in some cases a gradual transition is more likely to work.
Infiltrating an organization you hate while preserving sanity and your true values is a task few people are capable of. I’m quite certain I wouldn’t make it.
I think that we need serious research + talking to people from the relevant countries to devise realistic strategies.
Do you think Soviet attempts to foster communism in the US during the cold war were a stabilising influence?
Well, they might have been stabilizing if they worked :) Although I think war between communist countries is much more likely than war between liberal democracies.
Countries generally and rightfully take affront at foreigners trying to meddle with their internal affairs.
I mostly agree with the descriptive claim but not with the normative claim. Why “rightfully”?
For a more recent example, look at the aftermath of the western coup in Ukraine.
“Western” coup? The revolutionaries were pro-Western to some extent, but why is it a good example of foreign meddling?
I agree that backfiring is a serious risk of such interventions but I don’t think we should write them off completely. Moreover, interventions by private organizations, especially private organization whose support base is spread over many countries, seem much less likely to precipitate a diplomatic crisis than direct interventions by governments.
I’m not sure what the EA movement can do that will have significant effect in the short term. In the long term we should be looking into establishing liberal democracy in countries which either posses nuclear weapons or have the capacity to develop them in the near future (Russia, China, North Korea, Pakistan, Iran...). For example we can support the pro-liberalisation groups which already exist in these countries.
Hi Tom, thx for commenting!
For me, the meta-point that we should focus on steering into better scenarios was a more important goal of the post than explaining the actual scenarios. The latter serve more as examples / food for thought.
Regarding objections to Utopian scenarios, I can try to address them if you state the objections you have in mind. :)
Regarding dictatorships, I indeed focused on situations that are long-term stable since I’m discussing long-term scenarios. A global dictatorship with existing technology might be possible but I find it hard to believe it can survive for more than a couple of thousand years.
If your only requirement is for all sentient beings to be happy, you should be satisfied with a universe completely devoid of sentient beings. However, I suspect you wouldn’t be (?)
Regarding definition of good, it’s pointless to argue about definitions. We should only make sure both of us know what each word we use means. So, let’s define “koodness(X)” to mean “the extent to which things X wants to happen actually happen” and “gudness” to mean “the extent to which what is happening to all beings is what they want to happen” (although the latter notion requires clarifications: how do we average between the beings? do we take non-existing beings into account? how do we define “happening to X”?)
So, by definition of kood, I want the future world to be kood(Squark). I also want the future world to be gud among other things (that is, gudness is a component of koodness(Squark)).
I disagree with Mill. It is probably better for a human being not become a pig, in the sense that a human being prefers not becoming a pig. However, I’m not at all convinced a pig prefers to become a human being. Certainly, I wouldn’t want to become a “Super-Droid” if it comes at a cost of losing my essential human qualities.
My distribution isn’t tight, I’m just saying there is a significant probability of large serial depth. You are right that much of the benefit of current work is “instrumental”: interesting results will convince other people to join the effort.
Hi Uri, thanks for the thoughtful reply!
It is not necessarily bad for future sentients to be different. However, it is bad for them to be devoid of properties that make humans morally valuable (love, friendship, compassion, humor, curiosity, appreciation of beauty...). The only definition of “good” that makes sense to me is “things I want to happen” and I definitely don’t want a universe empty of love. A random UFAI is likely to have none of the above properties.
I think downvoting as disagreement is terrible.
First, promoting content based on majority agreement is a great way to build an echo chamber. We should promote content which is high-quality (well written, well argumented, thought-provoking, contains novel insights, provides a balanced perspective etc.). Hearing repetitions of what you already believe just amplifies your confirmation bias. I want to learn something new.
Second, downvoting creates a strong negative incentive against posting. Silencing people you disagree with is also a great way to build an echo chamber.
Third, downvoting based on disagreement creates a battle atmosphere. Instead of a platform for rational, well-meaning debate we risk turning into a scuffle between factions with different ideologies.
All in all I think the rules for downvoting posts should be slightly more lax than for downvoting comments. Downvoting a low-quality post is acceptable (but be very cautious before deciding something you disagree with is “low-quality”). Downvoting a comment is only acceptable when the comment is not in good faith (spam, trolling, flaming etc.). I think this is essential to maintain a healthy amicable atmosphere.