I haven’t read the whole thing. But this seems to be one of the, if not the coolest idea in EA in 2018. Glad you did it.
Good luck for everyone who goes to live or work there!
Diego_Caleiro
Karma: 206
It has been about 3 years, and only very specific talent still matters for EA now. Earning to Give to institutions is gone, only giving to individuals still makes sense.
It is possible that there will be full scale repleaceability of non-researchers in EA related fields by 2020.
But only if, until then, we keep doing things!
Kaj, I tend to promote your stuff a fair amount to end the inferential silence, and it goes without saying that I agree with all else you said.
Don’t give up on your ideas or approach. I am dispirited that there are so few people thinking like you do out there.
It’s been less than two years and all the gaps have either been closed, or been kept open in purpose, which Ben Hoffman has been staunchly criticising.
But anyway, it has been less than 2 years and Open Phil has way more money than it knows what to do with.
QED.
Amanda Askell has interesting thoughts suggestive of using “care” to have a counterfactual meaning. She suggests we think of care as what you would have cared about if you were in a context such that this was a thing you could potentially change. In a way, the distinction is between people who think about “care” in terms of rank “oh, that isn’t the thing I most care about” and those who care in terms of absolutes “oh, I think the moral value of this is positive.” further complicated by the fact some people are thinking in expected value of action and others are thinking absolute value of the object the action affects.
Semantically, if we think it is a good idea to “expand our circle of care” we should probably adopt “care” to mean the counterfactual meaning, as that broadens the scope of things we can truthfully claim to care about.
Also related, on facebook: https://www.facebook.com/yudkowsky/posts/10151665252179228
They need not imply, but I would like a framework where they do under ideal circumstances. In that framework—which I paraphrase from Lewis—if I know a certain moral fact, e.g., that something is one of my fundamental values, then I will value it (this wouldn’t obtain if you are a hypocrite, in such case it wouldn’t be knowledge).
I should X = A/The moral function connects my potential actions to set X. I think I should X = The convolution of the moral function and my prudential function take my potential actions to set X.
I’m unsure I got your notation. =/= means different? yes What is the meaning of “/” in “A/The…”? same as person/persons, it means either.
In what sense do you mean psychopathy? I can see ways in which I would agree with you, and ways in which not.
I mean failure to exercise moral reasoning. You would be right about what you value, you would desire as you desire to desire, have all the relevant beliefs right, have no conflicting desires or values, but you would not act to serve your desires according to your beliefs. In your instance things would be more complicated given that it involves knowing a negation. Perhaps we can go about like this. You would be right maximizing welfare is not your fundamental value, you would have the motivation to stop solely desiring to desire welfare, you would cease to desire welfare, there would be no other desire inducing a desire on welfare, there would be no other value inducing desire on welfare, but you would fail to pursue what serves your desire. This fits well with the empirical fact psychopaths have low-IQ and low levels of achievement. Personally, I would bet your problem is more with allowing to have moral akrasia with the excuse of moral uncertainty.
I don’t think you carved reality at the joints here, let me do the heavy lifting: The distinction between our paradigms seems to be that I am using weightings for values and you are using binaries. Either you deem something a moral value of mine or not. I however think I have 100% of my future actions left to do, how do I allocate my future resources towards what I value. Part of it will be dedicated to moral goods, and other parts won’t. So I do think I have moral values which I’ll pay high opportunity cost for, I just don’t find them to take a load as large as the personal values, which happen to include actually implementing some sort of Max(Worldwide Welfare) up to a Brownian distance from what is maximally good. My point, overall is that the moral uncertainty is only part of the problem. The big problem is the amoral uncertainty, which contains the moral uncertainty as a subset.
Why just minds? What determines the moral circle? Why does the core need to be excluded from morality? I claim these are worthwhile questions.
Just minds because most of the value seems to lie in mental states, the core is excluded from morality by definition of morality. My immediate one second self, when thinking only about itself of having an experience simply is not a participant of the moral debate. There needs to be some possibility of reflection or debate for there to be morality, it’s a minimum complexity requirement (which by the way makes my Complexity value seem more reasonable).
If this is true, maximizing welfare cannot be the fundamental value because there is not anything that can and is epistemically accessible.
Approximate maximization under a penalty of distance from the maximally best outcome, and let your other values drift within that constraint/attractor.
Do you just mean VNM axioms? It seems to me that at least token commensurability certainly obtains. Type commensurability quite likely obtains. The problem is that people want the commensurability ratio to be linear on measure, which I see no justification for.
It is certainly true of VNM, I think it is true of a lot more of what we mean by rationality. Not sure I understood your token/type token, but it seems to me that token commensurability can only obtain if there is only one type. It does not matter if it is linear, exponential or whatever, if there is a common measure it would mean this measure is the fundamental value. It might also be that the function is not continuous, which would mean rationality has a few black spots (or that value monism has, which I claim are the same thing).
I was referring to the trivial case where the states of the world are actually better or worse in the way they are (token identity) and where another world, if it has the same properties this one has (type identity) the moral rankings would also be the same.
About black spots in value monism, it seems that dealing with infinities leads to paradoxes. I’m unaware of what else would be in this class.
I know a lot of reasonable philosophers that are not utilitarians, most of them are not mainstream utilitarians. I also believe the far future (e.g. Nick Beckstead) or future generations (e.g. Samuel Scheffler) is a more general concern than welfare monism, and that many utilitarians do not share this concern (I’m certain to know a few). I believe if you are more certain about the value of the future than about welfare being the single value, you ought to expand your horizons beyond utilitarianism. It would be hard to provide another Williams regarding convincingness, but you will find an abundance of all sort of reasonable non-utilitarian proposals. I already mentioned Jonathan Dancy (e.g. http://media.philosophy.ox.ac.uk/moral/TT15_JD.mp4), my Nozick’s Cube, value pluralism and so on. Obviously, it is not recommendable to let these matters depend on being pointed.
My understanding is that by valuing complexity and identity in addition to happiness I already am professing to be a moral pluralist. It also seems that I have boundary condition shadows, where the moral value of extremely small values of these things are undefined, in the same way that a color is undefined without tone, saturation and hue.
I find the idea that there are valid reasons to act that are not moral reasons weird; I think some folks call them prudential reasons. It seems that your reason to be an EA is a moral reason if utilitarianism is right, and “just a reason” if it isn’t. But if not what is your reason for doing it?
My understanding of prudential reasons is that they are reasons of the same class as those I have to want to live when someone points a gun at me. They are reasons that relate me to my own preferences and survival, not as a recipient of the utilitarian good, but as the thing that I want. They are more like my desire for a back massage than like my desire for a better world. A function from my actions to my reasons to act would be partially a moral function, partially a prudential function.
If you are not acting like you think you should after having complete information and moral knowledge, perfect motivation and reasoning capacity, then it does not seem like you are acting on prudential reasons, it seems you are being unreasonable.
Appearances deceive here because “that I should X” does not imply “that I think I should X”. I agree that if both I should X and I think I should X, then by doing Y=/=X I’m just being unreasoable. But I deny that mere knowledge that I should X implies that I think I should X. I translate
I should X = A/The moral function connects my potential actions to set X. I think I should X = The convolution of the moral function and my prudential function take my potential actions to set X.
In your desert scenario, I think I should(convolution) defend my self, though I know I should (morality) not
Hence, even if impersonal reasons are all the moral reasons there are, insofar as there are impersonal reasons for people to have personal reasons these latter are moral reasons.
We are in disagreement. My understanding is that the four quadrants can be empty or full. There can be impartial reasons for personal reasons, personal reasons for impartial reasons, impartial reasons for impartial reasons and personal reasons for personal reasons. Of course not all people will share personal reasons, and depending on which moral theory is correct, there may well be distinctions in impersonal reasons as well.
Being an EA while fully knowing maximizing welfare is not the right thing to do seems like an instance of psychopathy (in the odd case EA is only about maximizing welfare). Of course, besides these two pathologies, you might have some form of cognitive dissonance or other accidental failures.
In what sense do you mean psychopathy? I can see ways in which I would agree with you, and ways in which not.
Perhaps you are not really that sure maximizing welfare is not the right thing to do.
Most of my probability mass is that maximizing welfare is not the right thing to do, but maximizing a combination of identity, complexity and welfare is.
I prefer this solution of sophisticating the way moral reasons behave than to claim that there are valid reasons to act that are not moral reasons; the latter looks, even more than the former, like shielding the system of morality from the real world. If there are objective moral truths, they better have something to do with what people want to want to do upon reflection.
One possibility is that morality is a function from person time slices to a set of person time slices, and the size to which you expand your moral circle is not determined a priori. This would entail that my reasons to act morally only when considering time slices that have personal identity 60%+ with me would look a lot like prudential reasons, whereas my reasons to act morally accounting for all time slices of minds in this quantum branch and its descendants would be very distinct. The root theory would be this function.
The right thing to do will always be an open question, and all moral reasoning can do is recommend certain actions over others, never to require. If there is more than one fundamental value, or if this one fundamental value is epistemically inaccessible, I see no other way out besides this solution.
Seems plausible to me.
Incommensurable fundamental values are incompatible with pure rationality in its classical form.
Do you just mean VNM axioms? It seems to me that at least token commensurability certainly obtains. Type commensurability quite likely obtains. The problem is that people want the commensurability ratio to be linear on measure, which I see no justification for.
It seems to me Williams made his point; or the point I wished him to make to you. You are saying “if this is morality, I reject it”. Good. Let’s look for one you can accept.
I would look for one I can accept if I was given sufficient (convoluted) reasons to do so. At the moment it seems to me that all reasonable people are either some type of utilitarian in practice, or are called Bernard Williams. While I don’t get pointed thrice to another piece that may overwhelm the sentiment I was left with, I see no reason to enter exploration stage. For the time being, the EA in me is peace.
That.
I really appreciate your point about intersubjective tractability. It enters the question of how much should we let empirical and practical considerations spill into our moral preferences (ought implies can for example, does it also imply “in a not extremely hard to coordinate way”?)
At large I’d say that you are talking about how to be an agenty Moral agent. I’m not sure morality requires being agenty, but it certainly benefits from it.
Bias dedication intensity: I meant something ortogonal to optimality. Dedicating only to moral preferences, but more to some that actually don’t have that great of a standing, and less to others which normally do the heavy lifting (don’t you love when philosophers talk about this “heavy lifting”?). So doing it non-optimally.
Suggestion: Let people talk about any accomplishments, without special emphasis on the month level, or the name of the month.
Some of the moments when people most need to brag is when they need to recover a sense of identity with a self that is more than a month old, that did awesome stuff.
Example: Once upon a time 12 years ago I thought the most good I could do was fixing aging, so I found Aubrey, worked for them for a bit, and won a prize!
A thing I’m proud off is that a few days ago I gave an impromptu speech at Sproul Hall (where free speech started) at Berkeley, about technological improvement and EA, and several people came after to thank me for it.
I frequently use surnames, but in this case since it was a call to action of sorts, first names seemed more appropriate. Thanks for the feedback though, makes sense.
Agreed with 2 first paragraphs.
Activities that are more moral than EA for me: At the moment I think working directly on assembling and conveying knowledge in philosophy and psychology to the AI safety community has higher expected value. I’m taking the AI human compatible course at Berkeley, with Stuart Russell, I hang out at MIRI a lot, so in theory I’m in good position to do that research and some of the time I work on it. But I don’t work on it all the time, I would if I got funding for our proposal.
But actually I was referring to a counterfactual world where EA activities are less aligned with what I see as morally right than this world. There’s a dimension, call it “skepticism about utilitarianism” that reading Bernard Williams made me move along. If I moved more and more along that dimension, I’d still do EA activities, that’s all.
Your expectation is partially correct, I assign 3% to EA activities is morally required of everyone, I feel personally more required to do them than 25% (because this is the dream time, I was lucky, I’m at a high leverage position etc..), but although I think it is right for me to do them, I don’t do them because its right, and that’s my overall point.
Telofy: Trying to figure out the direction of the inferential gap here. Let me try to explain, I don’t promise to succeed.
Aggregative consequentialist utilitarianism holds that people in general should value most minds having the times of their lives, where “in general” here actually translated into a “should” operator. A moral operator. There’s a distinction between me wanting X, and morality suggesting, requiring, or demanding X. Even if X is the same, different things can hold a relation to it.
At the moment I both hold a personal preference relation to you having a great time as I do a moral one. But if the moral one was dropped (as Williams makes me drop sevral of my moral reasons) I’d still have the personal one, and it supersedes the moral considerations that could arise otherwise.
Moral Uncertainty: To confess, that was my bad not disentangling uncertainty about my preferences that happen to be moral, my preferences that happen to coincide with preferences that are moral, and the preferences that morality would, say, require me to have. That was bad philosophy and on my part and I can see Lewis, Chalmers and Muelhauser blushing at my failure.
I meant uncertainty I have as an empirical subject in determining which of the reasons for argument I find are moral reasons or not, and within that which belong to which moral perspective. For instance I assign high credence that breaking a promise is bad from a Kantian standpoint, times a low credence that Kant was right about what is right. So not breaking a promise has a few votes in my parliament, but not nearly as many as giving a speech about EA at UC Berkeley has, because I’m confident that a virtuous person would do that, and I’m somewhat confident it is good from a utilitarian standpoint too, so lots of votes.
I disagree that optimally satifying your moral preferences equals doing what is moral. For one thing you are not aware of all moral preferences that, on reflection you would agree with, for another, you could bias your dedication intensity in a way that even though you are acting on moral preferences, the outcome is not what is moral all things considered. Furthermore It is not obvious to me that a human is compelled necessarily to have all moral preferences that are “given” to them. You can flat out reject 3 preferences, act on all others, and in virtue of your moral gap, you would not be doing what is moral, even though you are satisfying all preferences in your moral preference class.
Nino: I’m not sure where I stand on moral realism (leaning against but weakly). The non-moral realist part of me replies:
wouldn’t {move the universe into a state I prefer} and {act morally} necessarily be the same because you could define your own moral values to match your preferences?
Definitely not the same. First of all to participate in the moral discussion, there is some element of intersubjectivity that kicks in, which outright excludes defining my moral values to a priori equate my preferences, they may a posteriori do so, but the part where they are moral values involves clashing them against something, be it someone else, a society, your future self, a state of pain, or, in the case of moral realism, the moral reality out there.
To argue that my moral values equate all my preferences would be equivalent to universal ethical preference egoism, the hilarious position which holds that the morally right thing to do is for everyone to satisfy my preferences, which would tile the universe with whiteboards, geniuses, ecstatic dance, cuddlepiles, orgasmium, freckles, and the feeling of water in your belly when bodysurfing a warm wave at 3pm, among other things. I don’t see a problem with that, but I suppose you do, and that is why it is not moral.
If morality is intersubjective, there is discussion to be had. If it is fully subjective, you still need to determine in which way it is subjective, what a subject is, which operations transfer moral content between subjects if any, what legitimizes you telling me that my morality is subjective, and finally why call it morality at all if you are just talking about subjective preferences.
It seems that you feel the moral obligation strongly from your comment. Like the Oxford student cited by Krishna you don’t want to do what you want to do, you want to do what you oughtto do.
I don’t experience that feeling, so let me reply to your questions:
Wouldn’t virtue ethics winning be contradicted by your pulling the lever?
Not really, the pulling of the lever is what I would do, it is what I would think I have reason to do, but it is not what I think I would have moral reason to do. I would reason that a virtuous person (ex hypothesi) wouldn’t kill someone, that the moral thing to do is let the lever be. Then I would act on my preference that is stronger than my preference that the moral thing be done. The only case where a contradiction would arise is if you subscribe to all reasons for action being moral reasons, or moral reasons having the ultimate call in all action choice. I don’t.
In the same spirit, you suggest I’m an ethical egoist. This is because when you simulated me in this lever conflict, you think “morality comes first” so you dropped the altruism requirement to make my beliefs compatible with my action. When I reason however I think “morality is one of the things I should consider here” and it doesn’t win over my preference for most minds having an exulting time. So I go with my preference even when it is against morality. This is orthogonal to Ethical Egoism, a position that I consider both despicable and naïve, to be frank. (Naïve because I know the subagents with whom I have personal identity care for themselves about more than just happiness or their preference satisfaction, and despicable because it is one thing to be a selfish prick, understandable in an unfair universe into which we are thrown into a finite life with no given meaning or sensible narrative, it is another thing to advocate a moral position in which you want everyone to be a selfish prick, and to believe that being a selfish prick is the right thing to do, that I find preposterous at a non-philosophical level.)
If you were truly convinced that something else were morally right, you would do it. Why wouldn’t you?
Because I don’t always do what I should do. In fact I nearly never do what is morally best. I try hard to not stay too far from the target, but I flinch from staring into the void almost as much as the average EA Joe. I really prefer knowing what the moral thing to do is in a situation, it is very informative and helpful to assess what I in fact will do, but it is not compelling above and beyond the other contextual considerations at hand. A practical necessity, a failure of reasoning, a little momentary selfishness, and an appreciation for aesthetic values have all been known to cause me to act for non-moral reasons at times. And of course I often did what I should do too. I often acted the moral way.
To reaffirm, we disagree on what Ethical Egoism means. I take it to be the position that individuals in general ought to be egoists (say, some of the time). You seem to be saying that , and furthermore that if I use any egoistic reason to justify my action, then merely in virtue of my using it as justification I mean that everyone should be (permitted to) doing the same. That makes sense if your conception of just-ice is contractualist and you were assuming just-ification has a strong connection to just-ice. From me to me, I take it to be a justification (between my selves perhaps), but from me to you, you could take it as an explanation of my behavior, to avoid the implications you assign to the concept of justification as demanding the choice for ethical egoism.
I’m not sure what my ethical (meta-ethical) position is, but I am pretty certain it isn’t, even in part, ethical egoism.
I’m not claiming this is optimal, but I might be claiming what I’m about to say may be more optimal than anything else that 98% of EAs are actually doing.
There are a couple thousand billionaires on the planet. There are also about as many EAs.
Let’s say 500 billionaires are EA friendly under some set of conditions. Then it may well be that the best use of the top 500 EAs is to minutiously study single individual billionaires. Understand their values, where they come from, what makes them tick. Draw their CT-chart, find out their attachment style, personality disorder, and childhood nostalgia. Then approach them to help them, and while solving many of their problems faster than they can even see it, also show them the great chance they have of helping the world.
Ready set go: http://www.forbes.com/billionaires/list/
As Luke and Nate would tell you, the shift from researcher to CEO is a hard one to make, even when you want to do good, as Hanson puts it “Yes, thinking is indeed more fun.”
I have directed an institute in Brazil before, and that was already somewhat a burden.
The main reason for the high variance though is that setting up an institute requires substantial funding. The people most likely to fundraise would be me, Stephen Frey (who is not on the website), and Daniel, and fundraising is taxing in many ways. Would be great if we had for instance the REG fundraisers to aid us (Liv, Ruari, Igor, wink wink) either by fundraising for us, or finding someone to, or teaching us to.
Money speaks. And it spells high variance.
1) Convergence Analysis: The idea here is to create a Berkeley affiliated research institute that operates mainly in two fronts 1)Strategy on the long term future 2)Finding Crucial Considerations that have not been considered or researched yet. We have an interesting group of academics and I would take a mixed position of CEO and researcher.
2) Altruism: past, present, propagation: this is a book whose table of contents I already wrote, and would need further research and spelling out each of the 250 sections I have in mind. It is very different in nature from Will’s book, or Singer’s book. The idea here is not to introduce to EA, but to reason about the history of cooperation and altruism that led to us, and where this can be taken in the future, inclusive by the EA movement. This would be major intellectual undertaking, likely consuming my next three years and doubling as a PHD dissertation. Perhaps, tripling as a series of blog posts, for quick feedback loops and reliable writer motivation.
3) FLI grant proposal: Our proposal intended to increase our understanding psychological theories of human morality in order to facilitate later work in formalizing moral cognition to AIs, a subset of the value loading and control problems of Artificial Generalized Intelligence. We didn’t win, so the plan here would be to try to find other funding sources for this research.
4) Accelerate the PHD: For that I need to do 3 field statements, one about the control problem in AI with Stuart, one about altruism with Deacon, and one to be determined, then only the dissertation would be still on the to do list.
All these plans scored sufficiently high in my calculations that it is hard to decide between them. Accelerating the PHD has a major disadvantage because it does not increase my funding. The book (via blog posts or not) has a strong advantage in that I think it will have sufficiently new material that it satisfies goal 1 best of all, it is probably the best for the world if I manage to get to the end of it and do it well. But again, it doesn’t increase funding. Convergence has the advantage of co-working with very smart people, and if it takes off sufficiently well, it could solve the problem of continuing to live in Berkeley and that of financial constraints all at once, putting me in a stable position to continue doing research in relevant topics almost indeterminately, instead of having to make ends meet by downsizing the EA goal substantially among my priorities. So very high stakes, but uncertain probabilities. If AI is (nearly) all that matters, then the FLI grant will be the highest impact, followed by Convergence, the book and the acceleration.
In any event all of those are incredible opportunities which I feel lucky to even have in my consideration space. It is a privilege to be making that choice, but it is also very hard. So conditional on the goals I stated before: 1)Making the world better by the most effective means possible. 2)Continuing to live in Berkeley 3)Receive more funding 4)Not stop PHD 5)Use my knowledge and background to do (1).
I am looking for some light, some perspective from the outside that will make me lean one way or another. I have been uncomfortably indecisive for months, and maybe your analysis can help.
Ok, so this doubles as an open thread?
I would like some light from the EA hivemind. For a while now I have been mostly undecided about what to do with my 2016-2017 period.
Roxanne and I even created a spreadsheet so I could evaluate my potential projects and drop most of them, mid-2015. My goals are basically an oscillating mixture of
1)Making the world better by the most effective means possible.
2)Continuing to live in Berkeley
3)Receive more funding
4)Not stop PHD
5)Use my knowledge and background to do (1).
This has proven an extremely hard decision to make. Here are the things I dropped because they were incompatible with time, or goals other than 1, but still think other EAs, who share goal 1, should carry on:
(1) Moral Economics: From when it started, Moral Econ is an attempt to install a different mindset in individuals, my goal has always been to have other people pick it up and take it forwards. I currently expect this to be done, and will go back to it only if it seems like it will fall apart.
(2) Effective Giving Pledge: This is a simple idea I applied to EA ventures with, though I actually want someone else to do it. The idea is simply to copy the Gates giving pledge website for an Effective Giving Pledge, which says that the wealthy benefactors will donate according to impact, tractability and neglectedness. If 3 or 4 signatories of the original pledge signed it, it would be the biggest shift in resource allocation from the non EA-money pool to the EA-money pool in history.
(3) Stuart Russell AI-safety course: I was going to spend some time helping Stuart to make an official Berkeley AI-safety course. His book is used in 1500+ Universities, so the if the trend caught, this would be a substantial win for the AI safety community. There was a non-credit course offered last semester in which some MIRI researchers, Katja, Paul, me and others were going to present. However it was very poorly attended and was not official, and it seems to me that the relevant metric is probability that this would become a trend.
(4) X-risk dominant paper: What are the things that would dominate our priority space on top of X-risk if they were true? Me and Daniel Kokotajlo began examining that question, but considered it to be too socially costly to publish anything about it, since many scenarios are too weird and could put off non-philosophers.
These are the things I dropped for reasons other than the EA goal 1. If you are interested in carrying on any of them, let me know and I’ll help you if I can.
In the comment below, by contrast are the things between which I am still undecided the ones I want help in deciding:
This is a little old, but it’s a similar concept with far higher level of investment: https://www.lesswrong.com/posts/GfRKvER8PWcMj6bbM/sidekick-matchmaking