I’m working on impact markets – markets to trade nonexcludable goods. (My profile.)
I have a conversation menu and a Calendly for you to pick from!
If you’re also interested in less directly optimific things – such as climbing around and on top of boulders or amateurish musings on psychology – then you may enjoy some of the posts I don’t cross-post from my blog, Impartial Priorities.
Pronouns: Ideally she or they. I also still go by Denis and Telofy in various venues.
Thanks for writing all of this up in one place!
One of my gripes with the community has long been that maximization is core to EA and we’re still really clueless about what it implies and most of the community (outside RP, QURI, etc. and some researchers) seems to have given up on figuring it out.
I feel like we’re like this one computer science professor I had who seemed a bit senile and only taught the sort of things that haven’t lost relevance in the last 30 years because he hadn’t kept up with anything that happened since the 80s. He probably had good personal and neurological reasons for that, but we don’t, right?
I haven’t read any EA introductory materials in a few years, but I hope they contain articles about expected value maximization along with articles on how EV is usually largely unknowable due to cluelessness and often undefined due to Pasadena games. That stochastic dominance is arguably a much better approach to prioritization but that Christian Tarsney is so far more or less the only one who has bothered to look into it. That there is perhaps a way forward to figure out what’s the best thing to do if we funded some big world-modeling efforts based on software like Squiggle but that hardly anyone outside RP and QURI (and Convergence?) currently bothers to do anything about it. (I’ve dabbled a bit in these fields but my personal fit doesn’t seem to be great.)
Maybe there’s even a way to scalably adjust for personal fit whatever recommendations this big model effort might yield. Maybe there are some common archetypes/personas plus quizzes that tell people which ones they are closest to.
Arguably this can wait until this whole AI thing is under control (if that’s even possible), but few people will want to work on AI safety, so maybe it doesn’t have to wait?
My takeaway has been mostly that I don’t have a clue and so will go with some sort of momentary best guess gives me enough fulfillment, enjoyment, and safety. I’ve written more about it here.
That said, EA has had a great effect on my mental health.
I used to be a crying wreck because of all the suffering in the world. I spread myself thin trying to help everyone. I felt guilty about the majority of terrible things in the world that I was powerless to prevent. (Suicidal too, except that would’ve been self-defeating.)
Then EA came along and gave me an excuse to “pick my battles,” i.e. focus on a few things where I could make a big difference, taking into account my skills and temperament. Now, if someone went, “Hey, you should become a politician to prevent X and Y,” I could go, “No, I wouldn’t be good at that and hate every second of it and it would come at a great cost to A, which I’m already doing.” EA, for the first time, allowed me to set boundaries.
EA also gave me an appreciation for the (perhaps, plausibly, who knows really) great absolute impact that I can have despite the minimal impact that I have (perhaps, plausibly, who knows really) relative to the totality of the suffering in the world. That made it much easier to find fulfillment.
The question often comes up how we should make decisions under epistemic uncertainty and normative diversity of opinion. Since I need to make such decisions every day, I had to develop a personal system, however inchoative, to assist me.
My personal system can be thought of like a pyramid.
At the top sits some sort of measurement of success. It’s highly abstract and impractical. Let’s call it the axiology. This is really a collection of all axiologies I relate to, including the amount of frustrated preferences and suffering across our world history. This also deals with hairy questions such as how to weigh Everett branches morally and infinite ethics.
Below that sits a kind of mission statement. Let’s call it the ethical theory. It’s just as abstract, but it is opinionated about the direction in which to push our world history. For example, it may desire a reduction in suffering, but for others this floor needn’t be consequentialist in flavor.
Both of these abstract floors of the pyramid are held up by a mess of principles and heuristics at the ground floor level to guide the actual implementation.
The ground floor of principles and heuristics is really the most interesting part for anyone who has to act in the world, so I won’t further explain the top two floors.
The principles and heuristics should be expected to be messy. That is, I think, because they are by necessity the result of an intersubjective process of negotiation and moral trade (positive-sum compromise) with all the other agents and their preferences. (This should probably include acausal moral trades like Evidential Cooperation in Large Worlds.)
It should also be expected to be messy because these principles and heuristics have to satisfy all sorts of awkward criteria:
They have to inspire cooperation or at least not generate overwhelming opposition.
They have to be easily communicable so people at least don’t misunderstand what you’re trying to achieve and call the police on you. Ideally so people will understand your goal well enough that they want to join you.
They have to be rapidly actionable, sometimes for split second decisions.
They have to be viable under imperfect information.
They have to be psychologically sustainable for a lifetime.
They have to avoid violating laws.
And many more.
But really that leaves us still a lot of freedom (for better or worse):
There are countless things that we can do that are highly impactful and hardly violate anyone’s preferences or expectations.
There are also plenty of things that don’t violate any preferences or expectations once we get to explain them.
Finally, there are many opportunities for positive-sum moral trade.
These suggest a particular stance toward other activists:
If someone is trying to achieve the same thing you’re trying to achieve, maybe you can collaborate.
If someone is trying to achieve something other than what you’re trying to achieve, but you think their goals are valuable, don’t stand in their way. In particular, it may sometimes feel like doing nothing (to further or hinder their cause) is a form of “not standing in their way.” But if your peers are actually collaborating with them to some extent, doing nothing (or collaborating less) can cause others to also reduce their collaboration and can prevent key threshold effects from taking hold. So the true neutral position is to try to understand how much you need to collaborate toward the valuable goal so it would not have been achieved sooner without you. This is usually very cheap to do and has a chance to get runaway threshold effects rolling.
If someone is trying to achieve something that you consider neutral, the above may still apply to some extent because perhaps you can still be friends. And for reasons of Evidential Cooperation in Large Worlds. (Maybe you’ll find that their (to you) neutral thing is easy to achieve here and that other agents like them will collaborate back elsewhere where your goal is easy to achieve.)
Finally, if someone is trying achieve something that you disapprove of… Well, that’s not my metier, temperamentally, but this is where compromise can generate gains from moral trade.
In my experience, principles and heuristics are best identified by chatting with friends and generalizing from their various intuitions.
Charitable donations are total anarchy. Mostly, you can just donate wherever the fluff you want, and (unless you’re Open Phil) no one will throw stones through your windows in retaliation. You can just optimize directly for your goals – except, Evidential Cooperation in Large Worlds will still make strong recommendations here, but what they are is still a bit underexplored.
Even if you’re not an animal welfare activist yourself, you’re still well-advised to cooperate with behavior change to avert animal suffering to the extent expected by your peers. (And certainly to avoiding inventing phony reasons to excuse your violation of these expectations. These might be even more detrimental to moral progress and rationality waterline.)
If you want to spend time with someone but they behave outrageously unempathetically toward you or someone else (e.g., say something like “Your suffering is nothing compared to the suffering of X” to their face), you should rather cut all ties with them even though, strictly speaking, this does not imply that no positive-sum trade is possible with them.
Trying to systematically put people in powerful positions can arouse suspicion and actually make it harder to put people in powerful positions. Trying to systematically put people into the sorts of positions they find fulfilling might put as many people in powerful positions and make their lives easier too. (Or training highly conscientious people in how to dare to accept responsibility so it’s not just those who don’t care who self-select into powerful positions.)
And hundreds more…
Various non-consequentialist ethical theories can come in handy here to generate further useful principles and heuristics. That is probably because they are attempts at generalizing from the intuitions of certain authors, which puts them almost on par (to the extent to which these authors are relateable to you) with generalizations from the intuitions of your friends.
(If you find my writing style hard to read, you can ask Claude to rephrase the message into a style that works for you.)
Hiii! I found this list of “Crucial questions for longtermists” to be quite impressive. It is also listed as part of “A central directory for open research questions,” which is broader than your question.
I met Marisa at EAG London in 2019. We had approximately weekly calls afterwards during the lockdown that I greatly enjoyed. That and all the virtual events helped me connect with the rest of the EA world – probably more so than in-person events. Sadly, I missed one of our calls, which prompted me to set up a comprehensive reminder solution that I use to this day.
When we supported each other again in the context of some job applications a few years later, I learned that she had just survived a very difficult phase of her life. Then, as now, I wish I had known and had been able to support her in some fashion.
When you lose hope in humanity and x-risk reduction seems pointless, she’s the sort of existence proof that keeps you going.
I love this research! Thank you so much for doing it!
My gut reaction to the results is that it’s odd that humans are so high up in terms of their capacity for welfare. Just as an uninformative prior, I would’ve expected us to be somewhere in the middle. Less confidently, I would’ve expected a similar number of orders of magnitude deviation from the human baseline in either direction, within reason. E.g. +/- ~.5 OOM.
Plus, we are humans, so there’s a risk that we’re biased in our favor. It could be simply a bias from our ability to emphasize with other humans. But it could also be the case that there are countless more markers of sentience that humans don’t have (but many other sentient animals do) that we are prone to overlook.
Have you investigated what the sources of this effect might be? There might be any number of biases at work as I mentioned, but perhaps our lives have become so comfy most of the time that we perceive slight problems very strongly (e.g., a disapproving gaze). If then something really bad happens, it feels enormously bad?
(I’ve in the past explicitly assumed that most beings with a few (million) neurons have a roughly human capacity for welfare – not because I thought that was likely but because I couldn’t tell in which direction it was off. Do you maybe already have a defense of the results for people like me?)
In any case, I’ll probably just adopt your results into my thinking now. I don’t expect them to change my priorities much given all the other factors.
Thank you again! <3
Update: When I mentioned this to a friend on a hike, I came up with two ways in which the criteria might be amended to include nonhuman ones: (1) In may cases, we probably have a theory for why a particular behavior or feature is likely to be indicative of conscious experience. Understanding this mechanism, we can look for other systems that might implement the same mechanism, sort of how the eyes of humans, eagles, and flies are very different but we infer that they are probably all for the purpose of vision. (2) Maybe a number of animals that show certain known criteria for consciousness also share suspiciously consistently some other features. One could then investigate whether these features are also indicative of consciousness and whether there are other animals that have these new features at the expense of the older, known ones. (The analysis could cluster features that usually co-occur to not overweight causally related features in cases where many of them are observable.)
Only half a person per sandal I think!
Even scandal-prone individuals can’t survive in a vacuum. (You may be thinking of sandals, not scandals?)
We have sympathies towards both movements, and consider ourselves to take the middle path. We race forward and accelerate as quickly as possible while mentioning safety.
Mentioning safety is a waste of resources that you could direct toward attaching propulsion to asteroids to get them here faster.
In fact, asteroids will inevitably hit earth earlier or later, and if they kill humanity, clearly they are superior to humanity. The true masters of our future lightcone are the asteroids. That which can be destroyed by asteroids ought to be destroyed by asteroids.
True progress is in speeding the inevitable. Resistance is futile.
This post is also a great info hazard. It risks causing impostors with sub-146 IQs (2009 LW survey) to feel adequate!
That’s a good point. Time discounting (a “time premium” for the past) has also made me very concerned about the welfare of Boltzmann brains in the first moments after the Big Bang. It might seem intractable to still improve their welfare, but if there is any nonzero chance, the EV will be overwhelming!
But I also see the value in longtermism because if these Boltzmann brains had positive welfare, it’ll be even more phenomenally positive from the vantage points of our descendants millions of years from now!
There is a Swiss canton Appenzell Innerrhoden (AI). Maybe we can hide there and trick the AI into thinking it already invaded it?
Scandals don’t just happen in the vacuum. You need to create the right conditions for them. So I suggest:
We spread concern about the riskiness of all altruistic action so that conscientious people (who are often not sufficiently scandal-prone) self-select out of powerful positions and open them up to people with more scandal potential.
We encourage more scathing ad-hom attacks on leadership so that those who take any criticism to heart self-select out of leadership roles.
We make these positions more attractive to scandal-prone people by abandoning cost-effectiveness analyses and instead base strategy and grantmaking on vibes and relationships.
We further improve the cushiness of these positions by centralizing power and funding around them to thwart criticism and prevent Hayekian diversity and experimentation.
We build stronger relationships with powerful, unscrupulous people and companies by, e.g., helping them with their hiring.
We emphasize in-person networking and move the most valuable networks to some of the most expensive spots in the world. That way access to the network comes with even greater dependency on centralized funding, making it easier to control.
[Meta: I’m not claiming anyone is doing these things on purpose! It would be nice, though, if more people were trying to counter these risk factors for scandals and generally bad epistemics.]
Great summary!
You probably base “Even though this use of funds was unintentional and sounds extremely sketchy, FTX’s general counsel testified that FTX’s terms of service did not prohibit it” on:
The government didn’t want to focus you on that. Why? Again, the only witness who said he had read the terms of service was Can Sun, the general counsel who had helped to draft it. Even though he was very careful in what he told you, he admitted that nowhere do the terms of service contain language that prevents FTX from loaning customer fiat deposits to Alameda or anyone else.
Can Sun didn’t think so. (Unless I misunderstand something.) He said that there was the margin lending program that did allow that but that had a few hundred million USD in it, so by far not enough to explain Alameda’s borrowing. He didn’t think that FTX or Alameda could’ve borrowed from capital outside the margin lending program because it was owned by the customers.
So I think what the defense lawyer is trying to do here is to say that the ToS did not explicitly prohibit such borrowing, but he omits that the borrowing is still implicitly prohibited just like it is generally prohibited to borrow other people’s fund without their permission.
Yep, makes a lot of sense!
Yep, failing fast is nice! So you were just skeptical on priors because any one new thing is unlikely to succeed?
Yep, that makes a lot of sense. I’ve done donation forwarding for < 10 projects once, and it was already quite time-consuming!
I’ve (so far) read the first post and love it! But when I was working full-time on trying to improve grantmaking in EA (with GiveWiki, aggregating the wisdom of donors), you mostly advised me against it. (And I am actually mostly back to ETG now.) Was that because you weren’t convinced by GiveWiki’s approach to decentralizing grantmaking or because you saw little hope of it succeeding? Or something else? (I mean, please answer from your current perspective; no need to try to remember last summer.)
Oh, great! I interpreted “This is an offer to donate this amount to the project on the condition that it eventually becomes active” to mean that the project might not become active for any number of reasons, only one of them being the funding goal.
Oh, brilliant! USDC would also be my top choice. But I’m basically paying into a DAF, and so can’t get a refund if this project doesn’t succeed, right? That would have a high cost in option value since I don’t know whether my second-best donation opportunity will be on Manifund. Is there a way to donate to Rethink Priorities or the Center on Long-Term Risk through Manifund? That would lower-bound the cost in option value.
I used the comment field in the form to note that a field in the form was marked as optional when it was actually mandatory. That comment got automatically published here, and out of context it made no sense whatsoever. I think it would’ve been clearer to not automatically transfer this form feedback here (some people might’ve even assumed that it’s private feedback).