Thanks! But I see your point
iporphyry
Karma: 240
Certainly not deliberately. I’ll try to read it more carefully and update my comment
Fair enough. I admit that I skimmed the post quickly, for which I apologize, and part of this was certainly a knee-jerk reaction to even considering Leverage as a serious intellectual project rather than a total failure as such, which is not entirely fair. But I think maybe a version of this post I would significantly prefer would first explain your interest in Leverage specifically: that while they are a particularly egregious failure of the closed-research genre, it’s interesting to understand exactly how they failed and how the idea of a fast, less-than-fully transparent think tank can be salvaged. It does bother me that you don’t try to look for other examples of organizations that do some part of this more effectively, and I have trouble believing that they don’t exist. It reads a bit like an analysis of nation-building that focuses specifically on the mistakes and complexities of North Korea without trying to compare it to other less awful entities.
Edit: I mostly retract this comment. I skimmed and didn’t read the post carefully (something one should never do before leaving a negative comment) and interpreted it as “Leverage wasn’t perfect, but it is worth trying to make Leverage 2.0 work or have similar projects with small changes”. On rereading, I see that Jeff’s emphasis is more on analyzing and quantifying the failure modes than on salvaging the idea.
That said, I just want to point out that (at least as far as I understand it), there is a significant collection of people within and around EA who think that Leverage is a uniquely awful organization which suffered a multilevel failure extremely reminiscent of your run-of-the mill cult (not just for those who left it, but also for many people who are still in it), which soft-core threatens members to avoid negative publicity, exerts psychological control on members in ways that seem scary and evil. This is context that I think some people reading the sterilized publicity around Leverage will lack.
There are many directions from which people could approach Leverage 1.0, but the one that I’m most interested in is lessons for people considering attempting similar things in the future.I think there’s a really clear lesson here: don’t.I’ll elaborate: Leverage was a multilevel failure. A fundamentally dishonest and charismatic leader. A group of people very convinced that their particular chain of flimsy inferences led them to some higher truth that gave them advantages over everyone else. A frenzied sense of secrecy and importance. Ultimately, psychological harm and abuse.It is very clearly a negative example, and if someone is genuinely trying to gain some positive insight into a project from “things they did right” (or noticeably imitate techniques from that project), that would make me significantly less likely to think of them as being on the right track.There are examples of better “secret projects”—the Manhattan project as well as other high-security government organizations, various secret revolutionary groups like the early US revolutionaries, the abolitionist movement and the underground railroad, even various pro-social masonic orders. Having as one’s go-to example of something to emulate an organization that significantly crossed the line into cult territory (or at least into Aleister Crowley level grandiosity around a bad actor) would indicate to me a potential enlarged sense of self-importance, an emphasis on deference and exclusivity (“being on our team”) instead of competence and accountability, and a lack of emphasis on appropriate levels of humility and self-regulation.To be clear, I believe in decoupling and don’t think it’s wrong to learn from bad actors. But with such a deeply rotten track record, and so many decent organizations that are better than it along all parameters, Leverage is perhaps the clearest example of a situation where people should just “say oops” and stop looking for ways to gain any value from it (other than as a cautionary tale) that I have heard of in the EA/LW community.
I enjoyed this post a lot!
I’m really curious about your mention of the “schism” pattern because I both haven’t seen it and I sort of believe a version of it. What were the schism posts? And why are they bad?
I don’t know if what you call “schismatics” want to burn the commons of EA cooperation (which would be bad), or if they just want to stop the tendency in EA (and really, everywhere) of people pushing for everyone to adopt convergent views (the focus of “if you believe X you should also believe Y” which I see and dislike in EA, versus “I don’t think X is the most important thing, but if you believe X here are some ways you could can do it more effectively” which I would like to see more).
Though I can see myself changing my mind on this, I currently like the idea of a more loose EA community with more moving parts that has a larger spectrum of vaguely positive-EV views. I’ve actually considered writing something about it inspired by this post by Eric Neyman https://ericneyman.wordpress.com/2021/06/05/social-behavior-curves-equilibria-and-radicalism/ which quantifies, among other things, the intuition that people are more likely to change their mind/behavior in a significant way if there is a larger spectrum of points of view rather than a more bimodal distribution.
I like hypothesis generation, and I particularly like that in this post a few of the points are mutually exclusive (like numbers 7 and 10), which should happen in a hypothesis generation post. However this list, as well as the topic, feels lazy to me, in the sense of needing much more specificity in other to generate more light than heat.
I think my main issue is the extremely vague use of”quality” here. It’s ok to use vague terms when a concept is hard to define, but in this case it feels like there are more useful ways to narrow it down. For example you could say “the average post seems less informative/well-researched” or “the average poster seems less experienced/ qualified”, or “I learned more from the old forum than the new one” (I think especially a focus on your experience would make the issue more precise, and open up new options such as “posts became less fun once I learned all the basics and new people who are just learning them became less interesting to me”). I would like to see a hypothesis generation post that focuses much more on the specific ways that posts are “worse” (and generates hypotheses on what they are) rather than on reasons for this to be the case. I suspect that once a concrete question is asked, the potential reasons will become more concrete and testable.
Another issue is that I think a lot of the points are more properly “reasons that posts on a forum can be bad” rather than issues with current vs old posts and I have trouble believing that these issues were absent or better in the past. This would also be solved by trying to make the complaint specific.
I’m one of the people who submitted a post right before the deadline of the criticism contest. FWIW I think number 6 is off base. In my case, the deadline felt like a Schelling point. My post was long and kind of technical, and I didn’t have any expectation of getting money—though having a fake deadline was very helpful and I would probably not have written it without the contest. I don’t think that any of the posts that got prizes were written with an expectation of making a profit. They all looked like an investment of multiple hours by talented people who could have made much more money (at least in expectation) by doing something else. In order for someone profit-motivated to take advantage of this they would have to be involved and knowledgeable enough to write a competitive post, and unable to make money in essentially any other way. This seems like an unlikely combination, but if it does exist then I’d assume that supporting such people financially is an additional benefit rather than a problem.
But I agree with your meta-point that I implicitly assumed SSA together with my “assumption 5” and SSA might not follow from the other assumptions
Thanks! I didn’t fully understand what people meant by that and how it’s related to various forms of longtermism. Skimming the linked post was helpful to get a better picture.
Thanks for the links! They were interesting and I’m happy that philosophers, including ones close to EA, are trying to grapple with these questions.
I was confused by SIA, and found that I agree with Bostrom’s critique of it much more than with the argument itself. The changes to the prior it proposes seem ad hoc, and I don’t understand how to motivate them. Let me know if you know how to motivate them (without a posteriori arguments that they—essentially by definition—cancel the update terms in the DA). It also seems to me to quickly lead to infinite expectations if taken at face value, unless there is a way to consistently avoid this issue by avoiding some kind of upper bound on population?
Anthropic decision theory seems more interesting to me, though I haven’t had a chance to try to understand it yet. I’ll take a look at the paper you linked when I get a chance.
On longtermism, Bayesianism, and the doomsday argument
I’m trying to optimise something like “expected positive impact on a brighter future conditional on being the person that I am with the skills available to/accessible for me”.
If this is true, then I think you would be an EA. But from what you wrote it seems that you have a relatively large term in your philosophical objective function (as opposed to your revealed objective function, which for most people gets corrupted by personal stuff) on status/glory. I think the question determining your core philosophy would be which term you consider primary. For example if you view them as a means to an end of helping people and are willing to reject seeking them if someone convinces you they are significantly reducing your EV then that would reconcile the “A” part of EA.
A piece of advice I think younger people tend to need to hear is that you should be more willing to accept that “X is something I like and admire, and I am also not X” without having to then worry about your exact relationship to X or redefining X to include themselves (or looking for a different label Y). You are allowed to be aligned with EA but not be an EA and you might find this idea freeing (or I might be fighting the wrong fight here).
I think a lot of people miss the idea that “being an EA” is a different thing from being “EA adjacent”/”in the EA community”/ “working for an EA organization” etc. I am saying this as someone who is close to the EA community, who has an enormous amount of intellectual affinity, but does not identify as an EA. If the difference between the EA label and the EA community is already clear to you, then I apologize for beating a dead horse.
It seems from your description of yourself like you’re actually not an Effective Altruist in the sense of holding a significantly consequentialist worldview that one tries to square with one’s choices (once again, neither am I). From your post, the main way that I see in which your worldview deviates from EA is that, while lots of EA’s are status-motivated, your worldview seems to include the idea that typical levels of status -based and selfish motivations aren’t a cognitive error that should be pushed against.
I think that’s great! You have a different philosophical outlook (from the very little I can see in this post, perhaps it’s a little close to the more pro-market and pro-self interest view of people like Zvi, who everyone I know in the community respects immensely). I think that if people call this “evil” or “being a bad person”, they are being narrow-minded and harmful to the EA cause. But I also don’t think that people like you (and me) who love the EA community and goals but have a personal philosophy that deviates significantly from the EA core should call ourselves EA’s, any more than a heterosexual person who has lots of gay friends and works for a gay rights organization should call themselves LGBT. There is a core meaning to being an “effective altruist”, and you and I don’t meet it.
No two people’s philosophies are fully aligned, and even the most modal EA working in the most canonical EA organization will end up doing some things that feel “corporate” or suboptimal, or that matter to other people but not to them. If you work for an EA org, you might experience some of that because of your philosophical differences, but as long as you’re intellectually honest with yourself and others, and able to still do the best you can (and not try to secretly take your project in a mission-unaligned direction) then I am sure everyone would have a great experience.
My guess is that most EA organizations would love to hire/fund someone with your outlook (and what some of the posts you got upset with are worried about are people who are genuinely unaligned/deceptive and want to abuse the funding and status of the organization for personal gain). However if you do come in to an EA org and do your best, but people decline to work with you because of your choices or beliefs, I think that would be a serious organizational problem and evidence of harmful cultishness/”evaporative cooling of group beliefs”.
I like that you admit that your examples are cherry-picked. But I’m actually curious what a non-cherry-picked track record would show. Can people point to Yudkowsky’s successes? What did he predict better than other people? What project did MIRI generate that either solved clearly interesting technical problems or got significant publicity in academic/AI circles outside of rationalism/EA? Maybe instead of a comment here this should be a short-form question on the forum.
I have some serious issues with the way the information here is presented which make me think that this is best shared as something other than an EA forum post. My main issues are:
This announcement is in large part a promotion for the Fistula Foundation, with advertising-esque language. It would be appropriate in an advertising banner of an EA-aligned site but not on the forum, where critical discussion or direct information-sharing is the norm.
It includes the phrase that Fistula Foundation is “widely regarded as one of the most effective charities in the world” (in addition to some other similar phrases). This is weaselly language which should not be used without immediate justification (e.g. ”...according to X rating”).
In this case this is also misleading/false. I went to the EA impact assessment page for the foundation and it is actually estimated to be below the cost-effectiveness range of EA programs (while it seems to be considered a reasonable charity).
In general the language here together with the fact that the OP is affiliated with this foundation makes me update to take the fistula foundation much less seriously and to avoid donating there in the future. I would suggest for the OP to remove this post or to edit it in a way that is informative rather than pandering (e.g. something like “join me to go skinny-dipping for fistula foundation on X day. While it has a mediocre impact assessment, I like the cause and think skinny dipping would be a good way to support it while also becoming less insecure about our bodies”).
“Cardinal” and “Ordinal” denote a certain extremely crude way in economics in which different utility functions can still be compared in certain cases. They gesture at a very important issue in EA which everybody who thinks about it encounters: that different people (/different philosophies) have different ideas of the good, which correspond to different utility functions.
But the two terms come from math and are essentially only useful in theoretical arguments. In practical applications they are extremely weak to the point of being essentially meaningless in any specific case—using them in a real-world analysis is like trying to do engineering by assuming every object is either a cube or a sphere. The real world is much more detailed, and a key assumption of people who do EA-style work, which turns out to be correct in most instances, is that most people’s models of utility overlap significantly, so that if you do an altruistic action that you think is has high utility, most other people will agree with your assessment to some degree, or at least not assign it negative utility.
Of course this breaks in certain situations, especially around zero-sum games like war. I think it would be interesting to analyze how this can break, but I recommend using less jargony terms like ordinal/cardinal (which I think aren’t very useful here) and more concrete examples (which can be speculative, but not too far removed from the real world or from the context of altruistic interventions).
I like this criticism, but I think there are two essentially disjoint parts here that are being criticized. The first is excess legibility, i.e., the issue of having explicit metrics and optimizing to the metrics at all. The second is that a few of the measurements that determine how many resources a group gets/how quickly it grows are correlated with things that are not inherently valuable at best and harmful at worst.
The first problem seems really hard to me: the legibility/autonomy trade-off is an age-old problem that happens in politics, business, and science, and seems to involve a genuine trade-off between organizational efficiency and the ability to capitalize on good but unorthodox ideas and individuals.
The second seems more accessible (though still hard), and reasonably separable from the first. Here I see a couple of things you flag (other than legibility/”corporateness” by itself) as parameters that positively contribute to growth but negatively contribute to the ability of EA to attract intellectually autonomous people. The first is “fire-and-brimstone” style arguments, where EA outreach tends to be all-or-nothing, “you either help save the sick children or you burn in Utilitarian Hell”, and the second is common-denominator level messaging that is optimized to build community (so things like slogans, manufactured community and sense of purpose; things that attract the people like Bob in your thought experiment), but not optimized to appeal to meta-level thinkers who understand the reasoning behind the slogans. Both are vaguely correlated with EA having commonalities with religious communities, and so I’m going to borrow the adjective “pious” to refer to ideas and individuals for which these factors are salient.
I like that you are pointing out that a lot of EA outreach is, in one way or another, an “appeal to piety”, and this is possibly bad. There might be a debate about whether this is actually bad and to what extent (e.g., the Catholic church is inefficient, but the sheer volume of charity it generates is nothing to sneer at), but I think I agree with the intuition that this is suboptimal, and that by Goodhart’s law, if pious people are more likely to react to outreach, eventually they will form a supermajority.
I don’t want to devalue the criticism that legibility is in itself a problem, and particularly ugh-y to certain types of people (e.g. to smart humanities majors). But I think that the problem of piety can be solved without giving up on legibility, and instead by using better metrics, that have more entanglement with the real world. This is something I believed before this post, so I might be shoe-horning it here: take this with a grain of salt.
But I want to point out that organizations that are constantly evaluated on some measurable parameter don’t necessarily tend to end up excessively pious. A sports team can’t survive by having the best team spirit; a software company will not see any profit if it only hires people who fervently believe in its advertising slogans. So maybe a solution to the problem of appeals to piety is to, as you say, reduce the importance of the metric of “HEA” generation in determining funding, clout, etc., but replace it with other hard-to-fake metrics that are less correlated with piety and more correlated with actually being effective at what you do.
I haven’t thought much about what the best metrics would be and am probably not qualified to make recommendations, but I just for plausibility’s sake here are a couple of examples of things that I think would be cool (if not necessarily realistic):
First, it would be neat (though potentially expensive) if there were a yearly competition between teams of EA’s (maybe student groups, or maybe something on a larger level) to use a funding source to create an independent real-world project and have their impact in QALY’s judged by an impartial third party.
Second, I think it would be nice to make “intramural” forms of existing competitions, such as the fiction contest, Scott Alexander’s book review contest, various super-forecasting contests, etc., and grading University groups on success (relative to past results). If something like this is implemented, I’d also like to see the focus of things like the fiction competition move away from “good messaging” (which smacks of piety) and towards “good fiction that happens to have an EA component, if you look hard enough”.
I think that if the funding culture becomes more explicitly focused on concentration of talent and on real-world effects and less on sheer numbers or uncritical mission alignment, then outreach will follow suit and some of the issues that you address will be addressed.
I think this is a great post! It addresses a lot of my discomfort with the EA point of view, while retaining the value of the approach. Commenting in the spirit of this post.
I want to point out two things that I think work in Eric’s favor in a more sophisticated model like the one you described.
First, I like the model that impact follows an approximately log distribution. But I would draw a different conclusion from this.
It seems to me that there is some set of current projects, S (this includes the project of “expand S”). They have impact given by some variable that I agree is closer to log normal than normal. Now one could posit two models: one idealized model in which people know (and agree on) magnitue of impacts and a second, more realistic model, where impact is extremely uncertain, with standard deviation on the same order as the potential impact. In the idealized model, you would maximize impact by working on the most impactful project, and get comparatively much less impact by working on a random project you happen to enjoy. But in the realistic world with very large uncertainty, you would maximize expected value by working on a project on the fuzzy Pareto frontier of “potentially very impactful projects”, but within this set you would prioritize projects that you have the largest competitive advantage in (which I think is also log distributed to a large extent). Presumably “how much you enjoy a subject” is correlated to “how much log advantage you have over the average person”, which makes me suspicious of the severe impact/enjoyment trade-off in your second graph.
I think a strong argument against this point would be to claim that the log difference between individual affinities is much less than the log difference between impacts. I intuitively think this is likely, but the much greater knowledge people have of their comparative strengths over the (vastly uncertain) guesses about impact will counteract this. Here I would enjoy an analysis of a model of impact vs. personal competitive advantage that takes both of these things into account.
Another point, which I think is somewhat orthogonal to the discussion of “how enjoyable is the highest-impact job”, and which I think is indirectly related to Eric’s point, is nonlinearity of effort.
Namely, there is a certain amount of nonlinearity in how “amount of time dedicated to cause X” correlates with “contribution to X”. There is some superlinearity at low levels (where at first most of your work goes into gaining domain knowledge and experience), and some sublinearity at high levels (where you run the risk of burnout as well as saturation potential if you chose a narrow topic). Because of the sublinearity at high levels, I think it makes sense for most people to have at least two “things” they do.
If you buy this I think it makes a lot of sense to make your second “cause” some version of “have fun” (or related things like “pursue personal growth for its own sake”). There are three reasons I believe this. First, this is a neglected cause: unless you’re famous or rich, no one else will work on it, which means that no one else will even try to pick the low-hanging fruit. Second, it’s a cause where you are an expert and, from your position, payoff is easy to measure and unambiguous. And third, if you are genuinely using a large part of your energy to have a high-impact career, being someone who has fun (and on a meta level, being a community that actively encourages people to have fun) will encourage others to be more likely to follow your career path.
I should caveat the third point: there are bad/dangerous arguments that follow similar lines, that result in people convincing themselves that they are being impactful by being hedonistic, or pursuing their favorite pet project. People are rationalizers and love coming up with stories that say “making myself happy is also the right thing to do”. But while this is something to be careful of, I don’t think it makes arguments of this type incorrect.
Some possible bugs:
*When I click on the “listen online” option it seems broken (using this on a mac)
*When I click on the “AGI safety fundamentals” courses as podcasts, they take me to the “EA forum curated and popular” podcast. Not sure if this is intentional, or if they’re meant to point to a podcast containing just the course