Researching Causality and Safe AI at Oxford.
Previously, founder (with help from Trike Apps) of the EA Forum.
Discussing research etc at https://twitter.com/ryancareyai.
RyanCarey
Karma: 9,371
Another relevant comment:
Overall a nice system card for Opus 4! It’s a strange choice to contract Apollo to evaluate sabotage, get feedback that “[early snapshot] schemes and deceives at such high rates that we advise against deploying this model....” and then not re-contracting Apollo for final evals
I think we should keep our eye the most important role that online EA (and adjacent) platforms have played historically. Over the last 20 years, there has always been one or two key locations for otherwise isolated EAs and utilitarians to discover like-minds online, get feedback on their ideas, and then become researchers or real-life contributors. Originally, it was (various incarnations of) Felicifia, and then the EA Forum. The rationalist community benefited in similar ways from the extropians mailing list, SL4, Overcoming Bias and LessWrong. The sheer geographical coverage, and the element of in-depth intellectual engagement aren’t practically replaceable by other community-building efforts.
I think that fulfilling this role is a lot more important than growing the EA community, and other goals that the EA Forum might have, and that it is worth doing until a better new venue comes along. Currently, I don’t think a better venue exists. I don’t think r/effectivegiving or LessWrong would be a great successor. You could make a case for Substack+Twitter, but that may flip to something else in a few years time, how people want to connect online can change completely on that kind of timescale. Overall, I think it important to keep things running for the next 5-10 as the future of EA and the future of online discussion declare themselves.
Of course, this role could be performed without a lot of new technology.
The other thing I wonder is: if the online team stopped stewarding the EA Forum’s content, would it really turn into a mere bulletin board? I’m not so sure. I can imagine that plenty of people might continue to use the Forum to discuss EA matters and to post original research. If so, then this might be another way to cut costs with less change to the forum’s core role, compared to declaring it a bulletin board or moving conversation to a different platform.
Nice, I’ll look forward to reading this!
How is EAIF performing in the value proposition that it provides to funding applicants, such as the speed of decisions, responsiveness to applicants’ questions, and applicants’ reported experiences? Historically your sister fund was pretty bad to applicants, and some were really turned off by the experience.
I guess a lot of these faulty ideas come from the role of morality as a system of rules for putting up boundaries around acceptable behaviour, and for apportioning blameworthiness moreso than praiseworthiness. Similar to how the legal system usually gives individuals freedom so long as they’re not doing harm, our moral system mostly speaks to harms (rather than benefits) from actions (rather than inaction). By extension, the basis of the badness of these harms has to be a violation of “rights” (things that people deserve not to have done to them). Insofar as morality serves as a series of heuristics for people to follow, having a negativity-bias and action-bias are not necessarily wrong. It causes problems, however, if it this distorted lens is used to make claims about intrinsic right and wrong, or the idea that non-existence is an ideal.
Another relevant dimension is that the forum (and Groups) are the most targeted to EAs, so they will be most sensitive to fluctuations in the size of the EA community, whereas 80k will be the least sensitive, and Events will be somewhere in-between.
Given this and the sharp decline in applications to events, it seems like the issue is really a decrease in the size of, or enthusiasm in the EA community, rather than anything specific to the forum.
I’m sure I have some thoughts, but to begin with, it would helpful for understanding what’s going on if the dashboard would tell us how 2024 went for the events and groups teams.
Worth noting that although high EA salaries increase the risk to EA organisations, they reduce risk to EA individuals, because people can spend less than their full salary, thereby saving for a time when EA funding dries up.
I think the core issue is that the lottery wins you government dollars, which you can’t actually spend freely. Government dollars are simply worth less, to Pablo, than Pablo’s personal dollars. One way to see this is that if Pablo could spend the government dollars on the other moonshot opportunities, then it would be fine that he’s losing his own money.
So we should stipulate that after calculating abstract dollar values, you have to convert them, by some exchange rate, to personal dollars. The exchange rate simply depends on how much better the opportunities are for personal spending, versus spending government money.
The fact that opportunities can get larger than your budget size seems not to be the core issue for the reason that you mention—that at realistic sizes of opportunity, it is possible to instead buy a lottery for a chance at the opportunity instead.
Also Nick Bostrom, Nick Beckstead, Will Macaskill, Ben Todd, some of whom have been lifelong academics.
Probably different factors in different cases.
It sounds like you would prefer the rationalist community prevent its members from taking taboo views on social issues? But in my view, an important characteristic of the rationalist community, perhaps its most fundamental, is that it’s a place where people can re-evaluate the common wisdom, with a measure of independence from societal pressure. If you want the rationalist community (or any community) to maintain that character, you need to support the right of people to express views that you regard as repulsive, not just the views that you like. This could be different if the views were an incitement to violence, but proposing a hypothesis for socio-economic differences isn’t that.
In my view, what’s going on is largely these two things:
[rationalists etc] are well to the left of the median citizens, but they are to the right of [typical journalists and academics]
Of course. And:
biodeterminism… these groups are very, very right-wing on… eugenics, biological race and gender differences etc.-but on everything else they are centre-left.
Yes, ACX readers do believe that genes influence a lot of life outcomes, and favour reproductive technologies like embryo selection, which are right-coded views. These views are actually not restricted to the far-right, however. Most people will choose to have an abortion when they know their child will have a disability, for example.
Various of your other hypotheses don’t ring true to me. I think:
People aren’t self-deceiving about their own politics very much. They know which politicians and intellectuals they support, and who they vote for.
Rationalist leadership is not very politically different from the rationalist membership.
Sexual misbehaviour doesn’t change perceived political alignment very much.
The high % of male rationalist is at most a minor factor in the difference between perceived and actual politics.
This was just a “where do you rate yourself from 1-10” type question, but you can see more of the questions and data here.
I think the trend you describe is mostly an issue with “progressives”, i.e. “leftists” rather than an issue for all those left of center. And the rationalists don’t actually lean right in my experience. They average more like anti-woke and centrist. The distribution in the 2024 ACX survey below has perhaps a bit more centre-left and a bit less centre and centre-right than the rationalists at large but not by much, in my estimation.
There is one caveat: if someone acting on behalf on an EA organisation truly did something wrong which contributed to this fraud, then obviously we need to investigate that. But I am not aware of any evidence to suggest that happened.
I tend to think EA did. Back in September 2023, I argued:
EA contributed to a vast financial fraud, through its:
People. SBF was the best-known EA, and one of the earliest 1%. FTX’s leadership was mostly EAs. FTXFF was overwhelmingly run by EAs, including EA’s main leader, and another intellectual leader of EA.
Resources. FTX had some EA staff and was funded by EA investors.
PR. SBF’s EA-oriented philosophy on giving, and purported frugality served as cover for his unethical nature.
Ideology. SBF apparently had an RB ideology, as a risk-neutral act-utilitarian, who argued a decade ago why stealing was not in-principle wrong, on Felicifia. In my view, his ideology, at least as he professed it, could best be understood as an extremist variant of EA.
Of course, you can argue that contributing (point 1) people-time and (2) resources is consistent with us having just been victims, although I think that glosses over the extent to which EA folks at FTX had bought into Sam’s vision, and folks at FTXFF might have more mildly lapsed in judgment. And we could regard (3) the PR issue as minor. But even so, (4) the ideology is important. FTX wasn’t just any scam. It was one that a mostly-EA group was motivated to commit, to some degree or other, based on EA-style/consequentialist reasoning. There were several other instances of crypto-related crimes in and around the EA community. And the FTX implosion shared some characteristics with those events, and with other EA scandals. As I argued:
Other EA scandals, similarly, often involve multiple of these elements:
[Person #1]: past sexual harassment issues, later reputation management including Wiki warring and misleading histories. (norm-violation, naive conseq.)
[Person #2]: sexual harassment (norm-violation? naive conseq?)
[Person #3] [Person #4] [Person #5]: three more instances of crypto crimes (scope sensitivity? Norm-violation, naive conseq.? naivete?)
Intentional Insights: aggressive PR campaigns (norm-violation, naive conseq., naivete?)
Leverage Research, including partial takeover of CEA (risk appetite, norm-violation, naive conseq, unilateralism, naivete)
(We’ve seen major examples of sexual misbehaviour and crypto crimes in the rationalist community too.)You could argue still that some of these elements are things that are shared with all financial crime. But then why have EAs committed >10% of the largest financial frauds of all-time, while consisting of about one millionth of the world’s population, and less than 0.1% and perhaps 0.01% of its startups? You can suppose that we were just unlucky, but I don’t find this particularly convincing.
I think that at this point, you should want to concede that EA appears to have contributed to FTX in quite a number of ways, and not all of them can be dismissed easily. That’s why I think a more thorough investigation is needed.
As for PR, I simply think that shouldn’t be the primary focus, and that it far from the most important consideration on the current margin. First, we need to get the facts in order. And then we need to describe the strategy. And then based on what kind of future EA deserves to have, we could decide how and whether to try to defend its image.
Yes, that’s who I meant when I said “those working for the FTX Future Fund”
This is who I thought would be responsible too, along with the CEO of CEA, that they report to, (and those working for the FTX Future Fund, although their conflictedness means they can’t give an unbiased evaluation). But since the FTX catastrophe, the community health team has apparently broadened their mandate to include “epistemic health” and “Special Projects”, rather than narrowing it to focus just on catastrophic risks to the community, which would seem to make EA less resilient in one regard, than it was before.
Of course I’m not necessarily saying that it was possible to put the pieces together ahead of time, just that if there was one group responsible for trying, they were it.
In summarising Why They Do It, Will says that usually, that most fraudsters aren’t just “bad apples” or doing “cost-benefit analysis” on their risk of being punished. Rather, they fail to “conceptualise what they’re doing as fraud”. And that may well be true on average, but we know quite a lot about the details of this case, which I believe point us in a different direction.
In this case, the other defendants have said they knew what they’re doing was wrong, that they were misappropriating customers’ assets, and investing them. That weighs somewhat against the misconceptualisation hypothesis, albeit without ruling it out as a contributing factor.
On the other hand, we have some support for the bad apples idea. SBF has said:
In a lot of ways I don’t really have a soul. This is a lot more obvious in some contexts than others. But in the end there’s a pretty decent argument that my empathy is fake, my feelings are fake, my facial reactions are fake.
So I agree with Spencer, that SBF was at least deficient in affective experience, whether or not he was psychopathic.
Regarding cost-benefit analysis, I would tend to agree with Will that it’s unlikely that SBF and company made a detailed calculation of the costs and benefits of their actions (and clearly they calculated incorrectly if they did), although the perceived costs and benefits could also be a contributing factor.
So based on the specific knowledge of the case, I think that the bad apples hypothesis makes more sense than the cost-benefit hypothesis and misconceptualisation hypotheses.
There is also a fourth category worth considering—whether SBF’s views on side constraints were a likely factor—and I think overwhelmingly yes. Sure, as Will points out, SBF may have commented approvingly about a recent article on side constraints. But more recently, he referred to ethics as “this dumb game we woke Westerners play where we say all say the right shibboleths and so everyone likes us.” Furthermore, if we’re doing Facebook archaeology, we should also consider his earlier writing. In May 2012, SBF wrote about the idea of stealing to give:
I’m not sure I understand what the paradox is here. Fundamentally if you are going to donate the money to [The Humane League] and he’s going to buy lots of cigarettes with it it’s clearly in an act utilitarian’s interest to keep the money as long as this doesn’t have consequences down the road, so you won’t actually give it to him if he drives you. He might predict this and thus not give you the ride, but then your mistake was letting Paul know that you’re an act utilitarian, not in being one. Perhaps this was because you’ve done this before, but then not giving him money the previous time was possibly not the correct decision according to act utilitarianism, because although you can do better things with the money than he can, you might run in to problems later if you keep in. Similarly, I could go around stealing money from people because I can spend the money in a more utilitarian way than they can, but that wouldn’t be the utilitarian thing to do because I was leaving out of my calculation the fact that I may end up in jail if I do so.
…
As others have said, I completely agree that in practice following rules can be a good idea. Even though stealing might sometimes be justified in the abstract, in practice it basically never is because it breaks a rule that society cares a lot about and so comes with lots of consequences like jail. That being said, I think that you should, in the end, be an act utilitarian, even if you often think like a rule utilitarian; here what you’re doing is basically saying that society puts up disincentives for braking rules and those should be included in the act utilitarian calculation, but sometimes they’re big enough that a rule utilitarian calculation approximates it pretty well in a much simpler fashion.I’m sure people will interpret this passage in different ways. But it’s clear that, at least at this point in time, he was a pretty extreme act utilitarian.
Taking this and other information on balance, it seems clear in retrospect that a major factor is that SBF didn’t take side constraints that seriously.
Of course, most of this information wasn’t available or wasn’t salient in 2022, so I’m not claiming that we should have necessarily worried based on it. Nor am I implying that improved governance is not a part of the solution. Those are further questions.
It will do a service to your reader if you choose a title that explains what your post is arguing.