I think it would be really useful for there to be more public clarification on the relationship between effective altruism and Open Philanthropy.
My impression is that: 1. OP is the large majority funder of most EA activity. 2. Many EAs assume that OP is a highly EA organization, including the top. 3. OP really tries to explicitly not take responsibility for EA and does not claim to themselves be highly EA. 4. EAs somewhat assume that OP leaders are partially accountable to the EA community, but OP leaders would mostly disagree. 5. From the point of view of many EAs, EA represents something like a community of people with similar goals and motivations. There’s some expectations that people will look out for each other. 6. From the point of view of OP, EA is useful insofar as it provides valuable resources (talent, sometimes ideas and money).
My impression is that OP basically treats the OP-EA relationship as a set of transactions, each with positive expected value. Like, they would provide a $20k grant to a certain community, if they expect said community to translate into over $20k of value via certain members who would soon take on jobs at certain companies. Perhaps in-part because there are some overlapping friendships, I think that OP staff often explicitly try to only fund EAs in ways that make the clearest practical sense for specific OP goals, like hiring AI safety researchers.
In comparison, I think a lot of EAs think of EA as some kind of holy-ish venture tied to an extended community of people who will care about each other. To them, EA itself is an incredibly valuable idea and community that itself has the potential to greatly change the world. (I myself is more in this latter camp)
So on one side, we have a group that often views EA through reductive lenses, like as a specific recruiting arm. And on the other side, it’s more of a critical cultural movement.
I think it’s very possible for both sides to live in unison, but I think at the moment there’s a lot of confusion about this. I think a lot of EAs assume that OP shares a lot of the same beliefs they do.
If it is the case that OP is fairly narrow in its views and goals with EA, I would hope that other EAs realize that there might be a gap of [leaders+funders who care about EA for the reasons that EAs care about EA]. It’s just weird and awkward to have your large-majority funder be a group that just treats you very reductively.
As a simple example, if one thinks of EA as something akin to a key social movement/community, one might care a lot about: - The long-term health and growth of EA - The personal health of specific EAs, not just the very most productive ones - EA being an institution known for being highly honest and trustworthy
But if one thinks of them through a reductive lens, I’d expect them to care more about: - Potential short-term hires or funding - Potential liabilities - Community members not being too annoying with criticisms and stuff
I’ve met a few people who felt very betrayed by EA—I suspect that the above confusion is one reason why. I think that a lot of EA recruiters argue that EA represents a healthy community/movement. This seems like the most viral message, it’s not a surprise that people doing recruiting and promotion would lean on this idea. But if much of EA funding is really functionally treating it as a recruiting network, then that would be a disconnect.
Related, I’ve though that “EA Global has major community and social movement vibes, but has the financial incentives in line with a recruiting fair.”
Both perspectives can coexist, but greater clarity seems very useful, and might expose some important gaps.
Ozzie my apologies for not addressing all of your many points here, but I do want to link you to two places where I’ve expressed a lot of my takes on the broad topic:
On medium, I talk about how I see the community at some length. tldr: aligned around the question of how to do the most good vs the answer, heterogenous, and often specifically alien to me and my values.
On the forum, I talk about the theoretical and practical difficulties of OP being “accountable to the community” (and here I also address an endowment idea specifically in a way that people found compelling). Similarly from my POV it’s pretty dang hard to have the community be accountable to OP, in spite of everything people say they believe about that. Yes we can withold funding, after the fact, and at great community reputational cost. But I can’t e.g. get SBF to not do podcasts nor stop the EA (or two?) that seem to have joined DOGE and started laying waste to USAID. (On Bsky, they blame EAs for the whole endeavor)
Yes we can withold funding, after the fact, and at great community reputational cost. But I can’t e.g. get SBF to not do podcasts nor stop the EA (or two?) that seem to have joined DOGE and started laying waste to USAID.
I believe most EAs would agree these examples should never have been in OP’s proverbial sphere of responsibility.
There are other examples we could discuss regarding OP’s role (as makes sense, no organization is perfect), but that might distract from the main topic: clarity on the OP-EA relationship and the mutual expectations between parties.
It seems obvious that such Bsky threads contain significant inaccuracies. The question is how much weight to give such criticisms.
My impression is that many EAs wouldn’t consider these threads important enough to drive major decisions like funding allocations. However, the fact you mention it suggests it’s significant to you, which I respect.
About the OP-EA relationship—if factors like “avoiding criticism from certain groups” are important for OP’s decisions, saying so clearly is the kind of thing that seems useful. I don’t want to get into arguments about if it should[1], the first thing is to just understand that that’s where a line is.
More specifically, I think these discussions could be useful—but I’m afraid they will get in the way of the discussions of how OP will act, which I think is more important.
This is probably off-topic, but I was very surprised to read this, given how much he supported the Harris campaign, how much he gives to reduce global poverty, and how similar your views are on e.g. platforming controversial people.
Just flagging that the EA Forum upvoting system is awkward here. This comment says: 1. “I can’t say that we agree on very much” 2. “you are often a voice of reason” 3. “your voice will be missed”
As such, I’m not sure what the Agree / Disagree reacts are referring to, and I imagine similar for others reading this.
This isn’t a point against David, just a challenge with us trying to use this specific system.
Thanks for the response here! I was not expecting that.
This is a topic that can become frustratingly combative if not handled gracefully, especially in public forums. To clarify, my main point isn’t disagreement with OP’s position, but rather I was trying to help build clarity on the OP-EA relationship.
Some points: 1. The relationship between the “EA Community” and OP is both important (given the resources involved) and complex[1] . 2. In such relationships, there are often unspoken expectations between parties. Clarity might be awkward initially but leads to better understanding and coordination long-term. 3. I understand you’re uncomfortable with OP being considered responsible for much of EA or accountable to EA. This aligns with the hypotheses in my original comment. I’m not sure we’re disagreeing on anything here. 4. I appreciate your comments, though I think many people might reasonably still find the situation confusing. This issue is critical to many people’s long-term plans. The links you shared are helpful but leave some uncertainty—I’ll review them more carefully. 5. At this point, we might be more bottlenecked by EAs analyzing the situation than by additional writing from OP (though both are useful). EAs likely need to better recognize the limitations of the OP-EA relationship and consider what that means for the community. 6. When I asked for clarification, I imagined that EA community members working at the OP-EA intersection would be well positioned to provide insight. One challenge is that many people feel uncomfortable discussing this relationship openly due to the power imbalance.[2]. As well as the funding issue (OP funds EA), there’s also the fact that OP has better ways of privately communicating[3]. (This is also one issue why I’m unusually careful and long with my words with these discussions, sorry if it comes across as harder to read.) That said, comment interactions and assurances from the OP do help build trust.
i.e. For example, say an EA community member writes something that upsets someone at OP. Then that person holds a silent grudge, decides they don’t like that person, then doesn’t fund them later. This is very human, and there’s a clear information asymmetry. The EA community member would never know if this happens, so it would make sense for them to be extra cautious.
People at OP can confidentially discuss with each other how to best handle their side of the OP-EA relationship. But in comparison, EA community members mainly have the public EA Forum, so there’s an inherent disadvantage.
I’m interested in hearing from those who provided downvotes. I could imagine a bunch of reasons why one might have done so (there were a lot of points included here).
(Upon reflection, I don’t think my previous comment was very good. I tried to balance being concise, defensive, and comprehensive, but ended up with something confusing. I’d be happy to clarify my stance on this more at any time if asked, though it might well be too late now for that to be useful. Apologies!)
Many people claim that Elon Musk is an EA person, @Cole Killian has an EA Forum account and mentioned effective altruism on his (now deleted) website, Luke Farritor won the Vesuvius Challenge mentioned in this post (he also allegedly wrote or reposted a tweet mentioning effective altruism, but I can’t find any proof and people are skeptical)
This reminds me of another related tension I’ve noticed. I think that OP really tries to not take much responsibility for EA organizations, and I believe that this has led to something of a vacuum of leadership.
I think that OP has functionally has great power over EA.
In many professional situations, power comes with corresponding duties and responsibilities.
CEOs have a lot of authority, but they are also expected to be agentic, to keep on the lookout for threats, to be in charge of strategy, to provide guidance, and to make sure many other things are carried out.
The President clearly has a lot of powers, and that goes hand-in-hand with great expectations and duties.
There’s a version of EA funding where the top funders take on both leadership and corresponding responsibilities. These people ultimately have the most power, so arguably they’re best positioned to take on leadership duties and responsibilities.
But I think nonprofit funders often try not to take much in terms of responsibilities, and I don’t think OP is an exception. I’d also flag that I think EA Funds and SFF are in a similar boat, though these are smaller.
My impression is that OP explicitly tries not to claim any responsibility for the EA ecosystem / environment, and correspondingly argues it’s not particularly accountable to EA community members. Their role as I understand it is often meant to be narrow. This varies by OP team, but I think is true for the “GCR Capacity Building” team, which is closest to many “EA” orgs. I think this team mainly thinks of itself as a group responsible for making good decisions on a bunch of specific applications that hits their desk.
Again, this is a far narrower mandate than any conventional CEO would have.
If we had a “CEO or President” that were both responsible for and accountable to these communities, I’d expect things like: 1. A great deal of communication with these communities. 2. Clear and open leadership structures and roles. 3. A good deal of high-level strategizing. 4. Agentic behavior, like taking significant action to “make sure specific key projects happen.” 5. When there are failures, acknowledgement of said failures, as well as plans to fix or change.
I think we basically don’t have this, and none of the funders would claim to be this.
So here’s a question: “Is there anyone in the EA community who’s responsible for these sorts of things?”
I think the first answer I’d give is “no.” The second answer is something like, “Well, CEA is sort of responsible for some parts of this. But CEA really reports to OP given their funding. CEA has very limited power of its own. And CEA has repeatedly try to express limits in its power, plus its gone through lots of management transitions.”
In a well-run bureaucracy, I imagine that key duties would be clearly delegated to specific people or groups, and that groups would have the corresponding powers necessary to actually do a good job at them. You want key duties to be delegated to agents with the power to carry them out.
The ecosystem of EA organizations is not a well-organized bureaucracy. But that doesn’t mean there aren’t a lot of important duties to be performed. In my opinion, the fact that EA represents a highly-fragmented set of small organizations was functionally a decision by the funders (at least, they had a great deal of influence on this), so I’d hope that they would have thoughts on how to make sure the key duties get done somehow.
This might seem pretty abstract, so I’ll try coming up with some more specific examples: 1. Say a tiny and poorly-resourced org gets funded. They put together a board of their friends (the only people available), then proceed to significantly emotionally abuse their staff. Who is ultimately responsible here? I’d expect the founders would not at all want to take responsibility for this. 2. Before the FTX Future Fund blew up, I assumed that EA leaders had vetted it. Later I find out that OP purposefully tried to keep its distance and not get involved (in this case meaning that they didn’t investigate or warn anyone), in part because they didn’t see it as their responsibility, and claimed that because FTX Future Fund was a “competitor”, it wasn’t right for them to get involved. From what I can tell now, it was no one’s responsibility to vet the FTX Future Fund team or FTX organization. You might have assumed CEA, but CEA was funded by FTX and previously even had SBF as a board member—they were clearly not powerful and independent enough for this. 3. There are many people in the EA scene who invest large amounts of time and resources preparing for careers that only exist under the OP umbrella. Many or all of their future jobs will be under this umbrella. At the same time, it’s easy to imagine that they have almost no idea what the power structures at the top of this umbrella are like. This umbrella could change leadership or direction at any time, with very little warning. 4. There were multiple “EAs” on the board of OpenAI during that board member spat. That event seemed like a mess, and it negatively influenced a bunch of other EA organizations. Was that anyone’s responsibility? Can we have any assurances that community members will do a better job next time? (if there is a next time) 5. I’m not sure if many people at all, in positions of power, are spending much time thinking about long-term strategic issues for EA. It seems very easy for me to imagine large failures and opportunities we’re missing out on. This also is true for the nonprofit EA AI Safety Landscape—many of the specific organizations are too small and spread out to be very agentic, especially in cases of dealing with diverse and private information. I’ve heard good things recently about Zach Robinson at CEA, but also would note that CEA has historically been highly focused on some long-running projects (EAG, the EA Forum, Community Health), with fairly limited strategic or agentic capacity, plus being heavily reliant on OP. 6. Say OP decides to shut down the GCR Capacity Building team one day, and gives a 2-years notice. I’d expect this to be a major mess. Few people outside OP understand how the internals of OP decisions get made, so it’s hard for other EA members to see this coming or gauge how likely it is. My guess is that they don’t seem like they’d do this, but I have limited confidence. As such, it’s hard for me to suggest that people make long-term plans (3+ years) in this area. 7. We know that OP generally maximizes expected value. What happens when narrow EV optimization conflicts with honesty and other cooperative values? Would they represent the same choices that other EAs might want? I believe that FTX justified their bad actions using utilitarianism, for instance, and lots of businesses and nonprofits carry out highly Machiavellian and dishonest actions to advance their interests. Is it possible that EAs working under the OP umbrella are unknowingly supporting actions they might not condone? It’s hard to know without much transparency and evaluation.
On the plus side, I think OP and CEA have improved a fair bit on this sort of thing in the last few years. OP seems to be working to assure that grantees follow certain basic managerial criteria. New hires and operations have come in, which has seemed to have helped.
I’ve previously discussed my thinking on the potential limitations we’re getting from having small orgs here. Also, I remember that Oliver Habryka has repeatedly mentioned the lack of leadership around this scene—I think that this topic is one thing he was sort-of referring to.
Ultimately, my guess is that OP has certain goals they want to achieve, and it’s unlikely they or the other funders will want to take many of the responsibilities that I suggest here.
Given that, I think it would be useful for people in the EA ecosystem to understand this and respond accordingly. I think that our funding situation really needs diversification, and I think that funders willing to be more agentic in crucial areas that are currently lacking could do a lot of good. I expect that when it comes to “senior leadership”, there are some significant gains to be made, if the right people and resources can come together.
This seems directionally correct, but I would add more nuance.
While OP, as a grantmaker, has a goal it wants to achieve with its grants (and they wouldn’t be EA aligned if they didn’t), this doesn’t necessarily mean they are very short term. The Open Phil EA/LT Survey seems to me to show best what they care about in outcomes (talent working in impactful areas) but also how hard it is to pinpoint the actions and inputs needed. This leads me to believe that OP instrumentally cares about the community/ecosystem/network as it needs multiple touchpoints and interactions to get most people from being interested in EA ideas to working on impactful things.
On the other side, we use the term community in confusing ways. I was on a Community Builder Grant by CEA for two years when working at EA Germany, which many call national community building. What we were actually doing was working on the talent development pipeline, trying to find promising target groups, developing them and trying to estimate the talent outcomes.
Working on EA as a social movement/community while being paid is challenging. On one hand, I assume OP would find it instrumentally useful (see above) but still desire to track short-term outcomes as a grantmaker. As a grant recipient, I felt I couldn’t justify any actions that lacked a clear connection between outcomes and impact. Hosting closed events for engaged individuals in my local community, mentoring, having one-on-ones with less experienced people, or renting a local space for coworking and group events appeared harder to measure. I also believe in the norm of doing this out of care, wanting to give back to the community, and ensuring the community is a place where people don’t need to be compensated to participate.
I think this is a complex issue. I imagine it would be incredibly hard to give it a really robust write-up, and definitely don’t mean for my post to be definitive.
I think this is downstream of a lot of confusion about what ‘Effective Altruism’ really means, and I realise I don’t have a good definition any more. In fact, because all of the below can be criticised, it sort of explains why EA gets seemingly infinite criticism from all directions.
Is it explicit self-identification?
Is it explicit membership in a community?
Is it implicit membership in a community?
Is it if you get funded by OpenPhilanthropy?
Is it if you are interested or working in some particular field that is deemed “effective”?
Is it if you believe in totalising utilitarianism with no limits?
To always justify your actions with quantitative cost-effectiveness analyses where you’re chosen course of actions is the top ranked one?
Is it if you behave a certain way?
Because in many ways I don’t count as EA based off the above. I certainly feel less like one than I have in a long time.
For example:
I think a lot of EAs assume that OP shares a lot of the same beliefs they do.
I don’t know if this refers to some gestalt ‘belief’ than OP might have, or Dustin’s beliefs, or some kind of ‘intentional stance’ regarding OP’s actions. While many EAs shared some beliefs (I guess) there’s also a whole range of variance within EA itself, and the fundamental issue is that I don’t know if there’s something which can bind it all together.
I guess I think the question should be less “public clarification on the relationship between effective altruism and Open Philanthropy” and more “what does ‘Effective Altruism’ mean in 2025?”
I had Claude rewrite this, if the terminology is confusing. I think it’s edit is decent. ---
The EA-Open Philanthropy Relationship: Clarifying Expectations
The relationship between Effective Altruism (EA) and Open Philanthropy (OP) might suffer from misaligned expectations. My observations:
OP funds the majority of EA activity
Many EAs view OP as fundamentally aligned with EA principles
OP deliberately maintains distance from EA and doesn’t claim to be an “EA organization”
EAs often assume OP leadership is somewhat accountable to the EA community, while OP leadership likely disagrees
Many EAs see their community as a unified movement with shared goals and mutual support
OP appears to view EA more transactionally—as a valuable resource pool for talent, ideas, and occasionally money
This creates a fundamental tension. OP approaches the relationship through a cost-benefit lens, funding EA initiatives when they directly advance specific OP goals (like AI safety research). Meanwhile, many EAs view EA as a transformative cultural movement with intrinsic value beyond any specific cause area.
These different perspectives manifest in competing priorities:
EA community-oriented view prioritizes:
Long-term community health and growth
Individual wellbeing of community members
Building EA’s reputation for honesty and trustworthiness
Transactional view prioritizes:
Short-term talent pipeline and funding opportunities
Risk management (not wanting EA activities to wind up reflecting poorly on OP)
Minimizing EA criticism of OP and OP activities. (This is both annoying to deal with, and could hurt their specific activities)
This disconnect explains why some people might feel betrayed by EA. Recruiters often promote EA as a supportive community/movement (which resonates better), but if the funding reality treats EA more as a talent network, there’s a fundamental misalignment.
Another thought I’ve had: “EA Global has major community and social movement vibes, but has the financial incentives in line with a recruiting fair.”
Both perspectives can coexist, but greater clarity could be really useful here.
I’ve heard multiple reports of people being denied jobs around AI policy because of their history in EA. I’ve also seen a lot of animosity against EA from top organizations I think are important—like A16Z, Founders Fund (Thiel), OpenAI, etc. I’d expect that it would be uncomfortable for EAs to apply or work in to these latter places at this point.
This is very frustrating to me.
First, it makes it much more difficult for EAs to collaborate with many organizations where these perspectives could be the most useful. I want to see more collaborations and cooperation—not having EAs be allowed in many orgs makes this very difficult.
Second, it creates a massive incentive for people not to work in EA or on EA topics. If you know it will hurt your career, then you’re much less likely to do work here.
And a lighter third—it’s just really not fun to have a significant stigma associated with you. This means that many of the people I respect the most, and think are doing some of the most valuable work out there, will just have a much tougher time in life.
Who’s at fault here? I think the first big issue is that resistances get created against all interesting and powerful groups. There are similar stigmas against people across the political spectrum, for example, to certain crowds. A big part of “talking about morality and important issues, while having something non-obvious to say” is being hated by a bunch of people. In this vein, arguably we should be aiming for a world where it winds up that there’s a larger stigma.
But a lot clearly has to do with the decisions made by what seems like a few EAs. FTX hurt the most. I think the OpenAI board situation resulted in a lot of ea-paranoia, arguably with very little upside. More recently, I think that certain EA actions in ai policy are getting a lot of flak.
There was a brief window, pre-FTX-fail, where there was a very positive EA media push. I’ve seen almost nothing since. I think that “EA marketing” has been highly neglected, and that doesn’t seem to be changing.
Also, I suspect that the current EA AI policy arm could find ways to be more diplomatic and cooperative. When this arm upsets people, all of EA gets blamed.
My guess is that there are many other changes to do here too.
CEA is the obvious group to hypothetically be in charge of the EA parts of this. In practice, it seems like CEA has been very busy with post-FTX messes and leadership changes.
So I think CEA, as it becomes stable, could do a lot of good work making EA marketing work somehow. And I hope that the AI safety governance crowd can get better at not pissing off people. And hopefully, other EAs can figure out other ways to make things better and not worse.
If the above doesn’t happen, honestly, it could be worth it for EAs themselves to try to self-fund or coordinate efforts on this. The issue isn’t just one of “hurting long-term utility”, it’s one that just directly hurts EAs—so it could make a lot of sense for them to coordinate on improvements, even just in their personal interests.
On the positive front, I know its early days but GWWC have really impressed me with their well produced, friendly yet honest public facing stuff this year—maybe we can pick up on that momentum?
Also EA for Christians is holding a British conference this year where Rory Stewart and the Archbishop of Canterbury (biggest shot in the Anglican church) are headlining which is a great collaboration with high profile and well respected mainstream Christian / Christian-adjacent figures.
I think in general their public facing presentation and marketing seems a cut above any other EA org—happy to be proven wrong by other orgs which are doing a great job too. What I love is how they present their messages with such positivity, while still packing a real punch and not watering down their message. Check out their web-page and blog to see their work.
A few concrete examples - This great video “How rich are you really?”
- Nice rebranding of “Giving what we can pledge” to the snappier and clearer “10% pledge”— The diamond symbol as a simple yet strong sign of people taking the pledge, both on the forum here and linkedin - An amazing linked-in push with lots of people putting the diamond and explaining why they took the pledge. Many posts have been received really positively on my wall.
(Jumping in for our busy comms/exec team) Understanding the status of the EA brand and working to improve it is a top priority for CEA :) We hope to share more work on this in future.
I wrote a downvoted post recently about how we should be warning AI Safety talent about going into labs for personal branding reasons (I think there are other reasons not to join labs, but this is worth considering).
I think people are still underweighting how much the public are going to hate labs in 1-3 years.
I think from an advocacy standpoint it is worth testing that message, but based on how it is being received on the EAF, it might just bounce off people.
My instinct as to why people don’t find it a compelling argument;
They don’t have short timelines like me, and therefore chuck it out completely
Are struggling to imagine a hostile public response to 15% unemployment rates
At least at the time, Holly Elmore seemed to consider it at least somewhat compelling. I mentioned this was an argument I provided framed in the context of movements like PauseAI—a more politicized, and less politically averse coalition movement, that includes at least one arm of AI safety as one of its constituent communities/movements, distinct from EA.
>They don’t have short timelines like me, and therefore chuck it out completely
Among the most involved participants in PauseAI, presumably there may estimates of short timelines comparable to the rate of such estimates among effective altruists.
>Are struggling to imagine a hostile public response to 15% unemployment rates
Those in PauseAI and similar movements don’t.
>Copium
While I sympathize with and appreciate why there would be high rates of huffing copium among effective altruists (and adjacent communities, such as rationalists), others who have been picking up slack effective altruists have dropped in the last couple years, are reacting differently. At least in terms of safeguarding humanity from both the near-term and long-term vicissitudes of advancing AI, humanity has deserved better than EA has been able to deliver. Many have given up hope that EA will ever rebound to the point it’ll be able to muster living up to the promise of at least trying to safeguard humanity. That includes both many former effective altruists, and those who still are effective altruists. I consider there to still be that kind of ‘hope’ on a technical level, though on a gut level I don’t have faith in EA. I definitely don’t blame those who have any faith left in EA, let alone those who see hope in it.
Much of the difference here is the mindset towards ‘people’, and how they’re modeled, between those still firmly planted in EA but somehow with a fatalistic mindset, and those who still care about AI safety but have decided to move on in EA. (I might be somewhere in between, though my perspective as a single individual among general trends is barely relevant.) The last couple years have proven that effective altruists direly underestimated the public, and the latter group of people didn’t. While many here on the EA Forum may not agree that much—or even most—of what movements like PauseAI are doing are as effective as they could or should be, they at least haven’t succumbed to a plague of doomerism beyond what can seemingly even be justified.
To quote former effective altruist Kerry Vaughan, in a message addressed to those who still are effective altruists: “now is not the time for moral cowardice.” There are some effective altruists who heeded that sort of call when it was being made. There are others who weren’t effective altruists who heeded it too, when they saw most effective altruists had lost the will to even try picking up the ball again after they dropped it a couple times. New alliances between emotionally determined effective altruists and rationalists, and thousands of other people the EA community always underestimated, might from now on be carrying the team that is the global project of AI risk reduction—from narrow/near-term AI, to AGI/ASI.
EA can still change, though either it has to go beyond self-reflection and just change already, or get used to no longer being team captain of AI Safety.
Very sorry to hear these reports, and was nodding along as I read the post.
If I can ask, how do they know EA affiliation was the decision? Is this an informal ‘everyone knows’ thing through policy networks in the US? Or direct feedback for the prospective employer than EA is a PR-risk?
Of course, please don’t share any personal information, but I think it’s important for those in the community to be as aware as possible of where and why this happens if it is happening because of EA affiliation/history of people here.
I think that certain EA actions in ai policy are getting a lot of flak.
On Twitter, a lot of VCs and techies have ranted heavily about how much they dislike EAs.
See this segment from Marc Andreeson, where he talks about the dangers of Eliezer and EA. Marc seems incredibly paranoid about the EA crowd now.
(Go to 1 hour, 11min in, for the key part. I tried linking to the timestamp, but couldn’t get it to work in this editor after a few minutes of attempts)
Organized, yes. And so this starts with a mailing list. In the nineties is a transhumanist mailing list called the extropions. And these extropions, they might have got them wrong, extropia or something like that, but they believe in the singularity. So the singularity is a moment of time where AI is progressing so fast, or technology in general progressing so fast that you can’t predict what happens. It’s self evolving and it just. All bets are off. We’re entering a new world where you.
[00:25:27]
Just can’t predict it, where technology can’t.
[00:25:29]
Be controlled, technology can’t be controlled. It’s going to remake, remake everything. And those people believe that’s a good thing because the world now sucks so much and we are imperfect and unethical and all sorts of irrational whatever. And so they really wanted for the singularity to happen. And there’s this young guy on this list, his name’s Iliezer Itkowski, and he claims he can write this AI and he would write really long essays about how to build this AIH suspiciously. He never really publishes code, and it’s all just prose about how he’s going to be able to build AI anyways. He’s able to fundraise. They started this thing called the Singularity Institute. A lot of people were excited about the future, kind of invested in him. Peter Thiel, most famously. And he spent a few years trying to build an AI again, never published code, never published any real progress. And then came out of it saying that not only you can’t build AI, but if you build it, it will kill everyone. So he switched from being this optimist. Singularity is great to actually, AI will for sure kill everyone. And then he was like, okay, the reason I made this mistake is because I was irrational.
[00:26:49]
And the way to get people to understand that AI is going to kill everyone is to make them rational. So he started this blog called less wrong and less wrong walks you through steps to becoming more rational. Look at your biases, examine yourself, sit down, meditate on all the irrational decisions you’ve made and try to correct them. And then they start this thing called center for Advanced Rationality or something like that. Cifar. And they’re giving seminars about rationality, but.
[00:27:18]
The intention seminar about rationality, what’s that like?
[00:27:22]
I’ve never been to one, but my guess would be they will talk about the biases, whatever, but they have also weird things where they have this almost struggle session like thing called debugging. A lot of people wrote blog posts about how that was demeaning and it caused psychosis in some people. 2017, that community, there was collective psychosis. A lot of people were kind of going crazy. And this all written about it on the Internet, debugging.
[00:27:48]
So that would be kind of your classic cult technique where you have to strip yourself bare, like auditing and Scientology or. It’s very common, yes.
[00:27:57]
Yeah.
[00:27:59]
It’s a constant in cults.
[00:28:00]
Yes.
[00:28:01]
Is that what you’re describing?
[00:28:02]
Yeah, I mean, that’s what I read on these accounts. They will sit down and they will, like, audit your mind and tell you where you’re wrong and all of that. And it caused people huge distress on young guys all the time talk about how going into that community has caused them huge distress. And there were, like, offshoots of this community where there were so suicides, there were murders, there were a lot of really dark and deep shit. And the other thing is, they kind of teach you about rationality. They recruit you to AI risk, because if you’re rational, you’re a group. We’re all rational now. We learned the art of rationality, and we agree that AI is going to kill everyone. Therefore, everyone outside of this group is wrong, and we have to protect them. AI is going to kill everyone. But also they believe other things. Like, they believe that polyamory is rational and everyone that.
[00:28:57]
Polyamory?
[00:28:57]
Yeah, you can have sex with multiple partners, essentially, but they think that’s.
[00:29:03]
I mean, I think it’s certainly a natural desire, if you’re a man, to sleep with more indifferent women, for sure. But it’s rational in the sense how, like, you’ve never meth happy, polyamorous, long term, and I’ve known a lot of them, not a single one.
[00:29:21]
So how would it might be self serving, you think, to recruit more impressionable.
[00:29:27]
People into and their hot girlfriends?
[00:29:29]
Yes.
[00:29:30]
Right. So that’s rational.
[00:29:34]
Yeah, supposedly. And so they, you know, they convince each other of all these cult like behavior. And the crazy thing is this group ends up being super influential because they recruit a lot of people that are interested in AI. And the AI labs and the people who are starting these companies were reading all this stuff. So Elon famously read a lot of Nick Bostrom as kind of an adjacent figure to the rationale community. He was part of the original mailing list. I think he would call himself a rationale part of the rational community. But he wrote a book about AI and how AI is going to kill everyone, essentially. I think he monitored his views more recently, but originally he was one of the people that are kind of banging the alarm. And the foundation of OpenAI was based on a lot of these fears. Elon had fears of AI killing everyone. He was afraid that Google was going to do that. And so they group of people, I don’t think everyone at OpenAI really believed that. But some of the original founding story was that, and they were recruiting from that community so much.
[00:30:46]
So when Sam Altman got fired recently, he was fired by someone from that community, someone who started with effective altruism, which is another offshoot from that community, really. And so the AI labs are intermarried in a lot of ways with this community. And so it ends up, they kind of borrowed a lot of their talking points, by the way, a lot of these companies are great companies now, and I think they’re cleaning up house.
[00:31:17]
But there is, I mean, I’ll just use the term. It sounds like a cult to me. Yeah, I mean, it has the hallmarks of it in your description. And can we just push a little deeper on what they believe? You say they are transhumanists.
[00:31:31]
Yes.
[00:31:31]
What is that?
[00:31:32]
Well, I think they’re just unsatisfied with human nature, unsatisfied with the current ways we’re constructed, and that we’re irrational, we’re unethical. And so they long for the world where we can become more rational, more ethical, by transforming ourselves, either by merging with AI via chips or what have you, changing our bodies and fixing fundamental issues that they perceive with humans via modifications and merging with machines.
[00:32:11]
It’s just so interesting because. And so shallow and silly. Like a lot of those people I have known are not that smart, actually, because the best things, I mean, reason is important, and we should, in my view, given us by God. And it’s really important. And being irrational is bad. On the other hand, the best things about people, their best impulses, are not rational.
[00:32:35]
I believe so, too.
[00:32:36]
There is no rational justification for giving something you need to another person.
[00:32:41]
Yes.
[00:32:42]
For spending an inordinate amount of time helping someone, for loving someone. Those are all irrational. Now, banging someone’s hot girlfriend, I guess that’s rational. But that’s kind of the lowest impulse that we have, actually.
[00:32:53]
We’ll wait till you hear about effective altruism. So they think our natural impulses that you just talked about are indeed irrational. And there’s a guy, his name is Peter Singer, a philosopher from Australia.
[00:33:05]
The infanticide guy.
[00:33:07]
Yes.
[00:33:07]
He’s so ethical. He’s for killing children.
[00:33:09]
Yeah. I mean, so their philosophy is utilitarian. Utilitarianism is that you can calculate ethics and you can start to apply it, and you get into really weird territory. Like, you know, if there’s all these problems, all these thought experiments, like, you know, you have two people at the hospital requiring some organs of another third person that came in for a regular checkup or they will die. You’re ethically, you’re supposed to kill that guy, get his organ, and put it into the other two. And so it gets. I don’t think people believe that, per se. I mean, but there’s so many problems with that. There’s another belief that they have.
[00:33:57]
But can I say that belief or that conclusion grows out of the core belief, which is that you’re God. Like, a normal person realizes, sure, it would help more people if I killed that person and gave his organs to a number of people. Like, that’s just a math question. True, but I’m not allowed to do that because I didn’t create life. I don’t have the power. I’m not allowed to make decisions like that because I’m just a silly human being who can’t see the future and is not omnipotent because I’m not God. I feel like all of these conclusions stem from the misconception that people are gods.
[00:34:33]
Yes.
[00:34:34]
Does that sound right?
[00:34:34]
No, I agree. I mean, a lot of the. I think it’s, you know, they’re at roots. They’re just fundamentally unsatisfied with humans and maybe perhaps hate, hate humans.
[00:34:50]
Well, they’re deeply disappointed.
[00:34:52]
Yes.
[00:34:53]
I think that’s such a. I’ve never heard anyone say that as well, that they’re disappointed with human nature, they’re disappointed with human condition, they’re disappointed with people’s flaws. And I feel like that’s the. I mean, on one level, of course. I mean, you know, we should be better, but that, we used to call that judgment, which we’re not allowed to do, by the way. That’s just super judgy. Actually, what they’re saying is, you know, you suck, and it’s just a short hop from there to, you should be killed, I think. I mean, that’s a total lack of love. Whereas a normal person, a loving person, says, you kind of suck. I kind of suck, too. But I love you anyway, and you love me anyway, and I’m grateful for your love. Right? That’s right.
[00:35:35]
That’s right. Well, they’ll say, you suck. Join our rationality community. Have sex with us. So.
[00:35:43]
But can I just clarify? These aren’t just like, you know, support staff at these companies? Like, are there?
[00:35:50]
So, you know, you’ve heard about SBF and FDX, of course.
[00:35:52]
Yeah.
[00:35:52]
They had what’s called a polycule.
[00:35:54]
Yeah.
[00:35:55]
Right. They were all having sex with each other.
[00:35:58]
Given. Now, I just want to be super catty and shallow, but given some of the people they were having sex with, that was not rational. No rational person would do that. Come on now.
[00:36:08]
Yeah, that’s true. Yeah. Well, so, you know. Yeah. It’s what’s even more disturbing, there’s another ethical component to their I philosophy called longtermism, and this comes from the effective altruist branch of rationality, long termism. Long termism. What they think is, in the future, if we made the right steps, there’s going to be a trillion humans, trillion minds. They might not be humans, that might be AI, but they’re going to be trillion minds who can experience utility, who can experience good things, fun things, whatever. If you’re a utilitarian, you have to put a lot of weight on it, and maybe you discount that, sort of like discounted cash flows. Uh, but you still, you know, have to pause it that, you know, you know, if. If there are trillions, perhaps many more people in the future, you need to value that very highly. Even if you discount it a lot, it ends up being valued very highly. So a lot of these communities end up all focusing on AI safety, because they think that AI, because they’re rational. They arrived, and we can talk about their arguments in a second. They arrived at the conclusion that AI is going to kill everyone.
[00:37:24]
Therefore, effective altruists and rational community, all these branches, they’re all kind of focused on AI safety, because that’s the most important thing, because we want a trillion people in the future to be great. But when you’re assigning value that high, it’s sort of a form of Pascal’s wager. It is sort of. You can justify anything, including terrorism, including doing really bad things, if you’re really convinced that AI is going to kill everyone and the future holds so much value, more value than any living human today has value. You might justify really doing anything. And so built into that, it’s a.
[00:38:15]
Dangerous framework, but it’s the same framework of every genocidal movement from at least the French Revolution. To present a glorious future justifies a bloody present.
[00:38:28]
Yes.
[00:38:30]
And look, I’m not accusing them of genocidal intent, by the way. I don’t know them, but those ideas lead very quickly to the camps.
[00:38:37]
I feel kind of weird just talking about people, because generally I like to talk about ideas about things, but if they were just like a silly Berkeley cult or whatever, and they didn’t have any real impact on the world, I wouldn’t care about them. But what’s happening is that they were able to convince a lot of billionaires of these ideas. I think Elon maybe changed his mind, but at some point he was convinced of these ideas. I don’t know if he gave them money. I think there was a story at some point, Wall Street Journal, that he was thinking about it. But a lot of other billionaires, billionaires gave them money, and now they’re organized, and they’re in DC lobbying for AI regulation. They’re behind the AI regulation in California and actually profiting from it. There was a story in pirate wares where the main sponsor, Dan Hendrix, behind SB 1047, started a company at the same time that certifies the safety of AI. And as part of the bill, it says that you have to get certified by a third party. So there’s aspects of it that are kind of. Let’s profit from it.
[00:39:45]
By the way, this is all allegedly based on this article. I don’t know for sure. I think Senator Scott Weiner was trying to do the right thing with the bill, but he was listening to a lot of these cult members, let’s call them, and they’re very well organized, and also a lot of them still have connections to the big AI labs, and some of them work there, and they would want to create a situation where there’s no competition in AI regulatory capture, per se. I’m not saying that these are the direct motivations. All of them are true believers. But you might infiltrate this group and direct it in a way that benefits these corporations.
[00:40:32]
Yeah, well, I’m from DC, so I’ve seen a lot of instances where my bank account aligns with my beliefs. Thank heaven. Just kind of happens. It winds up that way. It’s funny. Climate is the perfect example. There’s never one climate solution that makes the person who proposes it poorer or less powerful.
To be fair to the CEO of Replit here, much of that transcript is essentially true, if mildly embellished. Many of those events or outcomes associated with EA, or adjacent communities during their histories, that should be the most concerning to anyone other than any FTX-related events and for reasons beyond just PR concerns, can and have been well-substantiated.
My guess is this is obvious, but the “debugging” stuff seems as far as I can tell completely made up.
I don’t know of any story in which “debugging” was used in any kind of collective way. There was some Leverage-research adjacent stuff that kind of had some attributes like this, “CT-charting”, which maybe is what it refers to, but that sure would be the wrong word, and I also don’t think I’ve ever heard of any psychoses or anything related to that.
The only in-person thing I’ve ever associated with “debugging” is when at CFAR workshops people were encouraged to create a “bugs-list”, which was just a random list of problems in your life, and then throughout the workshop people paired with other people where they could choose any problem of their choosing, and work with their pairing partner on fixing it. No “auditing” or anything like that.
I haven’t read the whole transcript in-detail, but this section makes me skeptical of describing much of that transcript as “essentially true”.
I have personally heard several CFAR employees and contractors use the word “debugging” to describe all psychological practices, including psychological practices done in large groups of community members. These group sessions were fairly common.
In that section of the transcript, the only part that looks false to me is the implication that there was widespread pressure to engage in these group psychology practices, rather than it just being an option that was around. I have heard from people in CFAR who were put under strong personal and professional pressure to engage in *one-on-one* psychological practices which they did not want to do, but these cases were all within the inner ring and AFAIK not widespread. I never heard any stories of people put under pressure to engage in *group* psychological practices they did not want to do.
For what it’s worth, I was reminded of Jessica Taylor’s account of collective debugging and psychoses as I read that part of the transcript. (Rather than trying to quote pieces of Jessica’s account, I think it’s probably best that I just link to the whole thing as well as Scott Alexander’s response.)
I presume this account is their source for the debugging stuff, wherein an ex-member of the rationalist Leverage institute described their experiences. They described the institute as having “debugging culture”, described as follows:
In the larger rationalist and adjacent community, I think it’s just a catch-all term for mental or cognitive practices aimed at deliberate self-improvement.
At Leverage, it was both more specific and more broad. In a debugging session, you’d be led through a series of questions or attentional instructions with goals like working through introspective blocks, processing traumatic memories, discovering the roots of internal conflict, “back-chaining” through your impulses to the deeper motivations at play, figuring out the roots of particular powerlessness-inducing beliefs, mapping out the structure of your beliefs, or explicating irrationalities.
and:
1. 2–6hr long group debugging sessions in which we as a sub-faction (Alignment Group) would attempt to articulate a “demon” which had infiltrated our psyches from one of the rival groups, its nature and effects, and get it out of our systems using debugging tools.
The podcast statements seem to be an embellished retelling of the contents of that blog post (and maybe the allegations made by scott alexander in the comments of this post). I don’t think describing them as “completely made up” is accurate.
Leverage was an EA-aligned organization, that was also part of the rationality community (or at least ‘rationalist-adjacent’), about a decade ago or more. For Leverage to be affiliated with the mantles of either EA or the rationality community was always contentious. From the side of EA, the CEA, and the side of the rationality community, largely CFAR, Leverage faced efforts to be shoved out of both within a short order of a couple of years. Both EA and CFAR thus couldn’t have then, and couldn’t now, say or do more to disown and disavow Leverage’s practices from the time Leverage existed under the umbrella of either network/ecosystem/whatever. They have. To be clear, so has Leverage in its own way.
At the time of the events as presented by Zoe Curzi in those posts, Leverage was basically shoved out the door of both the rationality and EA communities with—to put it bluntly—the door hitting Leverage on ass on the on the way out, and the door back in firmly locked behind them from the inside. In time, Leverage came to take that in stride, as the break-up between Leverage, and the rest of the institutional polycule that is EA/rationality, was extremely mutual.
Ien short, the course of events, and practices at Leverage that led to them, as presented by Zoe Curzi and others as a few years ago from that time circa 2018 to 2022, can scarcely be attributed to either the rationality or EA communities. That’s a consensus between EA, Leverage, and the rationality community agree on—one of few things left that they still agree on at all.
From the side of EA, the CEA, and the side of the rationality community, largely CFAR, Leverage faced efforts to be shoved out of both within a short order of a couple of years. Both EA and CFAR thus couldn’t have then, and couldn’t now, say or do more to disown and disavow Leverage’s practices from the time Leverage existed under the umbrella of either network/ecosystem/whatever…
At the time of the events as presented by Zoe Curzi in those posts, Leverage was basically shoved out the door of both the rationality and EA communities with—to put it bluntly—the door hitting Leverage on ass on the on the way out, and the door back in firmly locked behind them from the inside.
While I’m not claiming that “practices at Leverage” should be “attributed to either the rationality or EA communities”, or to CEA, the take above is demonstrably false. CEA definitely could have done more to “disown and disavow Leverage’s practices” and also reneged on commitments that would have helped other EAs learn about problems with Leverage.
Circa 2018 CEA was literally supporting Leverage/Paradigm on an EA community building strategy event. In August 2018 (right in the middle of the 2017-2019 period at Leverage that Zoe Curzi described in her post), CEA supported and participated in an “EA Summit” that was incubated by Paradigm Academy (intimately associated with Leverage). “Three CEA staff members attended the conference” and the keynote was delivered by a senior CEA staff member (Kerry Vaughan). Tara MacAulay, who was CEO of CEA until stepping down less than a year before the summit to co-found Alameda Research, personally helped fund the summit.
At the time, “the fact that Paradigm incubated the Summit and Paradigm is connected to Leverage led some members of the community to express concern or confusion about the relationship between Leverage and the EA community.” To address those concerns, Kerry committed to “address this in a separate post in the near future.” This commitment was subsequently dropped with no explanation other than “We decided not to work on this post at this time.”
This whole affair was reminiscent of CEA’s actions around the 2016 Pareto Fellowship, a CEA program where ~20 fellows lived in the Leverage house (which they weren’t told about beforehand), “training was mostly based on Leverage ideas”, and “some of the content was taught by Leverage staff and some by CEA staff who were very ‘in Leverage’s orbit’.” When CEA was fundraising at the end of that year, a community member mentioned that they’d heard rumors about a lack of professionalism at Pareto. CEA staff replied, on multiple occasions, that “a detailed review of the Pareto Fellowship is forthcoming.” This review was never produced.
Several years later, details emerged about Pareto’s interview process (which nearly 500 applicants went through) that confirmed the rumors about unprofessional behavior. One participant described it as “one of the strangest, most uncomfortable experiences I’ve had over several years of being involved in EA… It seemed like unscientific, crackpot psychology… it felt extremely cultish… The experience left me feeling humiliated and manipulated.”
I’ll also note that CEA eventually added a section to its mistakes page about Leverage, but not until 2022, and only after Zoe had published her posts and a commenter on Less Wrong explicitly asked why the mistakes page didn’t mention Leverage’s involvement in the Pareto Fellowship. The mistakes page now acknowledges other aspects of the Leverage/CEA relationship, including that Leverage had “a table at the careers fair at EA Global several times.” Notably, CEA has never publicly stated that working with Leverage was a mistake or that Leverage is problematic in any way.
The problems at Leverage were Leverage’s fault, not CEA’s. But CEA could have, and should have, done more to distance EA from Leverage.
Quick point—I think the relationship between CEA and Leverage was pretty complicated during a lot of this period.
There was typically a large segment of EAs who were suspicious of Leverage, ever since their founding. But Leverage did collaborate with EAs on some specific things early on (like the first EA Summit). It felt like an uncomfortable alliance type situation. If you go back on the forum / Lesswrong, you can read artifacts.
I think the period of 2018 or so was unusual. This was a period where a few powerful people at CEA (Kerry, Larissa) were unusually pro-Leverage and got to power fairly quickly (Tara left, somewhat suddenly). I think there was a lot of tension around this decision, and when they left (I think this period lasted around 1 year), I think CEA became much less collaborative with Leverage.
One way to square this a bit is that CEA was just not very powerful for a long time (arguably, its periods of “having real ability/agency to do new things” have been very limited). There were periods where Leverage had more employees (I’m pretty sure). The fact that CEA went through so many different leaders, each with different stances and strategies, makes it more confusing to look back on.
I would really love for a decent journalist to do a long story on this history, I think it’s pretty interesting.
Huh, yeah, that sure refers to those as “debugging”. I’ve never really heard Leverage people use those words, but Leverage 1.0 was a quite insular and weird place towards the end of its existence, so I must have missed it.
I think it’s kind of reasonable to use Leverage as evidence that people in the EA and Rationality community are kind of crazy and have indeed updated on the quotes being more grounded (though I also feel frustration with people equivocating between EA, Rationality and Leverage).
(Relatedly, I don’t particularly love you calling Leverage “rationalist” especially in a context where I kind of get the sense you are trying to contrast it with “EA”. Leverage has historically been much more connected to the EA community, and indeed had almost successfully taken over CEA leadership in ~2019, though IDK, I also don’t want to be too policing with language here)
I think it might describe how some people experienced internal double cruxing. I wouldn’t be that surprised if some people also found the ’debugging” frame in general to give too much agency to others relative to themselves, I feel like I’ve heard that discussed.
Based on the things titotal said, seems like it very likely refers to some Leverage stuff, which I feel a bit bad about seeing equivocated with the rest of the ecosystem, but also seems kind of fair. And the Zoe Curzi post sure uses the term “debugging” for those sessions (while also clarifying that the rest of the rationality community doesn’t use the term that way, but they sure seemed to)
I wouldn’t and didn’t describe that section of the transcript, as a whole, as essentially true. I said much of it is. As the CEO might’ve learned from Tucker Carlson, who in turned learned from FOX News, we should seek to be ‘fair and balanced.’
As to the debugging part, that’s an exaggeration that must have come out the other side of a game of broken telephone on the internet. It seems that on the other side of that telephone line would’ve been some criticisms or callouts I’ve read years ago of some activities happening in or around CFAR. I don’t recollect them in super-duper precise detail right now, nor do I have the time today to spend an hour or more digging them up on the internet
For the perhaps wrongheaded practices that were introduced into CFAR workshops for a period of time other than the ones from Leverage Research, I believe the others were some introduced by Valentine (e.g., ‘againstness,’ etc.). As far as I’m aware, at least as it was applied at one time, some past iterations of Connection Theory bore at least a superficial resemblance to some aspects of ‘auditing’ as practiced by Scientologists.
As to perhaps even riskier practices, I mean they happened not “in” but “around” CFAR in the sense of not officially happening under the auspices of CFAR, or being formally condoned by them, though they occurred within the CFAR alumni community and the Bay Area rationality community. It’s murky, though there was conduct in the lives of private individuals that CFAR informally enabled or emboldened, and could’ve/should’ve done more to prevent. For the record, I’m aware CFAR has effectively admitted those past mistakes, so I don’t want to belabor any point of moral culpability beyond what has been drawn out to death on LessWrong years ago.
Anyway, activities that occurred among rationalists in the social network that in CFAR’s orbit, that arguably arose to the level of triggering behaviour comparable in extremity to psychosis, include ‘dark arts’ rationality, and some of the edgier experiments of post-rationalists. That includes some memes spread and behaviours induced in some rationalists by Michael Vassar, Brent Dill, etc.
To be fair, I’m aware much of that was a result not of spooky, pseudo-rationality techniques, but some unwitting rationalists being effectively bullied into taking wildly mind-altering drugs, as guinea pigs in some uncontrolled DIY experiment. While responsibility for these latter outcomes may not be as attributable to CFAR, they can be fairly attributed to some past mistakes of the rationality community, albeit on a vague, semi-collective level.
I think it’s worth noting that the two examples you point to are right-wing, which the vast majority of Silicon Valley is not. Right-wing tech ppl likely have higher influence in DC, so that’s not to say they’re irrelevant, but I don’t think they are representative of silicon valley as a whole
I think Garry Tan is more left-wing, but I’m not sure. A lot of the e/acc community fights with EA, and my impression is that many of them are leftists.
I think that the right-wing techies are often the loudest, but there are also lefties in this camp too.
(Honestly though, the right-wing techies and left-wing techies often share many of the same policy ideas. But they seem to disagree on Trump and a few other narrow things. Many of the recent Trump-aligned techies used to be more left-coded.)
Garry Tan is the head of YCombinator, which is basically the most important/influential tech incubator out there. Around 8 years back, relations were much better, and 80k and CEA actually went through YCombinator.
I’d flag that Garry specifically is kind of wacky on Twitter, compared to previous heads of YC. So I definitely am not saying it’s “EA’s fault”—I’m just flagging that there is a stigma here.
I personally would be much more hesitant to apply to YC knowing this, and I’d expect YC would be less inclined to bring in AI safety folk and likely EAs.
Also, I suspect that the current EA AI policy arm could find ways to be more diplomatic and cooperative
My impression is that the current EA AI policy arm isn’t having much active dialogue with the VC community and the like. I see Twitter spats that look pretty ugly, I suspect that this relationship could be improved on with more work.
At a higher level, I suspect that there could be a fair bit of policy work that both EAs and many of these VCs and others would be more okay with than what is currently being pushed. My impression is that we should be focused on narrow subsets of risks that matter a lot to EAs, but don’t matter much to others, so we can essentially trade and come out better than we are now.
My impression is that we should be focused on narrow subsets of risks that matter a lot to EAs, but don’t matter much to others, so we can essentially trade and come out better than we are now.
That seems like the wrong play to me. We need to be focused on achieving good outcomes and not being popular.
My personal take is that there are a bunch of better trade-offs between the two that we could be making. I think that the narrow subset of risks is where most of the value is, so from that standpoint, that could be a good trade-off.
I’m nervous that the EA Forum might be having a small role for x-risk and some high-level prioritization work. - Very little biorisk content here, perhaps because of info-hazards. - Little technical AI safety work here, in part because that’s more for LessWrong / Alignment Forum. - Little AI governance work here, for whatever reason. - Not too much innovative, big-picture longtermist prioritization projects happening at the moment, from what I understand. - The cause of “EA community building” seems to be fairly stable, not much bold/controversial experimentation, from what I can tell. - Fairly few updates / discussion from grantmakers. OP is really the dominant one, and doesn’t publish too much, particularly about their grantmaking strategies and findings.
It’s been feeling pretty quiet here recently, for my interests. I think some important threads are now happening in private slack / in-person conversations or just not happening.
I don’t comment or post much on the EA forum because the quality of discourse on the EA Forum typically seems mediocre at best. This is especially true for x-risk.
The whole manifund debacle has left me quite demotivated. It really sucks that people are more interested debating contentious community drama, than seemingly anything else this forum has to offer.
Could you share a few examples of what you consider good quality EA Forum posts? Do you think the content linked on the EA Forum Digest also “typically seems mediocre at best”?
Very little biorisk content here, perhaps because of info-hazards.
When I write biorisk-related things publicly I’m usually pretty unsure of whether the Forum is a good place for them. Not because of info-hazards, since that would gate things at an earlier stage, but because they feel like they’re of interest to too small a fraction of people. For example, I could plausibly have posted Quick Thoughts on Our First Sampling Run or some of my other posts from https://data.securebio.org/jefftk-notebook/ here, but that felt a bit noisy?
It also doesn’t help that detailed technical content gets much less attention than meta or community content. For example, three days ago I wrote a comment on @Conrad K.’s thoughtful Three Reasons Early Detection Interventions Are Not Obviously Cost-Effective, and while I feel like it’s a solid contribution only four people have voted on it. On the other hand, if you look over my recent post history at my comments on Manifest, far less objectively important comments have ~10x the karma. Similarly the top level post was sitting at +41 until Mike bumped it last week, which wasn’t even high enough that (before I changed my personal settings to boost biosecurity-tagged posts) I saw it when it came out. I see why this happens—there are a lot more people with the background to engage on a community topic or even a general “good news” post—but it still doesn’t make me as excited to contribute on technical things here.
I’d be excited to have discussions of those posts here!
A lot of my more technical posts also get very little attention—I also find that pretty unmotivating. It can be quite frustrating when clearly lower-quality content on controversial stuff gets a lot more attention.
But this seems like a doom loop to me. I care much more about strong technical content, even if I don’t always read it, than I do most of the community drama. I’m sure most leaders and funders feel similarly.
Extended far enough, the EA Forum will be a place only for controversial community drama. This seems nightmarish to me. I imagine most forum members would agree.
I imagine that there are things the Forum or community can do to bring more attention or highlighting to the more technical posts.
I wonder if the forum is even a good place for a lot of these discussions? Feels like they need some combination of safety / shared context, expertise, gatekeeping etc?
establishing a reputation in writing as a person who can follow good argumentative norms (perhaps as a kind of extended courtship of EA jobs/orgs)
disseminating findings that are mainly meant for other forums, e.g. research reports
keeping track of what the community at large is thinking about/working on, which is mostly facilitated by organizations like RP & GiveWell using the forum to share their work.
I don’t think I would use the forum for hashing out anything I was really thinking hard about; I’d probably have in-person conversations or email particular persons.
I don’t know about you but I just learned about one of the biggest updates to OPs grantmaking in a year on the Forum.
That said, the data does show some agreement with your and commenters vibe of lowering quantity.
I agree that the Forum could be a good place for a lot of these discussions. Some of them aren’t happening at all to my knowledge.[1] Some of those should be, and should be discussed on the Forum. Others are happening in private and that’s rational, although you may be able to guess that my biased view is that a lot more should be public, and if they were, should be posted on the Forum.
Broadly: I’m quite bullish on the EA community as a vehicle for working on the world’s most pressing problems, and of open online discussion as a piece of our collective progress. And I don’t know of a better open place on the internet for EAs to gather.
Yep—I liked the discussion in that post a lot, but the actual post seemed fairly minimal, and written primarily outside of the EA Forum (it was a link post, and the actual post was 320 words total.)
For those working on the forum, I’d suggest work on bringing in more of these threads to the forum. Maybe reach out to some of the leaders in each group and see how to change things.
I think that AI policy in particular is most ripe for better infrastructure (there’s a lot of work happening, but no common public forums, from what I know), though it probably makes sense to be separate from the EA Forum (maybe like the Alignment Forum), because a lot of them don’t want to be associated too much with EA, for policy reasons.
I know less about Bio governance, but would strongly assume that a whole lot of it isn’t infohazardous. That’s definitely a field that’s active and growing.
For foundational EA work / grant discussions / community strategy, I think we might just need more content in the first place, or something.
I assume that AI alignment is well-handled by LessWrong / Alignment Forum, difficult and less important to push to happen here.
So I did used to do more sort of back of the envelope stuff, but it didn’t get much traction and people seemed to think it was unfished (it was) so I guess I had less enthusiasm.
There are still bunch of good discussions (see mostly posts with 10+ comments) in the last 6 months or so, its just that we can sometimes even go a week or two without more than one or two ongoing serious GHD chats. Maybe I’m wrong and there hasn’t actually been much (or any) meaningful change in activity this year looking at this.
As a random datapoint, I’m only just getting into the AI Governance space, but I’ve found little engagement with (some) (of[1]) (the) (resources) I’ve shared and have just sort of updated to think this is either not the space for it or I’m just not yet knowledgeable enough about what would be valuable to others.
I was especially disappointed with this one, because this was a project I worked on with a team for some time, and I still think it’s quite promising, but it didn’t receive the proportional engagement I would have hoped for. Given I optimized some of the project for putting out this bit of research specifically, I wouldn’t do the same now and would have instead focused on other parts of the project.
It seems from the comments that there’s a chance that much of this is just timing—i.e. right now is unusually quiet. It is roughly mid-year, maybe people are on vacation or something, it’s hard to tell.
I think that this is partially true. I’m not interested in bringing up this point to upset people, but rather to flag that maybe there could be good ways of improving this (which I think is possible!)
My sense of self-worth often comes from guessing what people I respect think of me and my work.
In EA… this is precarious. The most obvious people to listen to are the senior/powerful EAs.
In my experience, many senior/powerful EAs I know: 1. Are very focused on specific domains. 2. Are extremely busy. 3. Have substantial privileges (exceptionally intelligent, stable health, esteemed education, affluent/ intellectual backgrounds.) 4. Display limited social empathy (ability to read and respond to the emotions of others) 5. Sometimes might actively try not to sympathize/empathize with many people, because they are judging them for grants, and want don’t want to be biased. (I suspect this is the case for grantmakers). 6. Are not that interested in acting as a coach/mentor/evaluator to people outside their key areas/organizations. 7. Don’t intend or want others to care too much about what they think outside of cause-specific promotion and a few pet ideas they want to advance.
A parallel can be drawn with the world of sports. Top athletes can make poor coaches. Their innate talent and advantages often leave them detached from the experiences of others. I’m reminded by David Foster Wallace’s How Tracy Austin Broke My Heart.
If you’re a tennis player, tying your self-worth to what Roger Federer thinks of you is not wise. Top athletes are often egotistical, narrow-minded, and ambivalent to others. This sort of makes sense by design—to become a top athlete, you often have to obsess over your own abilities to an unnatural extent for a very long period.
Good managers are sometimes meant to be better as coaches than they are as direct contributors. In EA, I think those in charge seem more like “top individual contributors and researchers” than they do “top managers.” Many actively dislike management or claim that they’re not doing management. (I believe funders typically don’t see their work as “management*”, which might be very reasonable.)
But that said, even a good class of managers wouldn’t fully solve the self-worth issue. Tying your self-worth too much to your boss can be dangerous—your boss already has much power and control over you, so adding your self-worth to the mix seems extra precarious.
I think if I were to ask any senior EA I know, “Should I tie my self-worth with your opinion of me?” they would say something like,
“Are you insane? I barely know you or your work. I can’t at all afford the time to evaluate your life and work enough to form an opinion that I’d suggest you take really seriously.”
They have enough problems—they don’t want to additionally worry about others trying to use them as judges of personal value.
But this raises the question, Who, if anyone, should I trust to inform my self-worth?
Navigating intellectual and rationalist literature, I’ve grown skeptical of many other potential evaluators. Self-judgment carries inherent bias and ability to Goodhart. Many “personal coaches” and even “executive coaches” raise my epistemic alarm bells. Friends, family, and people who are “more junior” come with different substantial biases.
Some favored options are “friends of a similar professional class who could provide long-standing perspective” and “professional coaches/therapists/advisors.”
I’m not satisfied with any obvious options here. I think my next obvious move forward is to acknowledge that my current situation seems subpar and continue reflecting on this topic. I’ve dug into the literature a bit but haven’t found answers I’ve yet found compelling.
Who, if anyone, should I trust to inform my self-worth?
My initial thought is that it is pretty risky/tricky/dangerous to depend on external things for a sense of self-worth? I know that I certainly am very far away from an Epictetus-like extreme, but I try to not depend on the perspectives of other people for my self-worth. (This is aspirational, of course. A breakup or a job loss or a person I like telling me they don’t like me will hurt and I’ll feel bad for a while.)
A simplistic little thought experiment I’ve fiddled with: if I went to a new place where I didn’t know anyone and just started over, then what? Nobody knows you, and you social circle starts from scratch. That doesn’t mean that you don’t have a worth as a human being (although it might mean that you don’t have any worth in the ‘economic’ sense of other people wanting you, which is very different).
There might also be an intrinsic/extrinsic angle to this. If you evaluate yourself based on accomplishments, outputs, achievements, and so on, that has a very different feeling than the deep contentment of being okay as you are.
In another comment Austin mentions revenue and funding, but that seems to be a measure of things VERY different from a sense of self-worth (although I recognize that there are influential parts of society in which wealth or career success is seen as the proxies for worth). In favorable market conditions I have high self worth?
I would roughly agree with your idea of “trying not to tie my emotional state to my track record.”
I can relate, as someone who also struggles with self-worth issues. However, my sense of self-worth is tied primarily to how many people seem to like me / care about me / want to befriend me, rather than to what “senior EAs” think about my work.
I think that the framing “what is the objectively correct way to determine my self-worth” is counterproductive. Every person has worth by virtue of being a person. (Even if I find it much easier to apply this maxim to others than to myself.)
IMO you should be thinking about things like, how to do better work, but in the frame of “this is something I enjoy / consider important” rather than in the frame of “because otherwise I’m not worthy”. It’s also legitimate to want other people to appreciate and respect you for your work (I definitely have a strong desire for that), but IMO here also the right frame is “this is something I want” rather than “this is something that’s necessary for me to be worth something”.
It’s funny, I think you’d definitely be in the list of people I respect and care about their opinion of me. I think it’s just imposter syndrome all the way up.
Personally, one thing that seemed to work a bit for me is to find peers which I highly appreciate and respect and schedule weekly calls with them to help me prioritize and focus, and give me feedback.
After transitioning from for-profit entrepreneurship to co-leading a non-profit in the effective altruism space, I struggle to identify clear metrics to optimize for. Funding is a potential metric, but it is unreliable due to fluctuations in donors’ interests. The success of individual programs, such as user engagement with free products or services, may not accurately reflect their impact compared to other potential initiatives. Furthermore, creating something impressive doesn’t necessarily mean it’s useful.
Lacking a solid impact evaluation model, I find myself defaulting to measuring success by hours worked, despite recognizing the diminishing returns and increased burnout risk this approach entails.
This is brave of you to share. It sounds like there are a few related issues going on. I have a few thoughts that may or may not be helpful:
Firstly, you want to do well and improve in your work, and you want some feedback on that from people who are informed and have good judgment. The obvious candidates in the EA ecosystem are people who actually aren’t well suited to give this feedback to you. This is tough. I don’t have any advice to give you here.
However it also sounds like there are some therapeutic issues at play. You mention therapists as a favored option but one you’re not satisfied with and I’m wondering why? Personally I suspect that making progress on any therapeutic issues that may be at play may also end up helping with the professional feedback problem.
I think you’ve unfairly dismissed the best option as to who you can trust: yourself. That you have biases and flaws is not an argument against trusting yourself because everyone and everything has biases and flaws! Which person or AI are you going to find that doesn’t have some inherent bias or ability to Goodhart?
Five reasons why I think it’s unhelpful connecting our intrinsic worth to our instrumental worth (or anything aside from being conscious beings):
Undermines care for others (and ourselves): chickens have limited instrumental worth and often do morally questionable things. I still reckon chickens and their suffering are worthy of care. (And same argument for human babies, disabled people and myself)
Constrains effective work: continually assessing our self-worth can be exhausting (leaving less time/attention/energy for actually doing helpful work). For example, it can be difficult to calmy take on constructive feedback (on our work, or instrumental strengths or instrumental weaknesses) when our self-worth is on the line.
Constrains our personal wellbeing and relationships: I’ve personally found it hard to enjoy life when continuously questioning my self-worth and feeling guilty/shameful when the answer seems negative
Very hard to answer: including because the assessment may need to be continuously updated based on the new evidence from each new second of our lives
Seems pointless to answer (to me): how would accurately measuring our self-worth (against a questionable benchmark) make things better? We could live in a world where all beings are ranked so that more ‘worthy’ beings can appropriately feel superior, and less ‘worthy’ beings can appropriately feel ‘not enough’. This world doesn’t seem great from perspective
Despite thinking these things, I often unintentionally get caught up muddling my self-worth with my instrumental worth (can relate to the post and comments on here!) I’ve found ‘mindful self-compassion’ super helpful for doing less of this
The most obvious moves, to me, eventually, are to either be intensely neutral (as in, trying not to tie my emotional state to my track record), or to iterate on using AI to help here (futuristic and potentially dangerous, but with other nice properties).
A very simple example is, “Feed a log of your activity into an LLM with a good prompt, and have it respond with assessments of how well you’re doing vs. your potential at the time, and where/how you can improve.” You’d be free to argue points or whatever.
Reading this comment makes me think that you are basing your self-worth on your work output. I don’t have anything concrete to point to, but I suspect that this might have negative effects on happiness, and that being less outcome dependent will tend to result in a better emotional state.
(This is a draft I wrote in December 2021. I didn’t finish+publish it then, in part because I was nervous it could be too spicy. At this point, with the discussion post-chatGPT, it seems far more boring, and someone recommended I post it somewhere.)
Thoughts on the OpenAI Strategy
OpenAI has one of the most audacious plans out there and I’m surprised at how little attention it’s gotten.
First, they say flat out that they’re going for AGI.
Then, when they raised money in 2019, they had a clause that says investors will be capped at getting 100x of their returns back.
“Economic returns for investors and employees are capped… Any excess returns go to OpenAI Nonprofit… Returns for our first round of investors are capped at 100x their investment (commensurate with the risks in front of us), and we expect this multiple to be lower for future rounds as we make further progress.”[1]
On Hacker News, one of their employees says,
“We believe that if we do create AGI, we’ll create orders of magnitude more value than any existing company.” [2]
You can read more about this mission on the charter:
“We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power.
Our primary fiduciary duty is to humanity. We anticipate needing to marshal substantial resources to fulfill our mission, but will always diligently act to minimize conflicts of interest among our employees and stakeholders that could compromise broad benefit.”[3]
This is my [incredibly rough and speculative, based on the above posts] impression of the plan they are proposing:
Make AGI
Turn AGI into huge profits
Give 100x returns to investors
Dominate much (most?) of the economy, have all profits go to the OpenAI Nonprofit
Use AGI for “the benefit of all”?
I’m really curious what step 5 is supposed to look like exactly. I’m also very curious, of course, what they expect step 4 to look like.
Keep in mind that making AGI is a really big deal. If you’re the one company that has an AGI, and if you have a significant lead over anyone else that does, the world is sort of your oyster.[4] If you have a massive lead, you could outwit legal systems, governments, militaries.
I imagine that the 100x return cap means that the excess earnings would go to the hands of the nonprofit; which essentially means Sam Altman, senior leadership at OpenAI, and perhaps the board of directors (if legal authorities have any influence post-AGI).
This would be a massive power gain for a small subset of people.
If DeepMind makes AGI I assume the money would go to investors, which would mean it would be distributed to all of the Google shareholders. But if OpenAI makes AGI, the money will go to the leadership of OpenAI, on paper to fulfill the mission of OpenAI.
On the plus side, I expect that this subset is much more like the people reading this post than most other AGI competitors would be. (The Chinese government, for example). I know some people at OpenAI, and my hunch is that the people there are very smart and pretty altruistic. It might well be about the best we could expect from a tech company.
And, to be clear, it’s probably incredibly unlikely that OpenAI will actually create AGI, and even more unlikely they will do so with a decisive edge over competitors.
But, I’m sort of surprised so few other people seem at least a bit concerned and curious about the proposal? My impression is that most press outlets haven’t thought much at all about what AGI would actually mean, and most companies and governments just assume that OpenAI is dramatically overconfident in themselves.
(Aside on the details of Step 5) I would love more information on Step 5, but I don’t blame OpenAI for not providing it.
Any precise description of how a nonprofit would spend “a large portion of the entire economy” would upset a bunch of powerful people.
Arguably, OpenAI doesn’t really need to figure out Step 5 unless their odds of actually having a decisive AGI advantage seem more plausible.
I assume it’s really hard to actually put together any reasonable plan now for Step 5.
My guess is that we really could use some great nonprofit and academic work to help outline what a positive and globally acceptable (wouldn’t upset any group too much if they were to understand it) Step 5 would look like. There’s been previous academic work on a “windfall clause”[5] (their 100x cap would basically count), having better work on Step 5 seems very obvious.
Personally, I feel fairly strongly convinced to favor interventions that could help the future past 20 years from now. (A much lighter version of “Longtermism”).
If I had a budget of $10B, I’d probably donate a fair bit to some existing AI safety groups. But it’s tricky to know what to do with, say, $10k. And the fact that the SFF, OP, and others have funded some of the clearest wins makes it harder to know what’s exciting on-the-margin.
I feel incredibly unsatisfied with the public EA dialogue around AI safety strategy now. From what I can tell, there’s some intelligent conversation happening by a handful of people at the Constellation coworking space, but a lot of this is barely clear publicly. I think many people outside of Constellation are working on simplified models, like “AI is generally dangerous, so we should slow it all down,” as opposed to something like, “Really, there are three scary narrow scenarios we need to worry about.”
I recently spent a week in DC and found it interesting. But my impression is that a lot of people there are focused on fairly low-level details, without a great sense of the big-picture strategy. For example, there’s a lot of work into shovel-ready government legislation, but little thinking on what the TAI transition should really look like.
This sort of myopic mindset is also common in the technical space, where I meet a bunch of people focused on narrow aspects of LLMs, without much understanding of how their work exactly fits into the big picture of AI alignment. As an example, a lot of work seems like it would help with misuse risk, even when the big-picture EAs seem much more focused on accident risk.
Some (very) positive news is that we do have far more talent in this area than we did 5 years ago, and there’s correspondingly more discussion. But it still feels very chaotic.
A bit more evidence—it seems like OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?), but they have seemed to provide more support for AI safety community building / recruiting, and AI policy. But all of this still represents perhaps ~30% or so of their total budget, and I don’t sense that that’s about to change. Overall this comes off as measured and cautious. Meanwhile, it’s been difficult to convince other large donors to get into this area. (Other than Jaan Tallinn, he might well have been the strongest dedicated donor here).
Recently it seems like the community on the EA Forum has shifted a bit to favor animal welfare. Or maybe it’s just that the AI safety people have migrated to other blogs and organizations.
But again, I’m very hopeful that we can find interventions that will help in the long-term, so few of these excite me. I’d expect and hope that interventions that help the long-term future would ultimately improve animal welfare and more.
So on one hand, AI risk seems like the main intervention area for the long-term, but on the other, the field is a bit of a mess right now.
I feel quite frustrated that EA doesn’t have many other strong recommendations for other potential donors interested in the long-term. For example, I’d really hope that there could be good interventions to make the US government or just US epistemics more robust, but I barely see any work in that area.
“Forecasting” is one interesting area—it currently does have some dedicated support from OP. But it honestly seems to be in a pretty mediocre state to me right now. There might be 15-30 full-time people in the space at this point, and there’s surprisingly little in terms of any long-term research agendas.
Hi Ozzie – Peter Favaloro here; I do grantmaking on technical AI safety at Open Philanthropy. Thanks for this post, I enjoyed it.
I want to react to this quote: …it seems like OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?)
I agree that over the past year or two our grantmaking in technical AI safety (TAIS) has been too bottlenecked by our grantmaking capacity, which in turn has been bottlenecked in part by our ability to hire technical grantmakers. (Though also, when we’ve tried to collect information on what opportunities we’re missing out on, we’ve been somewhat surprised at how few excellent, shovel-ready TAIS grants we’ve found.)
Over the past few months I’ve been setting up a new TAIS grantmaking team, to supplement Ajeya’s grantmaking. We’ve hired some great junior grantmakers and expect to publish an open call for applications in the next few months. After that we’ll likely try to hire more grantmakers. So stay tuned!
OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?), but they have seemed to provide more support for AI safety community building / recruiting
Yeah, I find myself very confused by this state of affairs. Hundreds of people are being funneled through the AI safety community-building pipeline, but there’s little funding for them to work on things once they come out the other side.[1]
As well as being suboptimal from the viewpoint of preventing existential catastrophe, this also just seems kind of common-sense unethical. Like, all these people (most of whom are bright-eyed youngsters) are being told that they can contribute, if only they skill up, and then they later findout that that’s not the case.
These community-building graduates can, of course, try going the non-philanthropic route—i.e., apply to AGI companies or government institutes. But there are major gaps in what those organizations are working on, in my view, and they also can’t absorb so many people.
Yea, I think this setup has been incredibly frustrating downstream. I’d hope that people from OP with knowledge could publicly reflect on this, but my quick impression is that some of the following factors happened: 1. OP has had major difficulties/limitations around hiring in the last 5+ years. Some of this is lack of attention, some is that there aren’t great candidates, some is a lack of ability. This effected some cause areas more than others. For whatever reason, they seemed to have more success hiring (and retaining talent) for community than for technical AI safety. 2. I think there’s been some uncertainties / disagreements into how important / valuable current technical AI safety organizations are to fund. For example, I imagine if this were a major priority from those in charge of OP, more could have been done. 3. OP management seems to be a bit in flux now. Lost Holden recently, hiring a new head of GCR, etc. 4. I think OP isn’t very transparent and public with explaining their limitations/challenges publicly. 5. I would flag that there are spots at Anthropic and Deepmind that we don’t need to fund, that are still good fits for talent. 6. I think some of the Paul Christiano—connected orgs were considered a conflict-of-interest, given that Ajeya Cotra was the main grantmaker. 7. Given all of this, I think it would be really nice if people could at least provide warnings about this. Like, people entering the field are strongly warned that the job market is very limited. But I’m not sure who feels responsible / well placed to do this.
Thanks for the comment, I think this is very astute.
~
Recently it seems like the community on the EA Forum has shifted a bit to favor animal welfare. Or maybe it’s just that the AI safety people have migrated to other blogs and organizations.
I think there’s a (mostly but not entirely accurate) vibe that all AI safety orgs that are worth funding will already be approximately fully funded by OpenPhil and others, but that animal orgs (especially in invertebrate/wild welfare) are very neglected.
I don’t think that all AI safety orgs are actually fully funded since there are orgs that OP cannot fund for reasons (see Trevor’s post and also OP’s individual recommendations in AI) other than cost-effectiveness and also OP cannot and should not fund 100% of every org (it’s not sustainable for orgs to have just one mega-funder; see also what Abraham mentioned here). Also there is room for contrarian donation takes like Michael Dickens’s.
I think there’s a (mostly but not entirely accurate) vibe that all AI safety orgs that are worth funding will already be approximately fully funded by OpenPhil and others, but that animal orgs (especially in invertebrate/wild welfare) are very neglected.
That makes sense, but I’m feeling skeptical. There are just so many AI safety orgs now, and the technical ones generally aren’t even funded by OP.
On AI safety, I think it’s fairly likely (40%?) that the risk of x-risk (upon a lot of reflection) in the next 20 years is less than 20%, and that the entirety of the EA scene might be reducing it to say 15%.
This means that the entirety of the EA AI safety scene would help the EV of the world by ~5%.
On one hand, this is a whole lot. But on the other, I’m nervous that it’s not ambitious enough, for what could be one of the most [combination of well-resourced, well-meaning, and analytical/empirical] groups of our generation.
One thing I like about epistemic interventions is that the upper-bounds could be higher.
(There are some AI interventions that are more ambitious, but many do seem to be mainly about reducing x-risk by less than an order of magnitude, not increasing the steady-state potential outcome)
I’d also note here that an EV gain of 5% might not be particularly ambitious. It could well be the case that many different groups can do this—so it’s easier than it might seem if you think goodness is additive instead of multiplicative.
I want to see more discussion on how EA can better diversify and have strategically-chosen distance from OP/GV.
One reason is that it seems like multiple people at OP/GV have basically said that they want this (or at least, many of the key aspects of this).
A big challenge is that it seems very awkward for someone to talk and work on this issue, if one is employed under the OP/GV umbrella. This is a pretty clear conflict of interest. CEA is currently the main organization for “EA”, but I believe CEA is majority funded by OP, with several other clear strong links. (Board members, and employees often go between these orgs).
In addition, it clearly seems like OP/GV wants certain separation to help from their side. The close link means that problems with EA often spills over to the reputation of OP/GV.
I’d love to see some other EA donors and community members step up here. I think it’s kind of damning how little EA money comes from community members or sources other than OP right now. Long-term this seems pretty unhealthy.
One proposal is to have some “mini-CEA” that’s non-large-donor funded. This group’s main job would be to understand and act on EA interests that organizations funded by large donors would have trouble with.
I know Oliver Habryka has said that he thinks it would be good for the EA Forum to also be pulled away from large donors. This seems good to me, though likely expensive (I believe this team is sizable).
Another task here is to have more non-large-donor funding for CEA.
For large donors, one way of dealing with potential conflicts of interest would be doing funding in large blocks, like a 4-year contribution. But I realize that OP might sensibly be reluctant to do this at this point.
Also, related—I’d really hope that the EA Infrastructure Fund could help here, but I don’t know if this is possible for them. I’m dramatically more excited about large long-term projects on making EA more community-driven and independent, and/or well-managed, than I am the kinds of small projects they seem to fund. I don’t think they’ve ever funded CEA, despite that CEA might now represent the majority of funding on the direct EA community. I’d encourage people from this fund to think through this issue and be clear about what potential projects they might be excited for, around this topic.
Backing up a bit—it seems to me like EA is really remarkably powerless for what it is, outside of the OP/GV funding stream right now. This seems quite wrong to me, like large mistakes were made. Part of me think that positive change here is somewhat hopeless at this point (I’ve been thinking about this space for a few years now but haven’t taken much action because of uncertainty on this), but part of me also thinks that with the right cleverness or talent, there could be some major changes.
Another quick thought: This seems like a good topic for a “Debate Week”, in case anyone from that team is seeing this.
(To add clarity—I’m not suggesting that OP drops it’s funding of EA! It’s more that I think that non-OP donors should step up more, and that key EA services should be fairly independent.)
I’d love to see some other EA donors and community members step up here. I think it’s kind of damning how little EA money comes from community members or sources other than OP right now. Long-term this seems pretty unhealthy.
There was some prior relevant discussion in November 2023 in this CEA fundraising thread, such as my comment here about funder diversity at CEA. Basically, I didn’t think that there was much meaningful difference between a CEA that was (e.g.) 90% OP/GV funded vs. 70% OP/GV funded. So I think the only practical way for that percentage to move enough to make a real difference would be both an increase in community contributions/control and CEA going on a fairly severe diet.
As for EAIF, expected total grantmaking was ~$2.5MM for 2025. Even if a sizable fraction of that went to CEA, it would only be perhaps 1-2% of CEA’s 2023 budget of $31.4MM.
I recall participating in some discussions here about identifying core infrastructure that should be prioritized for broad-based funding for democratic and/or epistemic reasons. Identifying items in the low millions for more independent funding seems more realistic than meaningful changes in CEA’s funding base. The Forum strikes me as an obvious candidate, but a community-funded version would presumably need to run on a significantly leaner budget than I understand to be currently in place.
Basically, I didn’t think that there was much meaningful difference between a CEA that was (e.g.) 90% OP/GV funded vs. 70% OP/GV funded.
Personally, I’m optimistic that this could be done in specific ways that could be better than one might initially presume. One wouldn’t fund “CEA”—they could instead fund specific programs in CEA, for instance. I imagine that people at CEA might have some good ideas of specific things they could fund that OP isn’t a good fit for.
One complication is that arguably we’d want to do this in a way that’s “fair” to OP. Like, it doesn’t seem “fair” for OP to pay for all the stuff that both OP+EA agrees on, and EA only to fund the stuff that EA likes. But this really depends on what OP is comfortable with.
Lastly, I’d flag that CEA being 90% OP/GV funded really can be quite different than 70% in some important ways, still. For example, if OP/GV were to leave—then CEA might be able to go to 30% of its size—a big loss, but much better than 10% of its size.
Personally, I’m optimistic that this could be done in specific ways that could be better than one might initially presume. One wouldn’t fund “CEA”—they could instead fund specific programs in CEA, for instance. I imagine that people at CEA might have some good ideas of specific things they could fund that OP isn’t a good fit for.
That may be viable, although I think it would be better for both sides if these programs were not in CEA but instead in an independent organization. For the small-donor side, it limits the risk that their monies will just funge against OP/GV’s, or that OP/GV will influence how the community-funded program is run (e.g., through its influence on CEA management officials). On the OP/GV side, organizational separation is probably necessary to provide some of the reputational distance it may be looking for. That being said, given that small/medium donors have never to my knowledge been given this kind of opportunity, and the significant coordination obstacles involved, I would not characterize them not having taken it as indicative of much in particular.
~
More broadly, I think this is a challenging conversation without nailing down the objective better—and that may be hard for us on the Forum to do. Without any inside knowledge, my guess is that OP/GV’s concerns are not primarily focused on the existence of discrete programs “that OP isn’t a good fit for” or a desire not to fund them.
For example, a recent public comment from Dustin contain the following sentence: “But I can’t e.g. get SBF to not do podcasts nor stop the EA (or two?) that seem to have joined DOGE and started laying waste to USAID.” The concerns implied by that statement aren’t really fixable by the community funding discrete programs, or even by shelving discrete programs altogether. Not being the flagship EA organization’s predominant donor may not be sufficient for getting reputational distance from that sort of thing, but it’s probably a necessary condition.
I speculate that other concerns may be about the way certain core programs are run—e.g., I would not be too surprised to hear that OP/GV would rather not have particular controversialcontent allowed on the Forum, or have advocates for certain political positions admitted to EAGs, or whatever. I’m not going to name the content I have in mind in an attempt not to be drawn into an object-level discussion on those topics, but I wouldn’t want my own money being used to platform such content or help its adherents network either. Anyway, these types of issues can probably be fixed by running the program with community/other-donor funding in a separate organization, but these programs are expensive to run. And the community / non-OP/GV donors are not a monolithic constituency; I suspect that at least a significant minority of the community would share OP/GV’s concerns on the merits.
Lastly, I’d flag that CEA being 90% OP/GV funded really can be quite different than 70% in some important ways, still. For example, if OP/GV were to leave—then CEA might be able to go to 30% of its size—a big loss, but much better than 10% of its size.
I agree—the linked comment was focused more on the impact of funding diversity on conflicts of interest and cause prio. But the amount of smaller-EA-donor dollars to go around is limited,[1] and so we have to consider the opportunity cost of diverting them to fund CEA or similar meta work on an ongoing basis. OP/GV is usually a pretty responsible funder, so the odds of them suddenly defunding CEA without providing some sort of notice and transitional funding seems low.
For instance, I believe GWWC pledgers gave about $32MM/year on average from 2020-2022 [p. 12 of this impact assessment], and not all pledgers are EAs.
I think you bring up a bunch of good points. I’d hope that any concrete steps on this would take these sorts of considerations in mind.
> The concerns implied by that statement aren’t really fixable by the community funding discrete programs, or even by shelving discrete programs altogether. Not being the flagship EA organization’s predominant donor may not be sufficient for getting reputational distance from that sort of thing, but it’s probably a necessary condition.
I wasn’t claiming that this funding change would fix all of OP/GV’s concerns. I assume that would take a great deal of work, among many different projects/initiatives.
One thing I care about is that someone is paid to start thinking about this critically and extensively, and I imagine they’d be more effective if not under the OP umbrella. So one of the early steps to take is just trying to find a system that could help figure out future steps.
> I speculate that other concerns may be about the way certain core programs are run—e.g., I would not be too surprised to hear that OP/GV would rather not have particular controversialcontent allowed on the Forum, or have advocates for certain political positions admitted to EAGs, or whatever.
I think this raises an important and somewhat awkward point that levels of separation between EA and OP/GV would make it harder for OP/GV to have as much control over these areas, and there are times where they wouldn’t be as happy with the results.
Of course: 1. If this is the case, it implies that the EA community does want some concretely different things, so from the standpoint of the EA community, this would make funding more appealing. 2. I think in the big picture, it seems like OP/GV doesn’t want to be held as responsible for the EA community. Ultimately there’s a conflict here—on one hand, they don’t want to be seen as responsible for the EA community—on the other hand, they might prefer situations where they can have a very large amount of control over the EA community. I hope it can be understood that these two desires can’t easily go together. Perhaps they won’t be willing to compromise on the latter, but also will complain about the former. That might well happen, but I’d hope there could be a better arrangement made.
> OP/GV is usually a pretty responsible funder, so the odds of them suddenly defunding CEA without providing some sort of notice and transitional funding seems low.
I largely agree. That said, if I were CEA, I’d still feel fairly uncomfortable. When the vast majority of your funding comes from any one donor, you’ll need to place a whole lot of trust in them.
I’d imagine that if I were working within CEA, I’d be incredibly precautious not to upset OP or GV. I’d also imagine this to mess with my epistemics/communication/actions.
Also, of course, I’d flag that the world can change quickly. Maybe Trump will go on a push against EA one day, and put OP in an awkward spot, for example.
I want to see more discussion on how EA can better diversify and have strategically-chosen distance from OP/GV.
One reason is that it seems like multiple people at OP/GV have basically said that they want this (or at least, many of the key aspects of this).
I agree with the approach’s direction, but this premise doesn’t seem very helpful in shaping the debate. It doesn’t seem that there is a right level of funding for meta EA or that this is what we currently have.
My perception is that OP has specific goals, one of which is to reduce GCR risk. As there are not so many high absorbency funding opportunities and a lot of uncertainty in the field, they focus more on capacity building, of which EA has proven to be a solid investment in talent pipeline building.
If this is true, then the level of funding we are currently seeing is downstream from OP’s overall yearly spending and their goals. Other funders will come to very different conclusions as to why they would want to fund EA meta and to what extent.
If you’re a meta funder who agrees with GCR risks, you might see opportunities that either don’t want OPs’ money, that OP doesn’t want to fund, or that want to keep OPs’ funding under a certain bar. These are more neglected, but they are more cost-effective for you as they are not as fungible.
At the last, MCF funding diversification and the EA brand were the two main topics, but to me, meta-funding diversification seems much harder, especially for areas under the EA brand.
This is good to know. While mentioning MCF, I would bring up that it seems bad to me that MCF seems to be very much within the OP umbrella, as I understand it. I believe that it was funded by OP or CEA, and the people who set it up were employed by CEA, which was primarily funded by OP. Most of the attendees seem like people at OP or CEA, or else heavily funded by OP.
I have a lot of respect for many of these people and am not claiming anything nefarious. But I do think that this acts as a good example of the sort of thing that seems important for the EA community, and also that OP has an incredibly large amount of control over. It seems like an obvious potential conflict of interest.
Quickly: > I agree with the approach’s direction, but this premise doesn’t seem very helpful in shaping the debate.
Sorry, I don’t understand this. What is “the debate” that you are referring to?
I just meant the discussion you wanted to see; I probably used the wrong synonym.
This is good to know. While mentioning MCF, I would bring up that it seems bad to me that MCF seems to be very much within the OP umbrella, as I understand it. I believe that it was funded by OP or CEA, and the people who set it up were employed by CEA, which was primarily funded by OP. Most of the attendees seem like people at OP or CEA, or else heavily funded by OP.
I generally believe that EA is effective at being pragmatic, and in that regard, I think it’s important for the key organizations that are both giving and receiving funding in this area to coordinate, especially with topics like funding diversification. I agree that this is not the ideal world, but this goes back to the main topic.
I generally believe that EA is effective at being pragmatic, and in that regard, I think it’s important for the key organizations that are both giving and receiving funding in this area to coordinate, especially with topics like funding diversification. I agree that this is not the ideal world, but this goes back to the main topic.
For reference, I agree it’s important for these people to be meeting with each other. I wasn’t disagreeing with that.
However, I would hope that over time, there would be more people brought in who aren’t in the immediate OP umbrella, to key discussions of the future of EA. At least have like 10% of the audience be strongly/mostly independent or something.
I think its better to start something new. Reform is hard but no one is going to stop you from making a new charity. The EA brand isn’t in the best shape. Imo the “new thing” can take money from individual EAs but shouldn’t accept anything connected to OpenPhil/CEA/Dustin/etc.
If you start new you can start with a better culture.
I mean Dustin Moskovitz used to come on the forum and beg people to do earn to give yet I don’t think the number of donors has grown that much. More people should go to Jane Street and do Y-Combinator but it feels as though that’s taboo to say for some reason.
I have said this in other spaces since the FTX collapse: The original idea of EA, as I see it, was that it was supposed to make the kind of research work done at philanthropic foundations open and usable for well-to-do-but-not-Bill-Gates-rich Westerners. While it’s inadvisable to outright condemn billionaires using EA work to orient their donations for… obvious reasons, I do think there is a moral hazard in billionaires funding meta EA. Now, the most extreme policy would be to have meta EA be solely funded by membership dues (as plenty organizations are!). I’m not sure if that would really be workable for the amounts of money involved, but some kind of donation cap could be plausibly envisaged.
The original idea of EA, as I see it, was that it was supposed to make the kind of research work done at philanthropic foundations open and usable for well-to-do-but-not-Bill-Gates-rich Westerners
This part doesn’t resonate with me. I worked at 80k early on (~2014) and have been in the community for a long time. Then, I think the main thing was excitement over “doing good the most effectively”. The assumption was that most philanthropic foundations weren’t doing a good job—not that we wanted regular people to participate, specifically. I think then, most community members would be pretty excited about the idea of the key EA ideas growing as quickly as possible, and billionaires would help with that.
GiveWell specifically was started with a focus on smaller donors, but there was a always a separation between them and EA.
(I am of course more sympathetic to a general skepticism around any billionaire or other overwhelming donor. Though I’m personally also skeptical of most other donation options to other degrees—I want some pragmatic options that can understand the various strengths and weaknesses of different donors and respond accordingly)
GiveWell specifically was started with a focus on smaller donors, but there was a always a separation between them and EA.
… I’m confused by what you would mean by early EA then? As the history of the movement is generally told it started by the merger of three strands: GiveWell (which attempt to make charity effectiveness research available for well-to-do-but-not-Bill-Gates-rich Westerners), GWWC (which attempt to convince well-to-do-but-not-Bill-Gates-rich Westerners to give to charity too), and the rationalists and proto-longtermists (not relevant here).
Criticisms of ineffective charities (stereotypically, the Make a Wish Foundation) could be part of that, but they’re specifically the charities well-to-do-but-not-Bill-Gates-rich Westerners tend to donate to when they do donate, I don’t think people were going out claiming the biggest billionaire philanthropic foundations (like, say, well, the Bill Gates Foundation) didn’t knew what to do with their money.
Quickly: 1. Some of this gets into semantics. There are some things that are more “key inspirations for what was formally called EA” and other things that “were formally called EA, or called themselves EA.” GiveWell was highly influential around EA, but I think it was created before EA was coined, and I don’t think they publicly associated as “EA” for some time (if ever). 2. I think we’re straying from the main topic at this point. One issue is that while I think we disagree on some of the details/semantics of early EA, I also don’t think that matters much for the greater issue at hand. “The specific reason why the EA community technically started” is pretty different from “what people in this scene currently care about.”
Didn’t really want to in depth go beyond what @Ozzie Gooen already said and mentioning the event that originally prompted that line of thought, but added a link to @David Thorstad’s sequence on the subject.
Someone wrote to me in a PM that they think one good reason for EA donors not to have funded EA community projects was because OP was funding them, and arguably there are other more neglected projects.
I do think this is a big reason, and I was aware of this before. It’s a complex area.
At the same time, I think the current situation is really not the best, and can easily imagine healthier environments where motivated funders and community would have found good arrangements here.
I also take responsibility for not doing a better job around this (and more).
I really don’t like the trend of posts saying that “EA/EAs need to | should do X or Y”.
EA is about cost-benefit analysis. The phrases need and should implies binaries/absolutes and having very high confidence.
I’m sure there are thousands of interventions/measures that would be positive-EV for EA to engage with. I don’t want to see thousands of posts loudly declaring “EA MUST ENACT MEASURE X” and “EAs SHOULD ALL DO THING Y,” in cases where these mostly seem like un-vetted interesting ideas.
In almost all cases I see the phrase, I think it would be much better replaced with things like; ”Doing X would be high-EV” “X could be very good for EA” ”Y: Cost and Benefits” (With information in the post arguing the benefits are worth it) ”Benefits|Upsides of X” (If you think the upsides are particularly underrepresented)”
I think it’s probably fine to use the word “need” either when it’s paired with an outcome (EA needs to do more outreach to become more popular) or when the issue is fairly clearly existential (the US needs to ensure that nuclear risk is low). It’s also fine to use should in the right context, but it’s not a word to over-use.
Strong disagree. If the proponent of an intervention/cause area believes the advancement of it is extremely high EV such that they believe it is would be very imprudent for EA resources not to advance it, they should use strong language.
I think EAs are too eager to hedge their language and use weak language regarding promising ideas.
For example, I have no compunction saying that advancement of the Profit for Good (companies with charities in vast majority shareholder position) needs to be advanced by EA, in that I believe it not doing results in an ocean less counterfactual funding for effective charities, and consequently a significantly worse world.
First, I have a different issue with that phrase, as it’s not clear what “EA” is. To me, EA doesn’t seem like an agent. You can say, ”....CEA should” or ”...OP should”.
Normally, I prefer one says “I think X should”. There are some contexts, specifically small ones (talking to a few people, it’s clearly conversational) where saying, “X should do Y” clearly means “I feel like X should do Y, but I’m not sure”. And there are some contexts where it means “I’m extremely confident X should do Y”.
For example, there’s a big difference between saying “X should do Y” to a small group of friends, when discussing uncertain claims, and writing a mass-market book titled “X should do Y”.
There are a couple of strong “shoulds” in the EA Handbook (I went through it over the last two months as part of an EA Virtual program) and they stood out to me as the most disagreeable part of EA philosophy that was presented.
I’ve substantially revised my views on QURI’s research priorities over the past year, primarily driven by the rapid advancement in LLM capabilities.
Previously, our strategy centered on developing highly-structured numeric models with stable APIs, enabling:
Formal forecasting scoring mechanisms
Effective collaboration between human forecasting teams
Reusable parameterized world-models for downstream estimates
However, the progress in LLM capabilities has updated my view. I now believe we should focus on developing and encouraging superior AI reasoning and forecasting systems that can:
Generate high-quality forecasts on-demand, rather than relying on pre-computed forecasts for scoring
Produce context-specific mathematical models as needed, reducing the importance of maintaining generic mathematical frameworks
Leverage repositories of key insights, though likely not in the form of formal probabilistic mathematical models
This represents a pivot from scaling up traditional forecasting systems to exploring how we can enhance AI reasoning capabilities for forecasting tasks. The emphasis is now on dynamic, adaptive systems rather than static, pre-structured models.
(I rewrote with Claude, I think it’s much more understandable now)
Generate high-quality forecasts on-demand, rather than relying on pre-computed forecasts for scoring
Leverage repositories of key insights, though likely not in the form of formal probabilistic mathematical models
To be clear, I think there’s a lot of batch intellectual work we can do before users ask for specific predictions. So “Generating high-quality forecasts on-demand” doesn’t mean “doing all the intellectual work on-demand.”
However, I think there’s a broad set of information that this batch intellectual work could look like. I used to think that this batch work would produce a large set of connect mathematic models. Now I think we probably want something very compressed. If a certain mathematical model can easily be generated on-demand, then there’s not much of a benefit to having it made and saved ahead of time. However, I’m sure there are many crucial insights that are both expensive to find, and would be useful for many questions that LLM users ask about.
So instead of searching for and saving math models, a system might do a bunch of intellectual work and save statements like, ”When estimating the revenue of OpenAI, remember crucial considerations [A] and [B]. Also, a surprisingly good data source for this is Twitter user ai-gnosis-34.”
A lot of user-provided forecasts or replies should basically be the “last mile” or intellectual work. All the key insights are already found, now there just needs to be a bit of customization for the very specific questions someone has.
When EA was starting, there was a small amount of talent, and a smaller amount of funding. As one might expect, things went slowly for the first few years.
Then once OP decided to focus on X-risks, there was ~$8B potential funding, but still fairly little talent/capacity. I think the conventional wisdom then was that we were unlikely to be bottlenecked by money anytime soon, and lots of people were encouraged to do direct work.
Then FTX Future Fund came in, and the situation got even more out-of-control. ~Twice the funding. Projects got more ambitious, but it was clear there were significant capacity (funder and organization) constraints.
Then (1) FTX crashed, and (2) lots of smart people came into the system. Project capacity grew, AI advances freaked out a lot of people, and successful community projects helped train a lot of smart young people to work on X-risks.
But funding has not kept up. OP has been slow to hire for many x-risk roles (AI safety, movement building, outreach / fundraising). Other large funders have been slow to join in.
So now there’s a crunch for funding. There are a bunch of smart-seeming AI people now who I bet could have gotten funding during the FFF, likely even before then with OP, but are under the bar now.
I imagine that this situation will eventually improve, but of course, it would be incredibly nice if it could happen sooner. It seems like EA leadership eventually fix things, but it often happens slower than is ideal, with a lot of opportunity loss in that time.
Opportunistic people can fill in the gaps. Looking back, I think more money and leadership in the early days would have gone far. Then, more organizational/development capacity during the FFF era. Now, more funding seems unusually valuable.
If you’ve been thinking about donating to the longtermist space, specifically around AI safety, I think it’s likely that funding this year will be more useful than funding in the next 1-3 years. (Of course, I’d recommend using strong advisors or giving to funds, instead of just choosing directly, unless you can spend a fair bit of time analyzing things).
If you’re considering entering the field as a nonprofit employee, heed some caution. I still think the space can use great talent, but note that this is an unusually competitive time to get many paid roles or to get nonprofit grants.
If you have limited time to investigate / work with, I’d probably recommend either the LTFF or choosing a regranter you like at Manifund.
If you have a fair bit more time, and ideally the expectation of more money in the future, then I think a lot of small-to-medium (1-10 employee) organizations can use some long-term, high-touch donors. Honestly this may settle more down to fit / relationships than identifying the absolute best org—as long as it’s funded by one of the groups listed above or OP, as money itself is a bit fungible between orgs.
I think a lot of nonprofits have surprisingly few independent donors, or even strong people that can provide decent independent takes. I might write more about this later.
(That said, there are definitely ways to be annoying / a hindrance, as an active donor, so try to be really humble here if you are new to this)
EA seems to have been doing a pretty great job attracting top talent from the most prestigious universities. While we attract a minority of the total pool, I imagine we get some of the most altruistic+rational+agentic individuals.
If this continues, it could be worth noting that this could have significant repercussions for areas outside of EA; the ones that we may divert them from. We may be diverting a significant fraction of the future “best and brightest” in non-EA fields.
If this seems possible, it’s especially important that we do a really, really good job making sure that we are giving them good advice.
One of my main frustrations/criticisms with a lot of current technical AI safety work is that I’m not convinced it will generalize to the critical issues we’ll have at our first AI catastrophes ($1T+ damage).
From what I can tell, most technical AI safety work is focused on studying previous and current LLMs. Much of this work is very particular to specific problems and limitations these LLMs have.
I’m worried that the future decisive systems won’t look like “single LLMs, similar to 2024 LLMs.” Partly, I think it’s very likely that these systems will be ones made up of combinations of many LLMs and other software. If you have a clever multi-level system, you get a lot of opportunities to fix problems of the specific parts. For example, you can have control systems monitoring LLMs that you don’t trust, and you can use redundancy and checking to investigate outputs you’re just not sure about. (This isn’t to say that these composite systems won’t have problems—just that the problems will look different to those of the specific LLMs).
Here’s an analogy: Imagine that researchers had 1960s transistors but not computers, and tried to work on cybersecurity, in preparation of future cyber-disasters in the coming decades. They want to be “empirical” about it, so they go along investigating all the failure modes of 1960s transistors. They successfully demonstrate that in extreme environments transistors fail, and also that there are some physical attacks that could be done on the transistor level.
But as we know now, almost all of this has either been solved on the transistor level, or on levels shortly above the transistors that do simple error management. Intentional attacks on the transistor level are possible, but incredibly niche compared to all of the other cybersecurity capabilities.
So just as understanding 1960s transistors really would not get you far towards helping at all with future cybersecurity challenges, it’s possible that understanding 2024 LLM details won’t help with future 2030 composite AI system disasters.
(John Wentworth and others refer to much of this as the Streetlight effect. I think that specific post is too harsh, but I think I sympathize with the main frustration.)
All that said, here are some reasons to still do the LLM research anyway. Some don’t feel great, but might still make it worthwhile.
There’s arguably not much else we can do now.
While we’re waiting to know how things will shape up, this is the most accessible technical part we can work on.
Having a research base skilled with empirical work on existing LLMs will be useful later on, as we could re-focus it to whatever comes about in the future.
There’s some decent chance that future AI disasters will come from systems that look a lot like modern “LLM-only” systems. Perhaps these disasters will happen in the next few years, or perhaps AI development will follow a very specific path.
This research builds skills that are generally useful later—either to work in AI companies to help them do things safely, or to make a lot of money.
It’s good to have empirical work, because it will raise the respect/profile of this sort of thinking within the ML community.
I’m not saying I could do better. This is one reason why I’m not exactly working in on technical AI safety. I have been interested in strategy in the area (which feels more tractable to me), and have been trying to eye opportunities for technical work, but am still fairly unsure of what’s best at this point.
I think the main challenge is that it’s just fundamentally hard to prepare for a one-time event with few warning shots (i.e. the main situation we’re worried about), several years in the future, in a fast-moving technical space. This felt clearly true 10 years ago, before there were language models that seemed close to TAI. I feel like it’s become easier since to overlook this bottleneck, as there’s clearly a lot of work we can do with LLMs that naively seems interesting. But that doesn’t mean it’s no longer true—it might still very much be the case that things are so early that useful safety empirical technical work is very difficult to do.
(Note: I have timelines for TAI that are 5+ years out. If your timelines are shorter, it would make more sense that understanding current LLMs would help.)
A large reason to focus on opaque components of larger systems is that difficult-to-handle and existentially risky misalignment concerns are most likely to occur within opaque components rather than emerge from human built software.
I don’t see any plausible x-risk threat models that emerge directly from AI software written by humans? (I can see some threat models due to AIs building other AIs by hand such that the resulting system is extremely opaque and might takeover.)
In the comment you say “LLMs”, but I’d note that a substantial fraction of this research probably generalizes fine to arbitrary DNNs trained with something like SGD. More generally, various approaches that work for DNNs trained with SGD plausibly generalize to other machine learning approaches.
A large reason to focus on opaque components of larger systems is that difficult-to-handle and existentially risky misalignment concerns are most likely to occur within opaque components rather than emerge from human built software.
Yep, this sounds positive to me. I imagine it’s difficult to do this well, but to the extent it can be done, I expect such work to generalize more than a lot of LLM-specific work.
> I don’t see any plausible x-risk threat models that emerge directly from AI software written by humans?
I don’t feel like that’s my disagreement. I’m expecting humans to create either [dangerous system that’s basically one black-box LLM] or [something very different that’s also dangerous, like a complex composite system]. I expect AIs can also make either system.
Some musicians have multiple alter-egos that they use to communicate information from different perspectives. MF Doom released albums under several alter-egos; he even used these aliases to criticize his previous aliases.
Some musicians, like Madonna, just continued to “re-invent” themselves every few years.
Youtube personalities often feature themselves dressed as different personalities to represent different viewpoints.
It’s really difficult to keep a single understood identity, while also conveying different kinds of information.
Narrow identities are important for a lot of reasons. I think the main one is predictability, similar to a company brand. If your identity seems to dramatically change hour to hour, people wouldn’t be able to predict your behavior, so fewer could interact or engage with you in ways they’d feel comfortable with.
However, narrow identities can also be suffocating. They restrict what you can say and how people will interpret that. You can simply say more things in more ways if you can change identities. So having multiple identities can be a really useful tool.
Sadly, most academics and intellectuals can only really have one public identity.
---
EA researchers currently act this way.
In EA, it’s generally really important to be seen as calibrated and reasonable, so people correspondingly prioritize that in their public (and then private) identities. I’ve done this. But it comes with a cost.
One obvious (though unorthodox) way around this is to allow researchers to post content either under aliases. It could be fine if the identity of the author is known, as long as readers can keep these aliases distinct.
I’ve been considering how to best do this myself. My regular EA Forum name is just “Ozzie Gooen”. Possible aliases would likely be adjustments to this name.
These would be used to communicate in very different styles, with me attempting what I’d expect readers to expect of those styles.
(Normally this is done to represent viewpoints other than what they have, but sometimes it’s to represent viewpoints they have, but wouldn’t normally share)
I can’t seem to find much EA discussion about [genetic modification to chickens to lessen suffering]. I think this naively seems like a promising area to me. I imagine others have investigated and decided against further work, I’m curious why.
“I agree with Ellen that legislation / corporate standards are more promising.
I’ve asked if the breeders would accept $ to select on welfare, & the answer was no b/c it’s inversely correlated w/ productivity & they can only select on ~2 traits/generation.”
Dang. That makes sense, but it seems pretty grim. The second half of that argument is, “We can’t select for not-feeling-pain, because we need to spend all of our future genetic modification points on the chickens getting bigger and growing even faster.”
I’m kind of surprised that this argument isn’t at all about the weirdness of it. It’s purely pragmatic, from their standpoint. “Sure, we might be able to stop most of the chicken suffering, but that would increase costs by ~20% or so, so it’s a non-issue”
20% of the global cost of growing chickens is probably in the order of at least ~$20B, which is much more than the global economy is willing to spend on animal welfare.
As mentioned in the other comment, I think it’s extremely unlikely that there is a way to stop “most” of the chicken suffering while increasing costs by only ~20%.
Some estimate the better chicken commitment already increases costs by 20% (although there is no consensus on that, and factory farmers estimate 37.5%), and my understanding is that it doesn’t stop most of the suffering, but “just” reduces it a lot.
Has there been any discussion of improving chicken breeding using GWAS or similar?
Even if welfare is inversely correlated with productivity, I imagine there are at least a few gene variants which improve welfare without hurting productivity. E.g. gene variants which address health issues due to selective breeding.
Also how about legislation targeting the breeders? Can we have a law like: “Chickens cannot be bred for increased productivity unless they meet some welfare standard.”
Note that prohibiting breeding that causes suffering is different to encouraging breeding that lessens suffering, and that selective breeding is different to gene splicing, etc., which I think is what is typically meant by genetic modification.
Naively, I would expect that suffering is extremely evolutionarily advantageous for chickens in factory farm conditions, so chickens that feel less suffering will not grow as much meat (or require more space/resources). For example, based on my impression that broiler chickens are constantly hungry, I wouldn’t be surprised if they would try to eat themselves unless they felt pain when doing so. But this is a very uninformed take based on a vague understanding of what broiler chickens are optimized for, which might not be true in practice.
I think this idea might be more interesting to explore in less price-sensitive contexts, where there’s less evolutionary pressure and animals live in much better conditions, mostly animals used in scientific research. But of course it would help much fewer animals who usually suffer much less.
It was mentioned at the Constellation office that maybe animal welfare people who are predisposed to this kind of weird intervention are working on AI safety instead. I think this is >10% correct but a bit cynical; the WAW people are clearly not afraid of ideas like giving rodents contraceptives and vaccines. My guess is animal welfare is poorly understood and there are various practical problems like preventing animals that don’t feel pain from accidentally injuring themselves constantly. Not that this means we shouldn’t be trying.
Quick thought. Maybe people anticipate this being blocked by governments because it “seems like playing god” etc. I know that would be hypocritical given the breeding already used to make them overweight etc. But it seems to be the way a lot of people see this.
By coincidence, I just came across this layer-hen genetics project that got funding from OP. I don’t know much about the work or how promising it might be.
One thing that comes to mind is that this seems like a topic a lot of people care about, and there’s a lot of upvotes & agreements, but there also seemed to be a surprising lack of comments, overall.
I’ve heard from others elsewhere that they were nervous about giving their takes, because it’s a sensitive topic.
Obviously I’m really curious about what, if anything, could be done, to facilitate more discussion on these sorts of issues. I think they’re important to grapple with, and would hate it if they just weren’t talked about due to awkwardness.
One obvious failure mode is that I think these discussions can get heated and occasionally result in really bad conversation. But having incredibly little discussion seems almost more unnerving to me.
I imagine we want environments where: - People feel safe posing what they believe - Conversation doesn’t spiral out-of-control
As some of you may know, I post a fair bit on Facebook, and did make some posts there about some of these topics. Many of these are private to my network, and some get a fair bit of conversation that’s not on the EA Forum—often it’s clear that many people find this more comfortable. But obviously this particular setup doesn’t scale very well.
Hey Ozzie, a few quick notes on why I react but try not to comment on community based stuff these days:
I try to limit how many meta-level comments I make. In general I’d like to see more object-level discussion of things and so I’m trying (to mixed success) to comment mostly about cause areas directly.
Partly it’s a vote for the person I’d like to be. If I talk about community stuff, part of my headspace will be thinking about it for the next few days. (I fully realize the irony of making this comment.)
It’s emotionally tricky since I feel responsibility for how others react. I know how loaded this topic was for a younger me, and I feel an obligation to make younger me feel welcome
These conversations often feel aspirational and repetitive. Like “there should be more X” is too simple. Whereas something like “there should be more X. Y org should be responsible for it. Tradeoffs may be Z. Failure modes are A, B, and C.” is concrete enough to get somewhere.
One potential weakness is that I’m curious if it promotes the more well-known charities due to the voting system. I’d assume that these are somewhat inversely correlated with the most neglected charities.
Related, I’m curious if future versions could feature specific subprojects/teams within charities. “Rethink Priorities” is a rather large project compared to “PauseAI US”, I assume it would be interesting if different parts of it were put here instead.
(That said, in terms of the donation, I’d hope that we could donate to RP as a whole and trust RP to allocate it accordingly, instead of formally restricting the money, which can be quite a hassle in terms of accounting)
One potential weakness is that I’m curious if it promotes the more well-known charities due to the voting system. I’d assume that these are somewhat inversely correlated with the most neglected charities.
I guess this isn’t necessarily a weakness if the more well-known charities are more effective? I can see the case that: a) they might not be neglected in EA circles, but may be very neglected globally compared to their impact and that b) there is often an inverse relationship between tractability/neglectedness and importance/impact of a cause area and charity. Not saying you’re wrong, but it’s not necessarily a problem.
Furthermore, my anecdotal take from the voting patterns as well as the comments on the discussion thread seem to indicate that neglectedness is often high on the mind of voters—though I admit that commenters on that thread are a biased sample of all those voting in the election.
It can be a bit underwhelming if an experiment to try to get the crowd’s takes on charities winds up determining to, “just let the current few experts figure it out.”
Is it underwhelming? I guess if you want the donation election to be about spurring lots of donations to small, spunky EA-startups working in weird-er cause areas, it might be, but I don’t think that’s what I understand the intention of the experiment to be (though I could be wrong).
My take is that the election is an experiment with EA democratisation, where we get to see what the community values when we do a roughly 1-person-1-ballot system instead of those-with-the-moeny decide system which is how things work right now. Those takeaways seem to be:
The broad EA community values Animal Welfare a lot more than the current major funders
The broad EA community sees value in all 3 of the ‘big cause areas’ with high-scoring charities in Animal Welfare, AI Safety, and Global Health & Development.
I (with limited information) think the EA Animal Welfare Fund is promising, but wonder how much of that matches the intention of this experiment. It can be a bit underwhelming if an experiment to try to get the crowd’s takes on charities winds up determining to, “just let the current few experts figure it out.” Though I guess, that does represent a good state of the world. (The public thinks that the current experts are basically right)
When I hear of entrepreneurs excited about prediction infrastructure making businesses, I feel like they gravitate towards new prediction markets or making new hedge funds.
I really wish it were easier to make new insurance businesses (or similar products). I think innovative insurance products could be a huge boon to global welfare. The very unfortunate downside is that there’s just a ton of regulation and lots of marketing to do, even in cases where it’s a clear win for consumers.
Ideally, it should be very easy and common to get insurance for all of the key insecurities of your life.
Having children with severe disabilities / issues
Having business or romantic partners defect on you
Having your dispreferred candidate get elected
Increases in political / environmental instability
Some unexpected catastrophe will hit a business
Nonprofits losing their top donor due to some unexpected issue with said donor (i.e. FTX)
I think a lot of people have certain issues that both:
They worry about a lot
They over-weight the risks of these issues
In these cases, insurance could be a big win!
In a better world, almost all global risks would be held primarily by asset managers / insurance agencies. Individuals could have highly predictable lifestyles.
(Of course, some prediction markets and other markets can occasionally be used for this purpose as well!)
Some of these things are fundamentally hard to insure against, because of information asymmetries / moral hazard.
e.g. insurance against donor issues would disproportionately be taken by people who had some suspicions about their donors, which would drive up prices, which would get more people without suspicions to decline taking insurance, until the market was pretty tiny with very high prices and a high claim rate. (It would also increase the incentives to commit fraud to give, which seems bad.)
Some of these harms seem of a sort that does not really feel compensable with money. While romantic partner’s defection might create some out-of-pocket costs, but I don’t think the knowledge that I’d get some money out of my wife defecting would make me feel any better about the possibility!
Also, I’d note that some of the harms are already covered by social insurance schemes to a large extent. For instance, although parents certainly face a lot of costs associated with “[h]aving children with severe disabilities / issues,” a high percentage of costs in the highest-cost scenarios are already borne by the public (e.g., Medicaid, Social Security/SSI, the special education system, etc.) or by existing insurers (e.g., employer-provided health insurance). So I’d want to think more about the relative merits of novel private-sector insurance schemes versus strengthening the socialized schemes.
While romantic partner’s defection might create some out-of-pocket costs, but I don’t think the knowledge that I’d get some money out of my wife defecting would make me feel any better about the possibility
Consider this, as examples of where it might be important: 1. You are financially dependent on your spouse. If they cheated on you, you would likely want to leave them, but you wouldn’t want to be trapped due to finances. 2. You’re nervous about the potential expenses of a divorce.
I think that this situation is probably a poor fit for insurance at this point, just because of moral risks that would happen, but perhaps one day it might be viable to some extent.
> So I’d want to think more about the relative merits of novel private-sector insurance schemes versus strengthening the socialized schemes.
I’m all for improvements on socialized schemes too. No reason not for both strategies to be tested and used. In theory, insurance could be much easier and faster to be implemented. It can take ages for nation-wide reform to happen.
A few junior/summer effective altruism related research fellowships are ending, and I’m getting to see some of the research pitches.
Lots of confident-looking pictures of people with fancy and impressive sounding projects.
I want to flag that many of the most senior people I know around longtermism are really confused about stuff. And I’m personally often pretty skeptical of those who don’t seem confused.
So I think a good proposal isn’t something like, “What should the EU do about X-risks?” It’s much more like, “A light summary of what a few people so far think about this, and a few considerations that they haven’t yet flagged, but note that I’m really unsure about all of this.”
Many of these problems seem way harder than we’d like for them to be, and much harder than many seem to assume at first. (perhaps this is due to unreasonable demands for rigor, but an alternative here would be itself a research effort).
I imagine a lot of researchers assume they won’t stand out unless they seem to make bold claims. I think this isn’t true for many EA key orgs, though it might be the case that it’s good for some other programs (University roles, perhaps?).
Not sure how to finish this post here. I think part of me wants to encourage junior researchers to lean on humility, but at the same time, I don’t want to shame those who don’t feel like they can do so for reasons of not-being-homeless (or simply having to leave research). I think the easier thing is to slowly spread common knowledge and encourage a culture where proper calibration is just naturally incentivized.
Do people know what’s going on with the EA Funds financial position? Some of the graphs look a bit worrying. But I’ve also noticed some (what seem like) inconsistencies, so I suspect that some key data is missing or something.
The following graph seems incorrect. It seems like it just wasn’t updated since Dec 2023. And I’m curious if there really was the ~$2M jump in the LTFF that month.
This has been on the stack to look into for a few weeks. I think we might just take the graphs down until we’re confident they are all accurate. They were broken (and then fixed) after we moved accounting systems, but it looks like they might have broken again.
If it happens to be the case that the numbers of active donors are continuing to go down, I’d of course be quite curious for more discussion in why that is. I think a survey could be useful here, or maybe just a EA Forum Question. Maybe even something to just all potential donors, like, “What are your main reservations?”
The loss of active donors seems like a big deal to me. I’m really curious what’s happening there.
1. Some donors probably just changed targets, for example, to Manifund or other new funds 2. I’m sure some got cynical or disenchanted by the movement or something 3. Maybe some got hurt financially, from FTX / a crypto bust?
> EAIF currently has $3.3M in available funds. So far over 2024 EAIF has made grants worth $1.1M, and I expect this to be around $1.4M by the end of 2024.
So it seems like EAIF at least is in a decent position.
I’m not sure how this coincides with the above numbers though. Just eyeballing it—maybe, while donations in have been low, donations out have been lower, so there’s been a surplus of funding.
Quick idea - I think that “signs of empathy/appreciation/politeness” get undervalued online. Obviously, LLMs can be good at this.
I sort of like the idea of things like:
We rank public commenters for how polite / appreciative they are. We show the top ~10, maybe give weekly rewards. Ranking could be done automatically.
We formally estimate the value/disvalue that comments provide in terms of [encouragement/discouragement], and reveal/estimate this
I’d like to see my metrics improving over time. Like, to show that this year, my comments have made people feel more positive than they did last year.
It seems easy enough for commenters to learn to spend more effort here. They could work with LLMs to do better. Arguably, there could be some fairly-cheap wins here.
There’s obviously the potential problem of encouraging niceness that’s not genuine, then devaluing all of it. But I think this could also be done well, and such a thing might lead to a more positive community.
I think for many people, positive comments would be much less meaningful if they were rewarded/quantified, because you would doubt that they’re genuine. (Especially if you excessively feel like an imposter and easily seize onto reasons to dismiss praise.)
I disagree with your recommendations despite agreeing that positive comments are undersupplied.
I’d quickly flag: 1. Any decent intervention should be done experimentally. It’s not like there would be “one system, hastily put-together, in place forever.” More like, early work would try out some things and see what the response is like in practice. I imagine that many original ideas would be mediocre, but with the right modifications and adjustments to feedback, it’s possible to make something decent. 2. I think that positive comments are often already rewarded—and that’s a major reason people give them. But I don’t think this is necessarily a bad thing. My quick guess is that this is a situation of adjusting incentives—certain incentive structures would encourage certain classes of good and bad behaviors, so it’s important to continue to tune these. Right now we have some basic incentives that were arrived at by default, and in my opinion are quite unsophisticated (people are incentivized to be extra nice to people who are powerful and who will respond, and mean to people in the outgroup). I think semi-intentional work can improve this, but I realize it would need to be done well.
On my side it feels a bit like, ”We currently have an ecosystem of very mediocre incentives, that produce the current results. It’s possible to set up infrastructure to adjust those incentives and experiment with what those results would be. I’m optimistic that this problem is both important enough and tractable enough for some good efforts to work on.”
I upvoted and didn’t disagree-vote, because I generally agree that using AI to nudge online discourse in more productive directions seems good. But if I had to guess where disagree votes come from, it might be a combination of:
It seems like we probably want politeness-satisficing rather than politeness-maximizing. (This could be consistent with some versions of the mechanism you describe, or a very slightly tweaked version).
There’s a fine line between politiness-moderating and moderating the substance of ideas that make people uncomfortable. Historically, it has been hard to police this line, and given the empirically observable political preferences of LLMs, it’s reasonable for people who don’t share those preferences to worry that this will disadvantage them (though I expect this bias issue to get better over time, possibly very soon)
There is a time and place for spirited moral discourse that is not “polite,” because the targets of the discourse are engaging in highly morally objectionable action, and it would be bad to always discourage people from engaging in such discourse.*
*This is a complicated topic that I don’t claim to have either (a) fully coherent views on, or (b) have always lived up to the views I do endorse.
As AI improves, there’s a window for people to get involved and make changes regarding AI alignment and policy.
The window arguably starts small, then widens as it becomes clearer what to do.
But at some point it gets too close to TAI, I expect that the window narrows. The key decisions get made by a smaller and smaller group of people, and these people have less ability get help from others, given the quickening pace of things.
For example, at T minus 1 month, there might ultimately be a group of 10 people with key decision-making authority on the most powerful and dangerous AI project. The ‘room where it happens’ has become quite small.
This is somewhat similar to tech projects. An ambitious initiative will start with a few people, then slowly expand to hundreds. But over time decisions get locked into place. Eventually the project goes into “bug fixing” stage, then a marketing/release phase, after which the researchers will often get re-allocated. Later execs can decide to make decisions like killing the project.
One thing this means is that I expect that there could be a decent amount of time where many of us have “basically nothing to do” about AI safety, even though TAI still hasn’t happened. I imagine it could still be good for many people to try to grow capital and influence other people in order to create positive epistemics/lock-in, but the key AI safety issues belong to a narrow group.
If it is the case that TAI will happen in 2 years, for example, I imagine very few people will be able to do much at all at this point, for the key aspects of AI alignment, especially if you’re not actively working in the field.
Obviously, roles working on legislation with at 5+ time horizon will stop being relevant relevant over 5 years before TAI. And people working in tech at non-leading labs might not be relevant once it’s clear these are non-leading labs.
(I don’t mean to discourage people. Rather, I think it’s important to realize when one should strive hard, and when one should chill out a bit and focus on other issues. Personally I’m sort of looking forward to the time where I’m extremely confident that I can’t contribute much to the most major things. It’s basically the part of the project where it’s ‘in someone else’s hands’.)
Hmm maybe it could still be good to try things in case timelines are a bit longer or an unexpected opportunity arises? For example, what if you thought it was 2 years but actually 3-5?
I wasn’t trying to make the argument that it would definitely be clear when this window closes. I’m very unsure of this. I also expect that different people have different beliefs, and that it makes sense for them to then take corresponding actions.
Could/should altruistic activist investors buy lots of Twitter stock, then pressure them to do altruistic things?
---
So, Jack Dorsey just resigned from Twitter.
Some people on Hacker News are pointing out that Twitter has had recent issues with activist investors, and that this move might make those investors happy.
From a quick look… Twitter stock really hasn’t been doing very well. It’s almost back at its price in 2014.
Square, Jack Dorsey’s other company (he was CEO of two), has done much better. Market cap of over 2x Twitter ($100B), huge gains in the last 4 years.
I’m imagining that if I were Jack… leaving would have been really tempting. On one hand, I’d have Twitter, which isn’t really improving, is facing activist investor attacks, and worst, apparently is responsible for global chaos (of which I barely know how to stop). And on the other hand, there’s this really tame payments company with little controversy.
Being CEO of Twitter seems like one of the most thankless big-tech CEO positions around.
That sucks, because it would be really valuable if some great CEO could improve Twitter, for the sake of humanity.
One small silver lining is that the valuation of Twitter is relatively small. It has a market cap of $38B. In comparison, Facebook/Meta is $945B and Netflix is $294B.
So if altruistic interests really wanted to… I imagine they could become activist investors, but like, in a good way? I would naively expect that even with just 30% of the company you could push them to do positive things. $12B to improve global epistemics in a major way.
The US could have even bought Twitter for 4% of the recent $1T infrastructure bill. (though it’s probably better that more altruistic ventures do it).
If middle-class intellectuals really wanted it enough, theoretically they could crowdsource the cash.
I think intuitively, this seems like clearly a tempting deal.
I’d be curious if this would be a crazy proposition, or if this is just not happening due to coordination failures.
Admittingly, it might seem pretty weird to use charitable/foundation dollars on “Buying lots of Twitter” instead of direct aid, but the path to impact is pretty clear.
One futarchy/prediction market/coordination idea I have is to find some local governments and see if we could help them out by incorporating some of the relevant techniques.
This could be neat if it could be done as a side project. Right now effective altruists/rationalists don’t actually have many great examples of side projects, and historically, “the spare time of particularly enthusiastic members of a jurisdiction” has been a major factor in improving governments.
Berkeley and London seem like natural choices given the communities there. I imagine it could even be better if there were some government somewhere in the world that was just unusually amenable to both innovative techniques, and to external help with them.
Given that EAs/rationalists care so much about global coordination, getting concrete experience improving government systems could be interesting practice.
There’s so much theoretical discussion of coordination and government mistakes on LessWrong, but very little discussion of practical experience implementing these ideas into action.
(This clearly falls into the Institutional Decision Making camp)
I have a bunch of friends/colleagues who are either trying to slow AGI down (by stopping arms races) or align it before it’s made (and would much prefer it be slowed down).
Then I have several friends who are actively working to *speed up* AGI development. (Normally just regular AI, but often specifically AGI)[1]
Then there are several people who are apparently trying to align AGI, but who are also effectively speeding it up, but they claim that the trade-off is probably worth it (to highly varying degrees of plausibility, in my rough opinion).
In general, people seem surprisingly chill about this mixture? My impression is that people are highly incentivized to not upset people, and this has led to this strange situation where people are clearly pushing in opposite directions on arguably the most crucial problem today, but it’s all really nonchalant.
[1] To be clear, I don’t think I have any EA friends in this bucket. But some are clearly EA-adjacent.
Thinking about the idea of an “Evaluation Consent Policy” for charitable projects.
For example, for a certain charitable project I produce, I’d explicitly consent to allow anyone online, including friends and enemies, to candidly review it to their heart’s content. They’re free to use methods like LLMs to do this.
Such a policy can give limited consent. For example:
You can’t break laws when doing this evaluation
You can’t lie/cheat/steal to get information for this evaluation
Consent is only provided for under 3 years
Consent is only provided starting in 5 years
Consent is “contagious” or has a “share-alike provision”. Any writing that takes advantage of this policy, must itself have a consent policy that’s at least as permissive. If someone writes a really bad evaluation, they agree that you and others are correspondingly allowed to critique this evaluation.
The content must score less than 6⁄10 when run against Claude on a prompt roughly asking, “Is this piece written in a way that’s unnecessarily inflammatory?”
Consent can be limited to a certain group of people. Perhaps you reject certain inflammatory journalists, for example. (Though these might be the people least likely to care about getting your permission anyway)
This would work a lot like Creative Commons or Software Licenses. However, it would cover different territory, and (at this point at least) won’t be based on legal enforcement.
Criticisms:
“Why do we need this? People are already allowed to critique anything they want.”
While this is technically true, I think it would frequently break social norms. There are a lot of cases where people would get upset if their projects were provided any negative critique, even if it came with positive points. This would act as a signal that the owners might be particularly okay with critique. I think we live in a society that’s far from maximum-candidness, and it’s often difficult to tell where candidness would be accepted—so explicit communication could be useful.
“But couldn’t people who sign such a policy just attack evaluators anyway?”
I don’t think an explicit policy here will be a silver bullet, but I think it would help. I expect that a boss known for being cruel wouldn’t be trusted if they provided such a policy, but I imagine many other groups would be. Ideally there could be some common knowledge about which people/organizations fail to properly honor their policies. I don’t think this would work for Open Philanthropy that much (in the sense that effective altruists might expect OP to not complain publicly, but later not fund the writer’s future projects), but it could for many smaller orgs (that would have much less secretive power over public evaluators/writers)
---
Anyway, I’m interested in thoughts by this community.
There seem to be several longtermist academics who plan to spend the next few years (at least) investigating the psychology of getting the public to care about existential risks.
This is nice, but I feel like what we really could use are marketers, not academics. Those are the people companies use for this sort of work. It’s somewhat unusual that marketing isn’t much of a respected academic field, but it’s definitely a highly respected organizational one.
There are at least a few people in the community with marketing experience and an expressed desire to help out. The most recent example that comes to mind is this post.
If anyone reading this comment knows people who are interested in the intersection of longtermism and marketing, consider telling them about EA Funds! I can imagine the LTFF or EAIF being very interested in projects like this.
(That said, maybe one of the longtermist foundations should consider hiring a marketing consultant?)
Yep, agreed. Right now I think there are very few people doing active work in longtermism (outside of a few orgs that have people for that org), but this seems very valuable to improve upon.
Around discussions of AI & Forecasting, there seems to be some assumption like:
1. Right now, humans are better than AIs at judgemental forecasting. 2. When humans are better than AIs at forecasting, AIs are useless. 3. At some point, AIs will be better than humans at forecasting. 4. At that point, when it comes to forecasting, humans will be useless.
This comes from a lot of discussion and some research comparing “humans” to “AIs” in forecasting tournaments.
As you might expect, I think this model is incredibly naive. To me, it’s asking questions like, ”Are AIs better than humans at writing code?” “Are AIs better than humans at trading stocks?” ”Are AIs better than humans at doing operations work?”
I think it should be very clear that there’s a huge period, in each cluster, where it makes sense for humans and AIs to overlap. “Forecasting” is not one homogeneous and singular activity, and neither is programming, stock trading, or doing ops. There’s no clear line for automating “forecasting”—there are instead a very long list of different skills one could automate, with a long tail of tasks that would get increasingly expensive to automate.
Autonomous driving is another similar example. There’s a very long road between “helping drivers with driver-assist features” and “complete level-5 automation, to the extent that almost no human are no longer driving for work purposes.”
A much better model is a more nuanced one. Break things down into smaller chunks, and figure out where and how AIs could best augment or replace humans at each of those. Or just spend a lot of time working with human forecasting teams to augment parts of their workflows.
I am not so aware of the assumption you make up front, and would agree with you that anyone making such an assumption is being naive. Not least because humans on average (and even supers under many conditions) are objectively inaccurate at forecasting—even if relatively good given we don’t have anything better yet.
I think the more interesting and important when it comes to AI forecasting and claiming they are “good”, is to look at the reasoning process that they undertaken to do that. How are they forming reference classes, how are they integrating specific information, how are they updating their posterior to form an accurate inference and likelihood of the event occurring? Right now, they can sort of do (1), but from my experience don’t do well at all at integration, updating, and making a probabilistic judgment. In fairness, humans often don’t either. But we do it more consistently than current AI.
For your post, this suggests to me that AI could be used to help base rate/reference class creation, and maybe loosely support integration.
I’m happy that the phrase “scheming” seems to have become popular recently, that’s an issue that seems fairly specific to me. I have a much easier time imagining preventing an AI from successfully (intentionally) scheming than I do preventing it from being “unaligned.”
Yea, I think I’d classify that as a different thing. I see alignment typically as a “mistake” issue, rather than as a “misuse” issue. I think others here often use the phrase similarly.
When discussing forecasting systems, sometimes I get asked,
“If we were to have much more powerful forecasting systems, what, specifically, would we use them for?”
The obvious answer is,
“We’d first use them to help us figure out what to use them for”
Or,
“Powerful forecasting systems would be used, at first, to figure out what to use powerful forecasting systems on”
For example,
We make a list of 10,000 potential government forecasting projects.
For each, we will have a later evaluation for “how valuable/successful was this project?”.
We then open forecasting questions for each potential project. Like, “If we were to run forecasting project #8374, how successful would it be?”
We take the top results and enact them.
Stated differently,
Forecasting is part of general-purpose collective reasoning.
Prioritization of forecasting requires collective reasoning.
So, forecasting can be used to prioritize forecasting.
I think a lot of people find this meta and counterintuitive at first, but it seems pretty obvious to me.
All that said, I can’t be sure things will play out like this. In practice, the “best thing to use forecasting on” might be obvious enough such that we don’t need to do costly prioritization work first. For example, the community isn’t currently doing much of this meta stuff around Metaculus. I think this is a bit mistaken, but not incredibly so.
A quickly-written potential future, focused on the epistemic considerations:
It’s 2028.
MAGA types typically use DeepReasoning-MAGA. The far left typically uses DeepReasoning-JUSTICE. People in the middle often use DeepReasoning-INTELLECT, which has the biases of a somewhat middle-of-the-road voter.
Some niche technical academics (the same ones who currently favor Bayesian statistics) and hedge funds use DeepReasoning-UNBIASED, or DRU for short. DRU is known to have higher accuracy than the other models, but gets a lot of public hate for having controversial viewpoints. DRU is known to be fairly off-putting to chat with and doesn’t get much promotion.
Bain and McKinsey both have their own offerings, called DR-Bain and DR-McKinsey, respectively. These are a bit like DeepReasoning-INTELLECT, but are munch punchier and confident. They’re highly marketed to managers. These tools produce really fancy graphics, and specialize in things like not leaking information, minimizing corporate decision liability, being easy to use by old people, and being customizable to represent the views of specific companies.
For a while now, some evaluations produced by intellectuals have demonstrated that DeepReasoning-UNBIASED seems to be the most accurate, but few others really care or notice this. DeepReasoning-MAGA has figured out particularly great techniques to get users to distrust DeepReasoning-UNBIASED.
Betting gets kind of weird. Rather than making specific bets on specific things, users started to make meta-bets. “I’ll give money to DeepReasoning-MAGA to bet on my behalf. It will then make bets with DeepReasoning-UNBIASED, which is funded by its believers.”
At first, DeepReasoning-UNBIASED dominates the bets, and its advocates earn a decent amount of money. But as time passes, this discrepancy diminishes. A few things happen:
All DR agents converge on beliefs over particularly near-term and precise facts.
Non-competitive betting agents develop alternative worldviews in which these bets are invalid or unimportant.
Non-competitive betting agents develop alternative worldviews that are exceedingly difficult to empirically test.
In many areas, items 1-3 push people to believe more in the direction of the truth. Because of (1), many short-term decisions get to be highly optimal and predictable.
But because of (2) and (3), epistemic paths diverge, and Non-betting-competitive agents get increasingly sophisticated at achieving epistemic lock-in with their users.
Some DR agents correctly identify the game theory dynamics of epistemic lock-in, and this kickstarts a race to gain converts. It seems like advent users of DeepReasoning-MAGA are very locked-down in these views, and forecasts don’t see them ever changing. But there’s a decent population that isn’t yet highly invested in any cluster. Money spent convincing the not-yet-sure goes a much further way than money spent convincing the highly dedicated, so the cluster of non-deep-believers gets highly targeted for a while. It’s basically a religious race to gain the remaining agnostics.
At some point, most people (especially those with significant resources) are highly locked in to one specific reasoning agent.
After this, the future seems fairly predictable again. TAI comes, and people with resources broadly gain correspondingly more resources. People defer more and more to the AI systems, which are now in highly stable self-reinforcing feedback loops.
Coalitions of people behind each reasoning agent delegate their resources to said agents, then these agents make trade agreements with each other. The broad strokes of what to do with the rest of the lightcone are fairly straightforward. There’s a somewhat simple strategy of resource acquisition and intelligence enhancement, followed by a period of exploiting said resources. The specific exploitation strategy depends heavily on the specific reasoning agent cluster each segment of resources belongs to.
It looks like Concept2, a popular sports equipment company, just put ownership into a Purpose Trust.
As we look toward stepping back from day-to-day operations, we have researched options to preserve the company’s long-held values and mission into the future, including Purpose Trust ownership. With a Purpose Trust as owner, profits are reinvested in the business and also used to fulfill the designated purpose of the Trust. Company profits do not flow to individual shareholders or beneficiaries. And a Purpose Trust can endure in perpetuity.
We are excited to announce we have transferred 100% of Concept2’s ownership to the Concept2 Perpetual Purpose Trust as of January 1, 2025. The Concept2 Perpetual Purpose Trust will direct the management and operations of Concept2 in a manner that maintains continuity. The value we create through our business will be utilized for a greater purpose in serving the Concept2 community. Our vision and mission will carry on in the hands of our talented employee base, and Concept2 will remain the gold standard for providing best in-class products and unmatched customer service. We hope you share in our enthusiasm and will join us on this next phase of our journey as a company.′
I asked Perplexity for other Purchase Trusts, it mentioned that Patagonia is one, plus a few other companies I don’t know of.
My impression is that B-Corps have almost no legal guarantees of public good, and that 501c3s also really have minimal guarantees (if 501c3s fail to live up to their mission, the worst that happens is that they lose their charity and thus tax-deductability status. But this isn’t that bad otherwise).
I imagine that Trusts could be far more restrictive (in a good way). I worked with a company that made Irrevocable Trusts before, I think these might be the structure that would provide the best assurances that we currently have.
I occasionally hear implications that cyber + AI + rogue human hackers will cause mass devastation, in ways that roughly match “lots of cyberattacks happening all over.” I’m skeptical of this causing over $1T/year in damages (for over 5 years, pre-TAI), and definitely of it causing an existential disaster.
There are some much more narrow situations that might be more X-risk-relevant, like [A rogue AI exfiltrates itself] or [China uses cyber weapons to dominate the US and create a singleton], but I think these are so narrow they should really be identified individually and called out. If we’re worried about them, I’d expect we’d want to take very different actions then to broadly reduce cyber risks.
I’m worried that some smart+influential folks are worried about the narrow risks, but then there’s various confusion, and soon we have EAs getting scared and vocal about the broader risks.
Here’s the broader comment against cyber + AI + rogue human hacker risks, or maybe even a lot of cyber + AI + nation state risks.
Note: This was written quickly, and I’m really not a specialist/expert here.
1. There’s easily $10T of market cap of tech companies that would be dramatically reduced if AI systems could invalidate common security measures. This means a lot of incentive to prevent this.
2. AI agents could oversee phone calls and video calls, and monitor other conversations, and raise flags about potential risks. There’s already work here, there could be a lot more.
3. If LLMs could detect security vulnerabilities, this might be a fairly standardized and somewhat repeatable process, and actors with more money could have a big advantage. If person A spend $10M using GPT5 to discover 0-days, they’d generally find a subset compared to person B, who spends $100M. This could mean that governments and corporations would have a large advantage. They could do such investigation during the pre-release of software, and have ongoing security checks as new models are released. Or, companies would find bugs before attackers would. (There is a different question of whether the bug is cost-efficient to fix).
4. The way to do a ton of damage with LLMs and cyber is to develop offensive capabilities in-house, then release a bunch of them at once in a planned massive attack. In comparison, I’d expect that many online attackers using LLMs wouldn’t be very coordinated or patient. I think that attackers are already using LLMs somewhat, and would expect this to scale gradually, providing defenders a lot of time and experience.
5. AI code generation is arguably improving quickly. This could allow us to build much more secure software, and to add security-critical features.
6. If the state of cyber-defense is bad enough, groups like the NSA might use it to identify and stop would-be attackers. It could be tricky to have a world where it’s both difficult to protect key data, but also, it’s easy to remain anonymous when going after other’s data. Similarly, if a lot of the online finance world is hackable, then potential hackers might not have a way to store potential hacking earnings, so could be less motivated. It just seems tough to fully imagine a world where many decentralized actors carry out attacks that completely cripple the economy.
7. Cybersecurity has a lot of very smart people and security companies. Perhaps not enough, but I’d expect these people could see threats coming and respond decently.
8. Very arguably, a lot of our infrastructure is fairly insecure, in large part because it’s just not attacked that much, and when it is, it doesn’t cause all too much damage. Companies historically have skimped on security because the costs weren’t prohibitive. If cyberattacks get much worse, there’s likely a backlog of easy wins, once companies actually get motivated to make fixes.
9. I think around our social circles, those worried about AI and cybersecurity generally talk about it far more than those not worried about it. I think this is one of a few biases that might make things seem scarier than they actually are.
10. Some companies like Apple of gotten good at rolling out security updates fairly quickly. In theory, an important security update to iPhones could reach 50% penetration in a day or so. These systems can improve further.
11. I think we have yet to see the markets show worry about cyber-risk. Valuations of tech companies are very high, cyber-risk doesn’t seem like a major factor when discussing tech valuations. Companies can get cyber-insurance—I think the rates have been going up, but not exponentially.
12. Arguably, there’s many trillions of dollars being held to by billionaires and others that they don’t know what to do with. If something like this actually causes 50%+ global wealth to drop, it would be an enticing avenue for such money to go. Basically, we do have large reserves to spend, if the EV is positive enough, as a planet.
13. In worlds with much better AI, many AI companies (and others) will be a lot richer, and be motivated to keep the game going.
14. Very obviously, if there’s 10T+ at stake, this would be a great opportunity for new security companies and products to enter the market.
15. Again, if there’s 10T+ at stake, I’d assume that people could change practices a lot to use more secure devices. In theory all professionals could change to one of a few locked-down phones and computers.
16. The main scary actors potentially behind AI + Cyber would be nation states and rogue AIs. But nation-states have traditionally been hesitant to make these (meaning $1T+ damage) attacks outside of wartime, for similar reasons that they are hesitant to do military attacks outside wartime.
17. I believe that the US leads on cyber now. The US definitely leads on income. More cyber/hacking abilities would likely be used heavily by the US state. So, if they become much more powerful, the NSA/CIA might become far better at using cyber attacks to go after other potential international attackers. US citizens might have a hard time being private and secure, but so would would-be attackers. Cyber-crime becomes far less profitable if the attackers themselves can preserve their own privacy and security. There are only 8 Billion people in the world, so in theory it might be possible to oversee everyone with a risk of doing damage (maybe 1-10 million people)? Another way of putting this is that better cyber offense could directly lead to more surveillance by the US department. (This obviously has some other downsides, like US totalitarian control, but that is a very different risk)
I wonder if some of the worry on AI + Cyber is akin to the “sleepwalking fallacy”. Basically, if AI + Cyber becomes a massive problem, I think we should expect that there will be correspondingly massive resources spent then trying to fix it. I think that many people (but not all!) worried about this topic aren’t really imagining what $1-10T of decently-effective resources spent on defense would do.
I think that AI + Cyber could be critical threat vector for malicious and powerful AIs in the case of AI takeover. I also could easily see it doing $10-$100B/year of damage in the next few years. But I’m having trouble picturing it doing $10T/year of damage in the next few years, if controlled by humans.
Around prediction infrastructure and information, I find that a lot of smart people make some weird (to me) claims. Like:
If a prediction didn’t clearly change a specific major decision, it was worthless.
Politicians don’t pay attention to prediction applications / related sources, so these sources are useless.
There are definitely ways to steelman these, but I think on the face they represent oversimplified models of how information leads to changes.
I’ll introduce a different model, which I think is much more sensible:
Whenever some party advocates for belief P, they apply some pressure for that belief to those who notice this advocacy.
This pressure trickles down, often into a web of resulting beliefs that are difficult to trace.
People both decide what decisions to consider, and what choices to make, based on their beliefs.
For any agent having an important belief P, this is expected to have been influenced by the beliefs of those that they pay attention to. One can model this with social networks and graphs.
Generally, introducing more correct beliefs, and providing more support to them in directions where important decisions happen, is expected to make those decisions better. This often is not straightforward, but I think we can make decent and simple graphical models of how said beliefs propagate.
Decisions aren’t typically made all-at-once. Often they’re very messy. Beliefs are formed over time, and people randomly decide what questions to pay attention to or what decisions to even consider. Information changes the decisions one chooses to make, not just the outcomes of these decisions.
For example—take accounting. A business leader might look at their monthly figures without any specific decisions in mind. But if they see something that surprises them, they might investigate further, and eventually change something important.
This isn’t at all to say “all information sources are equally useful” or “we can’t say anything about what information is valuable”.
But rather, more like,
“(directionally-correct) Information is useful on a spectrum. The more pressure it can excerpt on decision-relevant beliefs of people with power, the better.”
Occasionally I come across people who assume that “AI + judgemental forecasting” is a well-studies and funded area.
While I think this area is really high-potential (especially if done correctly), I think it’s quite small right now.
I know of two new startups in the space (one is FutureSearch, I heard of another on Twitter). Neither has a very serious public product out yet, both probably have fewer than 10 FTEs. Based on the rates of startup success, I’d guess both will fail or pivot.
Metaculus has done some public challenges for AI forecasting, but these are pretty small. A $~30k prize.
There are occasional research papers that come out in the area. But these typically don’t come with tools that the public can really use.
Overall, while I think this field is really neat, I think there are fairly few people actively working on it now, and I’d expect correspondingly limited results.
Instead of “Goodharting”, I like the potential names “Positive Alignment” and “Negative Alignment.”
″Positive Alignment” means that the motivated party changes their actions in ways the incentive creator likes. “Negative Alignment” means the opposite.
Whenever there are incentives offered to certain people/agents, there are likely to be cases of both Positive Alignment and Negative Alignment. The net effect will likely be either positive or negative.
“Goodharting” is fairly vague and typically just refers to just the “Negative Alignment” portion.
I’d expect this to make some discussion clearer. ”Will this new incentive be goodharted?” → “Will this incentive lead to Net-Negative Alignment?”
I think the term “goodharting” is great. All you have to do is look up goodharts law to understand what is talked about: the AI is optimising for the metric you evaluated it on, rather than the thing you actually want it to do.
Your suggestions would rob this term of the specific technical meaning, which makes thing much vaguer and harder to talk about.
I imagine that scientists will soon have the ability to be unusually transparent and provide incredibly low rates of fraud/bias, using AI. (This assumes strong AI progress in the next 5-20 years)
AI auditors could track everything (starting with some key things) done for an experiment, then flag if there was significant evidence of deception / stats gaming / etc. For example, maybe a scientist has an AI screen-recording their screen whenever it’s on, but able to preserve necessary privacy and throw out the irrelevant data.
AI auditors could review any experimental setups, software, and statistics, and flag if it can detect any errors or not.
Over time, AI systems will be able to do large parts of the scientific work. We can likely make guarantees of AI-done-science that we can’t with humans.
Such systems could hypothetically provide significantly stronger assurances than those argued for by some of the scientific reform communities today (the Open Science movement, for example).
I’ve been interested in this for some of QURI’s work, and would love to see AI-overseen experimentation be done in the AI safety world.
Perhaps most important, this could all be good experimentation on our way to systems that will oversee key AI progress. I ultimately want AI auditors for all risky AI development, but some of that will be a harder sell.
Agreed with this. I’m very optimistic about AI solving a lot of incentive problems in science. I don’t know if the end case (full audits) as you mention will happen, but I am very confident we will move in a better direction than where we are now.
I’m working on some software now that will help a bit in this direction!
As a working scientist, I strongly doubt that any of this will happen.
First, existing AI’s are nowhere near being able to do any of the things with an accuracy that makes them particularly useful. AI’s are equipped to do things similar to their training set, but all science is on the frontier: it is a much harder task to figure out the correct experimental setup for something that has never been done before in the history of humanity.
Right now I’m finishing up an article about how my field acually uses AI, and it’s nothing like anything you proposed here: LLMs are used for BS grant applications and low-level coding, almost exclusively. I don’t find it very useful for anything else.
The bigger issue here is with the “auditors” themselves: who’s in charge of them? If a working scientist disagrees with what the “auditor” says, what happens? What happens if someone like Elon is in charge, and decides to use the auditors for a political crusade against “woke science”, as is currently literally happening right now?
Catching errors in science is not something that can be boiled down to a formula: a massive part of the process is socio-cultural. You push out AI auditors, people are just going to game them, like they have with p-values, etc. This is not a problem with a technological solution.
The bigger issue here is with the “auditors” themselves: who’s in charge of them? If a working scientist disagrees with what the “auditor” says, what happens? What happens if someone like Elon is in charge, and decides to use the auditors for a political crusade against “woke science”, as is currently literally happening right now?
I think this is a very sensible question.
My obvious answer is that the auditors should be held up to higher standards than the things they are auditing. This means that these should be particularly open, and should be open to other auditing. For example, the auditing code could be open-source, highly tested, and evaluated by both humans and AI systems.
I agree that there are ways one could do a poor job with auditing. I think this is generally true for most powerful tools we can bring in—we need to be sure to use it well, else it could do harm.
On your other points—it sounds like you have dramatically lower expectations for AI than I do or much of the AI safety community does. I agree that if you don’t think AI is very exciting, then AI-assisted auditing probably won’t go that far.
From my post: > this could all be good experimentation on our way to systems that will oversee key AI progress. I ultimately want AI auditors for all risky AI development, but some of that will be a harder sell.
If it’s the case that AI-auditors won’t work, then I assume we wouldn’t particularly need to oversee key AI progress anyway, as there’s not much to oversee.
My obvious answer is that the auditors should be held up to higher standards than the things they are auditing. This means that these should be particularly open, and should be open to other auditing. For example, the auditing code could be open-source, highly tested, and evaluated by both humans and AI systems.
Yeah, I just don’t buy that we could ever establish such a code in a way that would make it viable. Science chases novel projects and experiments, what is “meant” to happen will be different for each miniscule subfield of each field. If you release an open source code that has been proven to work for subfields A,B,C,D,E,F, someone in subfield G will immediately object that it’s not transferable, and they may very well be right. And the only people who can tell if it works on subfield G is people who are in subfield G.
You cannot avoid social and political aspects to this: Imagine if the AI-auditor code starts declaring that a controversial and widely used technique in, say, evolutionary psychology, is bad science. Does the evo-psych community accept this and abandon the technique, or do they say that the auditor code is flawed due to the biases of the code creators, and fork/reject the code? Essentially you are allowing whoever is controlling the auditor code to suppress fields they don’t agree with. It’s a centralization of science that is at odds with what allows science to actually work.
This strikes comment strikes me as so different to my view that I imagine you might be envisioning a very specific implementation of AI auditors that I’m not advocating for.
I tried having a discussion with an LLM about this to get some more insight, you can see this here if you like (though I suspect that you won’t wind this useful, as you seem to not trust LLMs much at all.) It wound up suggesting implementations that could still provide benefits while minimizing potential costs.
I don’t mind you using LLMs for elucidating discussion, although I don’t think asking it to rate arguments is very valuable.
The additional details of having subfield specific auditors that are opt-in does lessen my objections significantly. Of course, the issue of what counts as a subfield is kinda thorny. It would make most sense for, as claude suggests, journals to have an “auditor verified” badge, but then maybe you’re giving too much power over content to the journals, which usually stick to accept/reject decisions (and even that can get quite political).
Coming back to your original statement, ultimately I just don’t buy that any of this can lead to “incredibly low rates of fraud/bias”. If someone wants to do fraud or bias, they will just game the tools, or submit to journals with weak/nonexistent auditors. Perhaps the black box nature of AI might even make it easier to hide this kind of thing.
Next: there are large areas of science where a tool telling you the best techniques to use will never be particularly useful. On the one hand there is research like mine, where it’s so frontier that the “best practices” to put into such an auditor don’t exist yet. On the other, you have statistics stuff that is so well known that there already exist software tools that implement the best practices: you just have to load up a well documented R package. What does an AI auditor add to this?
If I was tasked with reducing bias and fraud, I would mainly push for data transparency requirements in journal publications, and in beefing up the incentives for careful peer review, which is currently unpaid and unrewarding labour. Perhaps AI tools could be useful in parts of that process, but I don’t see it as anywhere near as important than those other two things.
Looking back, I think this part of my first comment was poorly worded: > I imagine that scientists will soon have the ability to be unusually transparent and provide incredibly low rates of fraud/bias, using AI.
I meant > I imagine that scientists will [soon have the ability to] be unusually transparent and provide incredibly low rates of fraud/bias], using AI.
So it’s not that this will lead to low rates of fraud/bias, but that AI will help enable that for scientists willing to go along with it—but at the same time, there’s a separate question of if scientists are willing to go along with it.
But I think even that probably is not fair. A a better description of my beliefs is something like,
I think that LLM auditing tools could be useful for some kinds of scientific research for communities open to them.
I think in the short-term, sufficiently-motivated groups could develop these tools and use them to help decrease the levels of statistical and algorithmic accidents that happen. Correspondingly, I’d expect this to help with fraud.
In the long-run, whenever AI approaches human-level intelligence (which I think will likely happen in the next 20 years, but I realize others disagree), I expect that more and more of the scientific process will be automated. I think there are ways this could go very well using things like AI auditing, whereby the results will be much more reliable than those currently made by humans. There are of course also worlds in which humans do dumb things with the AIs and the opposite happens.
I think that at least, AI safety researchers should consider using these kinds of methods, and that the AI safety landscape should investigate efforts to make decent auditing tools.”
My core hope with the original message is to draw attention to AI science auditing tools as things that might be interesting/useful, not to claim that they’re definitely a major game changer.
If we could have LLM agents that could inspect other software applications (including LLM agents) and make strong claims about them, that could open up a bunch of neat possibilities.
There could be assurances that apps won’t share/store information.
There could be assurances that apps won’t be controlled by any actor.
There could be assurances that apps can’t be changed in certain ways (eventually).
I assume that all of this should provide most of the benefits people ascribe to blockchain benefits, but without the costs of being on the blockchain.
Some neat options from this:
Companies could request that LLM agents they trust inspect the code of SaaS providers, before doing business with them. This would be ongoing.
These SaaS providers could in turn have their own LLM agents that verify that these investigator LLM agents are trustworthy (i.e. won’t steal anything).
Any bot on social media should be able to provide assurances of how they generate content. I.E. they should be able to demonstrate that they aren’t secretly trying to promote any certain agenda or anything.
Statistical analysis could come with certain assurances. Like, “this analysis was generated with process X, which is understood to have minimal bias.”
It’s often thought that LLMs make web information more opaque and less trustworthy. But with some cleverness, perhaps it could do just the opposite. LLMs could enable information that’s incredibly transparent and trustworthy (to the degrees that matter.)
Criticisms:
“But as LLMs get more capable, they will also be able to make software systems that hide subtler biases/vulnerabilities”
-> This is partially true, but only goes so far. A whole lot of code can be written simply, if desired. We should be able to have conversations like, “This codebase seems needlessly complex, which is a good indication that it can’t be properly trusted. Therefore, we suggest trust other agents more.”
“But the LLM itself is a major black box”
-> True, but it might be difficult to intentionally bias if an observer has access to the training process. Also, it should be understood that off-the-shelf LLMs are more trustworthy than proprietary ones / ones developed for certain applications.
Quick list of some ideas I’m excited about, broadly around epistemics/strategy/AI.
1. I think AI auditors / overseers of critical organizations (AI efforts, policy groups, company management) are really great and perhaps crucial to get right, but would be difficult to do well.
2. AI strategists/tools telling/helping us broadly what to do about AI safety seems pretty safe.
3. In terms of commercial products, there’s been some neat/scary military companies in the last few years (Palantir, Anduril). I’d be really interested if there could be some companies to automate core parts of the non-military government. I imagine there are some parts of the government that are particularly tractable/influenceable/tractable. For example, just making great decisions on which contractors the government should work with. There’s a ton of work to do here, between the federal government / state government / local government.
6. I think there are a lot of interesting ways for us to experiment with [AI tools to help our research/epistemics]. I want to see a wide variety of highly creative experimentation here. I think people are really limiting themselves in this area to a few narrow conceptions of how AI can be used in very specific ways that humans are very comfortable with. For example, I’d like to see AI dashboards of “How valuable is everything in this space” or even experiments where AIs negotiate on behalf of people and they use the result of that. A lot of this will get criticized for being too weird/disruptive/speculative, but I think that’s where good creative works should begin.
7. Right now, I think the field of “AI forecasting” is actually quite small and constrained. There’s not much money here, and there aren’t many people with bold plans or research agendas. I suspect that some successes / strong advocates could change this.
8. I think that it’s likely that Anthropic (and perhaps Deepmind) would respond well to good AI+epistemics work. “Control” was quickly accepted at Anthropic, for example. I suspect that it’s possible that things like the idea of an “Internal AI+human auditor” or an internal “AI safety strategist” could be adopted if done well.
Some AI questions/takes I’ve been thinking about: 1. I hear people confidently predicting that we’re likely to get catastrophic alignment failures, even if things go well up to ~GPT7 or so. But if we get to GPT-7, I assume we could sort of ask it, “Would taking this next step, have a large chance of failing?“. Basically, I’m not sure if it’s possible for an incredibly smart organization to “sleepwalk into oblivion”. Likewise, I’d expect trade and arms races to get a lot nicer/safer, if we could make it a few levels deeper without catastrophe. (Note: This is one reason I like advanced forecasting tech)
2. I get the impression that lots of EAs are kind of assuming that, if alignment issues don’t kill us quickly, 1-2 AI companies/orgs will create decisive strategic advantages, in predictable ways, and basically control the world shortly afterwards. I think this is a possibility, but would flag that right now, probably 99.9% of the world’s power doesn’t want this to happen (basically, anyone who’s not at the top of OpenAI/Anthropic/the next main lab). It seems to me like these groups would have to be incredibly incompetent to just let one org predictably control the world, within 2-20 years. This both means that I find this scenario unlikely, but also, almost every single person in the world should be an ally in helping EAs make sure these scenarios don’t happen.
3. Related to #2, I still get the impression that it’s far easier to make a case of, “Let’s not let one organization, commercial or government, get a complete monopoly on global power, using AI”, then, “AI alignment issues are likely to kill us all.” And a lot of the solutions to the former also seem like they should help the latter.
Depends on what assurance you need. If GPT-7 reliably provides true results in most/all settings you can find, that’s good evidence.
If GPT-7 is really Machiavellian, and is conspiring against you to make GPT-8, then it’s already too late for you, but it’s also a weird situation. If GPT-7 were seriously conspiring against you, I assume it wouldn’t need to wait until GPT-8 to take action.
In comparison to the LTFF, I think the average grant is more generically exciting, but less effective altruist focused. (As expected)
Lots of tiny grants (<$10k), $150k is the largest one.
These rapid grant programs really seem great and I look forward to them being scaled up.
That said, the next big bottleneck (which is already a bottleneck) is funding for established groups. These rapid grants get things off the ground, but many will need long-standing support and scale.
Scott seems to have done a pretty strong job researching these groups, and also has had access to a good network of advisors. I guess it’s no surprise; he seems really good at “doing a lot of reading and writing”, and he has an established peer group now.
I’m really curious how/if these projects will be monitored. At some point, I think more personnel would be valuable.
This grant program is kind of a way to “scale up” Astral Codex Ten. Like, instead of hiring people directly, he can fund them this way.
I’m curious if he can scale up 10x or 1000x, we could really use more strong/trusted grantmakers. It’s especially promising if he gets non-EA money. :)
On specific grants:
A few forecasters got grants, including $10k for Nuño Sempere Lopez Hidalgo for work on Metaforecast. $5k for Nathan Young to write forecasting questions.
$17.5k for 1DaySooner/Rethink Priorities to do surveys to advance human challenge trials.
$40k seed money to Spencer Greenberg to “produce rapid replications of high-impact social science papers”. Seems neat, I’m curious how far $40k alone could go though.
A bunch of biosafety grants. I like this topic, seems tractable.
$40k for land value tax work.
$20k for a “Chaotic Evil” prediction market. This will be interesting to watch, hopefully won’t cause net harm.
$50k for the Good Science Project, to “improve science funding in the US”. I think science funding globally is really broken, so this warms my heart.
Lots of other neat things, I suggest just reading directly.
I’ve heard from friends outside the EA scene that they think most AI-risk workers have severe mental issues like depression and burnout.
I don’t mean to downplay this issue, but I think a lot of people get the wrong idea.
My hunch is that many of the people actively employed and working on AI safety are fairly healthy and stable. Many are very well paid and have surprisingly nice jobs/offices.
I think there is this surrounding ring of people who try to enter this field, who have a lot of problems. It can be very difficult to get the better positions, and if you’re stubborn enough, this could lead to long periods of mediocre management and poor earnings.
I think most of the fully employed people typically are either very busy, or keep to a limited social circle, so few outsiders will meet them. Instead, outsiders will meet people who are AI safety adjacent or trying to enter the field, and those people can have a much tougher time.
So to me, most people who are succeeding in the field come across a lot like typical high-achievers, with the profiles of typical high-achievers. And people not succeeding in the field come across as people trying and not succeeding in other competitive fields. I’d expect that statistics / polls would broadly reflect this.
All that to say, if you think that people shouldn’t care about AI safety / x-risks because then they’ll go through intense depression and anxiety, I think you might be missing some of the important demographic details.
Some recent tech unions, like the one in Google, have been pushing more for moral reforms than for payment changes.
Likewise, a bunch of AI engineers could use collective bargaining to help ensure that safety measures get more attention, in AI labs.
There are definitely net-negative unions out there too, so it would need to be done delicately.
In theory there could be some unions that span multiple organizations. That way one org couldn’t easily “fire all of their union staff” and hope that recruiting others would be trivial.
Really, there aren’t too many AI engineers, and these people have a ton of power, so they could be a highly advantaged place to make a union.
I made a quick Manifold Market for estimating my counterfactual impact from 2023-2030.
One one hand, this seems kind of uncomfortable—on the other, I’d really like to feel more comfortable with precise and public estimates of this sort of thing.
Feel free to bet!
Still need to make progress on the best resolution criteria.
Note that while we’ll have some clarity in 2030, we’d presumably have less clarity than at the end of history (and even then things could be murky, I dunno)
You could use prediction setups to resolve specific cruxes on why prediction setups outputted certain values.
My guess is that this could be neat, but also pretty tricky. There are lots of “debate/argument” platforms out there, it’s seemed to have worked out a lot worse than people were hoping. But I’d love to be proven wrong.
P.S. I’d be keen on working on this, how do I get involved?
If “this” means the specific thing you’re referring to, I don’t think there’s really a project for that yet, you’d have to do it yourself. If you’re referring more to forecasting projects more generally, there are different forecasting jobs and stuff popping up. Metaculus has been doing some hiring. You could also do academic research in the space. Another option is getting an EA Funds grant and pursuing a specific project (though I realize this is tricky!)
1) Humanity has a >80% chance of completely perishing in the next ~300 years.
2) The expected value of the future is incredibly, ridiculously, high!
The trick is that the expected value of a positive outcome could be just insanely great. Like, dramatically, incredibly, totally, better than basically anyone discusses or talks about.
Expanding to a great deal of the universe, dramatically improving our abilities to convert matter+energy to net well-being, researching strategies to expand out of the universe.
A 20%, or even a 0.002%, chance at a 10^20 outcome, is still really good.
One key question is the expectation of long-term negative[1] vs. long-term positive outcomes. I think most people are pretty sure that in expectation things are positive, but this is less clear.
So, remember:
Just because the picture of X-risks might look grim in terms of percentages, you can still be really optimistic about the future. In fact, many of the people most concerned with X-risks are those *most* optimistic about the future.
I’ve been getting more requests recently to have calls/conversations to give advice, review documents, or be part of extended sessions on things. Most of these have been from EAs.
I find a lot of this work fairly draining. There can be surprisingly high fixed costs to having a meeting. It often takes some preparation, some arrangement (and occasional re-arrangement), and a fair bit of mix-up and change throughout the day.
My main work requires a lot of focus, so the context shifts make other tasks particularly costly.
Most professional coaches and similar charge at least $100-200 per hour for meetings. I used to find this high, but I think I’m understanding the cost more now. A 1-hour meeting at a planned time costs probably 2-3x as much time as a 1-hour task that can be done “whenever”, for example, and even this latter work is significant.
Another big challenge is that I have no idea how to prioritize some of these requests. I’m sure I’m providing vastly different amounts of value in different cases, and I often can’t tell.
The regular market solution is to charge for time. But in EA/nonprofits, it’s often expected that a lot of this is done for free. My guess is that this is a big mistake. One issue is that people are “friends”, but they are also exactly professional colleagues. It’s a tricky line.
One minor downside of charging is that it can be annoying administratively. Sometimes it’s tricky to get permission to make payments, so a $100 expense takes $400 of effort.
Note that I do expect that me helping the right people, in the right situations, can be very valuable and definitely worth my time. But I think on the margin, I really should scale back my work here, and I’m not sure exactly how to draw the line.
[All this isn’t to say that you shouldn’t still reach out! I think that often, the ones who are the most reluctant to ask for help/advice, represent the cases of the highest potential value. (The people who quickly/boldly ask for help are often overconfident). Please do feel free to ask, though it’s appreciated if you give me an easy way out, and it’s especially appreciated if you offer a donation in exchange, especially if you’re working in an organization that can afford it.]
I think it would be really useful for there to be more public clarification on the relationship between effective altruism and Open Philanthropy.
My impression is that:
1. OP is the large majority funder of most EA activity.
2. Many EAs assume that OP is a highly EA organization, including the top.
3. OP really tries to explicitly not take responsibility for EA and does not claim to themselves be highly EA.
4. EAs somewhat assume that OP leaders are partially accountable to the EA community, but OP leaders would mostly disagree.
5. From the point of view of many EAs, EA represents something like a community of people with similar goals and motivations. There’s some expectations that people will look out for each other.
6. From the point of view of OP, EA is useful insofar as it provides valuable resources (talent, sometimes ideas and money).
My impression is that OP basically treats the OP-EA relationship as a set of transactions, each with positive expected value. Like, they would provide a $20k grant to a certain community, if they expect said community to translate into over $20k of value via certain members who would soon take on jobs at certain companies. Perhaps in-part because there are some overlapping friendships, I think that OP staff often explicitly try to only fund EAs in ways that make the clearest practical sense for specific OP goals, like hiring AI safety researchers.
In comparison, I think a lot of EAs think of EA as some kind of holy-ish venture tied to an extended community of people who will care about each other. To them, EA itself is an incredibly valuable idea and community that itself has the potential to greatly change the world. (I myself is more in this latter camp)
So on one side, we have a group that often views EA through reductive lenses, like as a specific recruiting arm. And on the other side, it’s more of a critical cultural movement.
I think it’s very possible for both sides to live in unison, but I think at the moment there’s a lot of confusion about this. I think a lot of EAs assume that OP shares a lot of the same beliefs they do.
If it is the case that OP is fairly narrow in its views and goals with EA, I would hope that other EAs realize that there might be a gap of [leaders+funders who care about EA for the reasons that EAs care about EA]. It’s just weird and awkward to have your large-majority funder be a group that just treats you very reductively.
As a simple example, if one thinks of EA as something akin to a key social movement/community, one might care a lot about:
- The long-term health and growth of EA
- The personal health of specific EAs, not just the very most productive ones
- EA being an institution known for being highly honest and trustworthy
But if one thinks of them through a reductive lens, I’d expect them to care more about:
- Potential short-term hires or funding
- Potential liabilities
- Community members not being too annoying with criticisms and stuff
I’ve met a few people who felt very betrayed by EA—I suspect that the above confusion is one reason why. I think that a lot of EA recruiters argue that EA represents a healthy community/movement. This seems like the most viral message, it’s not a surprise that people doing recruiting and promotion would lean on this idea. But if much of EA funding is really functionally treating it as a recruiting network, then that would be a disconnect.
Related, I’ve though that “EA Global has major community and social movement vibes, but has the financial incentives in line with a recruiting fair.”
Both perspectives can coexist, but greater clarity seems very useful, and might expose some important gaps.
Ozzie my apologies for not addressing all of your many points here, but I do want to link you to two places where I’ve expressed a lot of my takes on the broad topic:
On medium, I talk about how I see the community at some length. tldr: aligned around the question of how to do the most good vs the answer, heterogenous, and often specifically alien to me and my values.
On the forum, I talk about the theoretical and practical difficulties of OP being “accountable to the community” (and here I also address an endowment idea specifically in a way that people found compelling). Similarly from my POV it’s pretty dang hard to have the community be accountable to OP, in spite of everything people say they believe about that. Yes we can withold funding, after the fact, and at great community reputational cost. But I can’t e.g. get SBF to not do podcasts nor stop the EA (or two?) that seem to have joined DOGE and started laying waste to USAID. (On Bsky, they blame EAs for the whole endeavor)
Minor points, from your comment:
I believe most EAs would agree these examples should never have been in OP’s proverbial sphere of responsibility.
There are other examples we could discuss regarding OP’s role (as makes sense, no organization is perfect), but that might distract from the main topic: clarity on the OP-EA relationship and the mutual expectations between parties.
It seems obvious that such Bsky threads contain significant inaccuracies. The question is how much weight to give such criticisms.
My impression is that many EAs wouldn’t consider these threads important enough to drive major decisions like funding allocations. However, the fact you mention it suggests it’s significant to you, which I respect.
About the OP-EA relationship—if factors like “avoiding criticism from certain groups” are important for OP’s decisions, saying so clearly is the kind of thing that seems useful. I don’t want to get into arguments about if it should[1], the first thing is to just understand that that’s where a line is.
More specifically, I think these discussions could be useful—but I’m afraid they will get in the way of the discussions of how OP will act, which I think is more important.
Ozzie I’m not planning to discuss it any further and don’t plan to participate on the forum anymore.
Please come back. I can’t say that we agree on very much, but you are often a voice of reason and your voice will be missed.
This is probably off-topic, but I was very surprised to read this, given how much he supported the Harris campaign, how much he gives to reduce global poverty, and how similar your views are on e.g. platforming controversial people.
Presumably https://reflectivealtruism.com/category/billionaire-philanthropy/?
Just flagging that the EA Forum upvoting system is awkward here. This comment says:
1. “I can’t say that we agree on very much”
2. “you are often a voice of reason”
3. “your voice will be missed”
As such, I’m not sure what the Agree / Disagree reacts are referring to, and I imagine similar for others reading this.
This isn’t a point against David, just a challenge with us trying to use this specific system.
This seems like quite a stretch.
Thanks for the response here! I was not expecting that.
This is a topic that can become frustratingly combative if not handled gracefully, especially in public forums. To clarify, my main point isn’t disagreement with OP’s position, but rather I was trying to help build clarity on the OP-EA relationship.
Some points:
1. The relationship between the “EA Community” and OP is both important (given the resources involved) and complex[1] .
2. In such relationships, there are often unspoken expectations between parties. Clarity might be awkward initially but leads to better understanding and coordination long-term.
3. I understand you’re uncomfortable with OP being considered responsible for much of EA or accountable to EA. This aligns with the hypotheses in my original comment. I’m not sure we’re disagreeing on anything here.
4. I appreciate your comments, though I think many people might reasonably still find the situation confusing. This issue is critical to many people’s long-term plans. The links you shared are helpful but leave some uncertainty—I’ll review them more carefully.
5. At this point, we might be more bottlenecked by EAs analyzing the situation than by additional writing from OP (though both are useful). EAs likely need to better recognize the limitations of the OP-EA relationship and consider what that means for the community.
6. When I asked for clarification, I imagined that EA community members working at the OP-EA intersection would be well positioned to provide insight. One challenge is that many people feel uncomfortable discussing this relationship openly due to the power imbalance.[2]. As well as the funding issue (OP funds EA), there’s also the fact that OP has better ways of privately communicating[3]. (This is also one issue why I’m unusually careful and long with my words with these discussions, sorry if it comes across as harder to read.) That said, comment interactions and assurances from the OP do help build trust.
there’s a fair bit of nuance involved—I’m sure that you have noticed confusion on the side of EAs at least
i.e. For example, say an EA community member writes something that upsets someone at OP. Then that person holds a silent grudge, decides they don’t like that person, then doesn’t fund them later. This is very human, and there’s a clear information asymmetry. The EA community member would never know if this happens, so it would make sense for them to be extra cautious.
People at OP can confidentially discuss with each other how to best handle their side of the OP-EA relationship. But in comparison, EA community members mainly have the public EA Forum, so there’s an inherent disadvantage.
I’m interested in hearing from those who provided downvotes. I could imagine a bunch of reasons why one might have done so (there were a lot of points included here).
(Upon reflection, I don’t think my previous comment was very good. I tried to balance being concise, defensive, and comprehensive, but ended up with something confusing. I’d be happy to clarify my stance on this more at any time if asked, though it might well be too late now for that to be useful. Apologies!)
I’m out of the loop, who’s this allegedly EA person who works at DOGE?
Many people claim that Elon Musk is an EA person, @Cole Killian has an EA Forum account and mentioned effective altruism on his (now deleted) website, Luke Farritor won the Vesuvius Challenge mentioned in this post (he also allegedly wrote or reposted a tweet mentioning effective altruism, but I can’t find any proof and people are skeptical)
This reminds me of another related tension I’ve noticed. I think that OP really tries to not take much responsibility for EA organizations, and I believe that this has led to something of a vacuum of leadership.
I think that OP has functionally has great power over EA.
In many professional situations, power comes with corresponding duties and responsibilities.
CEOs have a lot of authority, but they are also expected to be agentic, to keep on the lookout for threats, to be in charge of strategy, to provide guidance, and to make sure many other things are carried out.
The President clearly has a lot of powers, and that goes hand-in-hand with great expectations and duties.
There’s a version of EA funding where the top funders take on both leadership and corresponding responsibilities. These people ultimately have the most power, so arguably they’re best positioned to take on leadership duties and responsibilities.
But I think nonprofit funders often try not to take much in terms of responsibilities, and I don’t think OP is an exception. I’d also flag that I think EA Funds and SFF are in a similar boat, though these are smaller.
My impression is that OP explicitly tries not to claim any responsibility for the EA ecosystem / environment, and correspondingly argues it’s not particularly accountable to EA community members. Their role as I understand it is often meant to be narrow. This varies by OP team, but I think is true for the “GCR Capacity Building” team, which is closest to many “EA” orgs. I think this team mainly thinks of itself as a group responsible for making good decisions on a bunch of specific applications that hits their desk.
Again, this is a far narrower mandate than any conventional CEO would have.
If we had a “CEO or President” that were both responsible for and accountable to these communities, I’d expect things like:
1. A great deal of communication with these communities.
2. Clear and open leadership structures and roles.
3. A good deal of high-level strategizing.
4. Agentic behavior, like taking significant action to “make sure specific key projects happen.”
5. When there are failures, acknowledgement of said failures, as well as plans to fix or change.
I think we basically don’t have this, and none of the funders would claim to be this.
So here’s a question: “Is there anyone in the EA community who’s responsible for these sorts of things?”
I think the first answer I’d give is “no.” The second answer is something like, “Well, CEA is sort of responsible for some parts of this. But CEA really reports to OP given their funding. CEA has very limited power of its own. And CEA has repeatedly try to express limits in its power, plus its gone through lots of management transitions.”
In a well-run bureaucracy, I imagine that key duties would be clearly delegated to specific people or groups, and that groups would have the corresponding powers necessary to actually do a good job at them. You want key duties to be delegated to agents with the power to carry them out.
The ecosystem of EA organizations is not a well-organized bureaucracy. But that doesn’t mean there aren’t a lot of important duties to be performed. In my opinion, the fact that EA represents a highly-fragmented set of small organizations was functionally a decision by the funders (at least, they had a great deal of influence on this), so I’d hope that they would have thoughts on how to make sure the key duties get done somehow.
This might seem pretty abstract, so I’ll try coming up with some more specific examples:
1. Say a tiny and poorly-resourced org gets funded. They put together a board of their friends (the only people available), then proceed to significantly emotionally abuse their staff. Who is ultimately responsible here? I’d expect the founders would not at all want to take responsibility for this.
2. Before the FTX Future Fund blew up, I assumed that EA leaders had vetted it. Later I find out that OP purposefully tried to keep its distance and not get involved (in this case meaning that they didn’t investigate or warn anyone), in part because they didn’t see it as their responsibility, and claimed that because FTX Future Fund was a “competitor”, it wasn’t right for them to get involved. From what I can tell now, it was no one’s responsibility to vet the FTX Future Fund team or FTX organization. You might have assumed CEA, but CEA was funded by FTX and previously even had SBF as a board member—they were clearly not powerful and independent enough for this.
3. There are many people in the EA scene who invest large amounts of time and resources preparing for careers that only exist under the OP umbrella. Many or all of their future jobs will be under this umbrella. At the same time, it’s easy to imagine that they have almost no idea what the power structures at the top of this umbrella are like. This umbrella could change leadership or direction at any time, with very little warning.
4. There were multiple “EAs” on the board of OpenAI during that board member spat. That event seemed like a mess, and it negatively influenced a bunch of other EA organizations. Was that anyone’s responsibility? Can we have any assurances that community members will do a better job next time? (if there is a next time)
5. I’m not sure if many people at all, in positions of power, are spending much time thinking about long-term strategic issues for EA. It seems very easy for me to imagine large failures and opportunities we’re missing out on. This also is true for the nonprofit EA AI Safety Landscape—many of the specific organizations are too small and spread out to be very agentic, especially in cases of dealing with diverse and private information. I’ve heard good things recently about Zach Robinson at CEA, but also would note that CEA has historically been highly focused on some long-running projects (EAG, the EA Forum, Community Health), with fairly limited strategic or agentic capacity, plus being heavily reliant on OP.
6. Say OP decides to shut down the GCR Capacity Building team one day, and gives a 2-years notice. I’d expect this to be a major mess. Few people outside OP understand how the internals of OP decisions get made, so it’s hard for other EA members to see this coming or gauge how likely it is. My guess is that they don’t seem like they’d do this, but I have limited confidence. As such, it’s hard for me to suggest that people make long-term plans (3+ years) in this area.
7. We know that OP generally maximizes expected value. What happens when narrow EV optimization conflicts with honesty and other cooperative values? Would they represent the same choices that other EAs might want? I believe that FTX justified their bad actions using utilitarianism, for instance, and lots of businesses and nonprofits carry out highly Machiavellian and dishonest actions to advance their interests. Is it possible that EAs working under the OP umbrella are unknowingly supporting actions they might not condone? It’s hard to know without much transparency and evaluation.
On the plus side, I think OP and CEA have improved a fair bit on this sort of thing in the last few years. OP seems to be working to assure that grantees follow certain basic managerial criteria. New hires and operations have come in, which has seemed to have helped.
I’ve previously discussed my thinking on the potential limitations we’re getting from having small orgs here. Also, I remember that Oliver Habryka has repeatedly mentioned the lack of leadership around this scene—I think that this topic is one thing he was sort-of referring to.
Ultimately, my guess is that OP has certain goals they want to achieve, and it’s unlikely they or the other funders will want to take many of the responsibilities that I suggest here.
Given that, I think it would be useful for people in the EA ecosystem to understand this and respond accordingly. I think that our funding situation really needs diversification, and I think that funders willing to be more agentic in crucial areas that are currently lacking could do a lot of good. I expect that when it comes to “senior leadership”, there are some significant gains to be made, if the right people and resources can come together.
This seems directionally correct, but I would add more nuance.
While OP, as a grantmaker, has a goal it wants to achieve with its grants (and they wouldn’t be EA aligned if they didn’t), this doesn’t necessarily mean they are very short term. The Open Phil EA/LT Survey seems to me to show best what they care about in outcomes (talent working in impactful areas) but also how hard it is to pinpoint the actions and inputs needed. This leads me to believe that OP instrumentally cares about the community/ecosystem/network as it needs multiple touchpoints and interactions to get most people from being interested in EA ideas to working on impactful things.
On the other side, we use the term community in confusing ways. I was on a Community Builder Grant by CEA for two years when working at EA Germany, which many call national community building. What we were actually doing was working on the talent development pipeline, trying to find promising target groups, developing them and trying to estimate the talent outcomes.
Working on EA as a social movement/community while being paid is challenging. On one hand, I assume OP would find it instrumentally useful (see above) but still desire to track short-term outcomes as a grantmaker. As a grant recipient, I felt I couldn’t justify any actions that lacked a clear connection between outcomes and impact. Hosting closed events for engaged individuals in my local community, mentoring, having one-on-ones with less experienced people, or renting a local space for coworking and group events appeared harder to measure. I also believe in the norm of doing this out of care, wanting to give back to the community, and ensuring the community is a place where people don’t need to be compensated to participate.
Thanks for the details here!
> would add more nuance
I think this is a complex issue. I imagine it would be incredibly hard to give it a really robust write-up, and definitely don’t mean for my post to be definitive.
I think this is downstream of a lot of confusion about what ‘Effective Altruism’ really means, and I realise I don’t have a good definition any more. In fact, because all of the below can be criticised, it sort of explains why EA gets seemingly infinite criticism from all directions.
Is it explicit self-identification?
Is it explicit membership in a community?
Is it implicit membership in a community?
Is it if you get funded by OpenPhilanthropy?
Is it if you are interested or working in some particular field that is deemed “effective”?
Is it if you believe in totalising utilitarianism with no limits?
To always justify your actions with quantitative cost-effectiveness analyses where you’re chosen course of actions is the top ranked one?
Is it if you behave a certain way?
Because in many ways I don’t count as EA based off the above. I certainly feel less like one than I have in a long time.
For example:
I don’t know if this refers to some gestalt ‘belief’ than OP might have, or Dustin’s beliefs, or some kind of ‘intentional stance’ regarding OP’s actions. While many EAs shared some beliefs (I guess) there’s also a whole range of variance within EA itself, and the fundamental issue is that I don’t know if there’s something which can bind it all together.
I guess I think the question should be less “public clarification on the relationship between effective altruism and Open Philanthropy” and more “what does ‘Effective Altruism’ mean in 2025?”
I had Claude rewrite this, if the terminology is confusing. I think it’s edit is decent.
---
The EA-Open Philanthropy Relationship: Clarifying Expectations
The relationship between Effective Altruism (EA) and Open Philanthropy (OP) might suffer from misaligned expectations. My observations:
OP funds the majority of EA activity
Many EAs view OP as fundamentally aligned with EA principles
OP deliberately maintains distance from EA and doesn’t claim to be an “EA organization”
EAs often assume OP leadership is somewhat accountable to the EA community, while OP leadership likely disagrees
Many EAs see their community as a unified movement with shared goals and mutual support
OP appears to view EA more transactionally—as a valuable resource pool for talent, ideas, and occasionally money
This creates a fundamental tension. OP approaches the relationship through a cost-benefit lens, funding EA initiatives when they directly advance specific OP goals (like AI safety research). Meanwhile, many EAs view EA as a transformative cultural movement with intrinsic value beyond any specific cause area.
These different perspectives manifest in competing priorities:
EA community-oriented view prioritizes:
Long-term community health and growth
Individual wellbeing of community members
Building EA’s reputation for honesty and trustworthiness
Transactional view prioritizes:
Short-term talent pipeline and funding opportunities
Risk management (not wanting EA activities to wind up reflecting poorly on OP)
Minimizing EA criticism of OP and OP activities. (This is both annoying to deal with, and could hurt their specific activities)
This disconnect explains why some people might feel betrayed by EA. Recruiters often promote EA as a supportive community/movement (which resonates better), but if the funding reality treats EA more as a talent network, there’s a fundamental misalignment.
Another thought I’ve had: “EA Global has major community and social movement vibes, but has the financial incentives in line with a recruiting fair.”
Both perspectives can coexist, but greater clarity could be really useful here.
I’ve heard multiple reports of people being denied jobs around AI policy because of their history in EA. I’ve also seen a lot of animosity against EA from top organizations I think are important—like A16Z, Founders Fund (Thiel), OpenAI, etc. I’d expect that it would be uncomfortable for EAs to apply or work in to these latter places at this point.
This is very frustrating to me.
First, it makes it much more difficult for EAs to collaborate with many organizations where these perspectives could be the most useful. I want to see more collaborations and cooperation—not having EAs be allowed in many orgs makes this very difficult.
Second, it creates a massive incentive for people not to work in EA or on EA topics. If you know it will hurt your career, then you’re much less likely to do work here.
And a lighter third—it’s just really not fun to have a significant stigma associated with you. This means that many of the people I respect the most, and think are doing some of the most valuable work out there, will just have a much tougher time in life.
Who’s at fault here? I think the first big issue is that resistances get created against all interesting and powerful groups. There are similar stigmas against people across the political spectrum, for example, to certain crowds. A big part of “talking about morality and important issues, while having something non-obvious to say” is being hated by a bunch of people. In this vein, arguably we should be aiming for a world where it winds up that there’s a larger stigma.
But a lot clearly has to do with the decisions made by what seems like a few EAs. FTX hurt the most. I think the OpenAI board situation resulted in a lot of ea-paranoia, arguably with very little upside. More recently, I think that certain EA actions in ai policy are getting a lot of flak.
There was a brief window, pre-FTX-fail, where there was a very positive EA media push. I’ve seen almost nothing since. I think that “EA marketing” has been highly neglected, and that doesn’t seem to be changing.
Also, I suspect that the current EA AI policy arm could find ways to be more diplomatic and cooperative. When this arm upsets people, all of EA gets blamed.
My guess is that there are many other changes to do here too.
CEA is the obvious group to hypothetically be in charge of the EA parts of this. In practice, it seems like CEA has been very busy with post-FTX messes and leadership changes.
So I think CEA, as it becomes stable, could do a lot of good work making EA marketing work somehow. And I hope that the AI safety governance crowd can get better at not pissing off people. And hopefully, other EAs can figure out other ways to make things better and not worse.
If the above doesn’t happen, honestly, it could be worth it for EAs themselves to try to self-fund or coordinate efforts on this. The issue isn’t just one of “hurting long-term utility”, it’s one that just directly hurts EAs—so it could make a lot of sense for them to coordinate on improvements, even just in their personal interests.
On the positive front, I know its early days but GWWC have really impressed me with their well produced, friendly yet honest public facing stuff this year—maybe we can pick up on that momentum?
Also EA for Christians is holding a British conference this year where Rory Stewart and the Archbishop of Canterbury (biggest shot in the Anglican church) are headlining which is a great collaboration with high profile and well respected mainstream Christian / Christian-adjacent figures.
Any examples you wish to highlight?
I think in general their public facing presentation and marketing seems a cut above any other EA org—happy to be proven wrong by other orgs which are doing a great job too. What I love is how they present their messages with such positivity, while still packing a real punch and not watering down their message. Check out their web-page and blog to see their work.
A few concrete examples
- This great video “How rich are you really?”
- Nice rebranding of “Giving what we can pledge” to the snappier and clearer “10% pledge”—
The diamond symbol as a simple yet strong sign of people taking the pledge, both on the forum here and linkedin
- An amazing linked-in push with lots of people putting the diamond and explaining why they took the pledge. Many posts have been received really positively on my wall.
That’s just what I’ve noticed.
(Jumping in for our busy comms/exec team) Understanding the status of the EA brand and working to improve it is a top priority for CEA :) We hope to share more work on this in future.
Thanks, good to hear! Looking forward to seeing progress here.
I wrote a downvoted post recently about how we should be warning AI Safety talent about going into labs for personal branding reasons (I think there are other reasons not to join labs, but this is worth considering).
I think people are still underweighting how much the public are going to hate labs in 1-3 years.
I was telling organizers with PauseAI like Holly Elmore they should be emphasizing this more several months ago.
I think from an advocacy standpoint it is worth testing that message, but based on how it is being received on the EAF, it might just bounce off people.
My instinct as to why people don’t find it a compelling argument;
They don’t have short timelines like me, and therefore chuck it out completely
Are struggling to imagine a hostile public response to 15% unemployment rates
Copium
At least at the time, Holly Elmore seemed to consider it at least somewhat compelling. I mentioned this was an argument I provided framed in the context of movements like PauseAI—a more politicized, and less politically averse coalition movement, that includes at least one arm of AI safety as one of its constituent communities/movements, distinct from EA.
>They don’t have short timelines like me, and therefore chuck it out completely
Among the most involved participants in PauseAI, presumably there may estimates of short timelines comparable to the rate of such estimates among effective altruists.
>Are struggling to imagine a hostile public response to 15% unemployment rates
Those in PauseAI and similar movements don’t.
>Copium
While I sympathize with and appreciate why there would be high rates of huffing copium among effective altruists (and adjacent communities, such as rationalists), others who have been picking up slack effective altruists have dropped in the last couple years, are reacting differently. At least in terms of safeguarding humanity from both the near-term and long-term vicissitudes of advancing AI, humanity has deserved better than EA has been able to deliver. Many have given up hope that EA will ever rebound to the point it’ll be able to muster living up to the promise of at least trying to safeguard humanity. That includes both many former effective altruists, and those who still are effective altruists. I consider there to still be that kind of ‘hope’ on a technical level, though on a gut level I don’t have faith in EA. I definitely don’t blame those who have any faith left in EA, let alone those who see hope in it.
Much of the difference here is the mindset towards ‘people’, and how they’re modeled, between those still firmly planted in EA but somehow with a fatalistic mindset, and those who still care about AI safety but have decided to move on in EA. (I might be somewhere in between, though my perspective as a single individual among general trends is barely relevant.) The last couple years have proven that effective altruists direly underestimated the public, and the latter group of people didn’t. While many here on the EA Forum may not agree that much—or even most—of what movements like PauseAI are doing are as effective as they could or should be, they at least haven’t succumbed to a plague of doomerism beyond what can seemingly even be justified.
To quote former effective altruist Kerry Vaughan, in a message addressed to those who still are effective altruists: “now is not the time for moral cowardice.” There are some effective altruists who heeded that sort of call when it was being made. There are others who weren’t effective altruists who heeded it too, when they saw most effective altruists had lost the will to even try picking up the ball again after they dropped it a couple times. New alliances between emotionally determined effective altruists and rationalists, and thousands of other people the EA community always underestimated, might from now on be carrying the team that is the global project of AI risk reduction—from narrow/near-term AI, to AGI/ASI.
EA can still change, though either it has to go beyond self-reflection and just change already, or get used to no longer being team captain of AI Safety.
Very sorry to hear these reports, and was nodding along as I read the post.
If I can ask, how do they know EA affiliation was the decision? Is this an informal ‘everyone knows’ thing through policy networks in the US? Or direct feedback for the prospective employer than EA is a PR-risk?
Of course, please don’t share any personal information, but I think it’s important for those in the community to be as aware as possible of where and why this happens if it is happening because of EA affiliation/history of people here.
(Feel free to DM me Ozzie if that’s easier)
I’m thinking of around 5 cases. I think in around 2-3 they were told, the others it was strongly inferred.
Would you be happy to expand on these points?
On Twitter, a lot of VCs and techies have ranted heavily about how much they dislike EAs.
See this segment from Marc Andreeson, where he talks about the dangers of Eliezer and EA. Marc seems incredibly paranoid about the EA crowd now.
(Go to 1 hour, 11min in, for the key part. I tried linking to the timestamp, but couldn’t get it to work in this editor after a few minutes of attempts)
I also came across this transcript, from Amjad Masad, CEO of Replit, on Tucker Carlson, recently
https://www.happyscribe.com/public/the-tucker-carlson-show/amjad-masad-the-cults-of-silicon-valley-woke-ai-and-tech-billionaires-turning-to-trump
To be fair to the CEO of Replit here, much of that transcript is essentially true, if mildly embellished. Many of those events or outcomes associated with EA, or adjacent communities during their histories, that should be the most concerning to anyone other than any FTX-related events and for reasons beyond just PR concerns, can and have been well-substantiated.
My guess is this is obvious, but the “debugging” stuff seems as far as I can tell completely made up.
I don’t know of any story in which “debugging” was used in any kind of collective way. There was some Leverage-research adjacent stuff that kind of had some attributes like this, “CT-charting”, which maybe is what it refers to, but that sure would be the wrong word, and I also don’t think I’ve ever heard of any psychoses or anything related to that.
The only in-person thing I’ve ever associated with “debugging” is when at CFAR workshops people were encouraged to create a “bugs-list”, which was just a random list of problems in your life, and then throughout the workshop people paired with other people where they could choose any problem of their choosing, and work with their pairing partner on fixing it. No “auditing” or anything like that.
I haven’t read the whole transcript in-detail, but this section makes me skeptical of describing much of that transcript as “essentially true”.
I have personally heard several CFAR employees and contractors use the word “debugging” to describe all psychological practices, including psychological practices done in large groups of community members. These group sessions were fairly common.
In that section of the transcript, the only part that looks false to me is the implication that there was widespread pressure to engage in these group psychology practices, rather than it just being an option that was around. I have heard from people in CFAR who were put under strong personal and professional pressure to engage in *one-on-one* psychological practices which they did not want to do, but these cases were all within the inner ring and AFAIK not widespread. I never heard any stories of people put under pressure to engage in *group* psychological practices they did not want to do.
For what it’s worth, I was reminded of Jessica Taylor’s account of collective debugging and psychoses as I read that part of the transcript. (Rather than trying to quote pieces of Jessica’s account, I think it’s probably best that I just link to the whole thing as well as Scott Alexander’s response.)
I presume this account is their source for the debugging stuff, wherein an ex-member of the rationalist Leverage institute described their experiences. They described the institute as having “debugging culture”, described as follows:
and:
The podcast statements seem to be an embellished retelling of the contents of that blog post (and maybe the allegations made by scott alexander in the comments of this post). I don’t think describing them as “completely made up” is accurate.
Leverage was an EA-aligned organization, that was also part of the rationality community (or at least ‘rationalist-adjacent’), about a decade ago or more. For Leverage to be affiliated with the mantles of either EA or the rationality community was always contentious. From the side of EA, the CEA, and the side of the rationality community, largely CFAR, Leverage faced efforts to be shoved out of both within a short order of a couple of years. Both EA and CFAR thus couldn’t have then, and couldn’t now, say or do more to disown and disavow Leverage’s practices from the time Leverage existed under the umbrella of either network/ecosystem/whatever. They have. To be clear, so has Leverage in its own way.
At the time of the events as presented by Zoe Curzi in those posts, Leverage was basically shoved out the door of both the rationality and EA communities with—to put it bluntly—the door hitting Leverage on ass on the on the way out, and the door back in firmly locked behind them from the inside. In time, Leverage came to take that in stride, as the break-up between Leverage, and the rest of the institutional polycule that is EA/rationality, was extremely mutual.
Ien short, the course of events, and practices at Leverage that led to them, as presented by Zoe Curzi and others as a few years ago from that time circa 2018 to 2022, can scarcely be attributed to either the rationality or EA communities. That’s a consensus between EA, Leverage, and the rationality community agree on—one of few things left that they still agree on at all.
While I’m not claiming that “practices at Leverage” should be “attributed to either the rationality or EA communities”, or to CEA, the take above is demonstrably false. CEA definitely could have done more to “disown and disavow Leverage’s practices” and also reneged on commitments that would have helped other EAs learn about problems with Leverage.
Circa 2018 CEA was literally supporting Leverage/Paradigm on an EA community building strategy event. In August 2018 (right in the middle of the 2017-2019 period at Leverage that Zoe Curzi described in her post), CEA supported and participated in an “EA Summit” that was incubated by Paradigm Academy (intimately associated with Leverage). “Three CEA staff members attended the conference” and the keynote was delivered by a senior CEA staff member (Kerry Vaughan). Tara MacAulay, who was CEO of CEA until stepping down less than a year before the summit to co-found Alameda Research, personally helped fund the summit.
At the time, “the fact that Paradigm incubated the Summit and Paradigm is connected to Leverage led some members of the community to express concern or confusion about the relationship between Leverage and the EA community.” To address those concerns, Kerry committed to “address this in a separate post in the near future.” This commitment was subsequently dropped with no explanation other than “We decided not to work on this post at this time.”
This whole affair was reminiscent of CEA’s actions around the 2016 Pareto Fellowship, a CEA program where ~20 fellows lived in the Leverage house (which they weren’t told about beforehand), “training was mostly based on Leverage ideas”, and “some of the content was taught by Leverage staff and some by CEA staff who were very ‘in Leverage’s orbit’.” When CEA was fundraising at the end of that year, a community member mentioned that they’d heard rumors about a lack of professionalism at Pareto. CEA staff replied, on multiple occasions, that “a detailed review of the Pareto Fellowship is forthcoming.” This review was never produced.
Several years later, details emerged about Pareto’s interview process (which nearly 500 applicants went through) that confirmed the rumors about unprofessional behavior. One participant described it as “one of the strangest, most uncomfortable experiences I’ve had over several years of being involved in EA… It seemed like unscientific, crackpot psychology… it felt extremely cultish… The experience left me feeling humiliated and manipulated.”
I’ll also note that CEA eventually added a section to its mistakes page about Leverage, but not until 2022, and only after Zoe had published her posts and a commenter on Less Wrong explicitly asked why the mistakes page didn’t mention Leverage’s involvement in the Pareto Fellowship. The mistakes page now acknowledges other aspects of the Leverage/CEA relationship, including that Leverage had “a table at the careers fair at EA Global several times.” Notably, CEA has never publicly stated that working with Leverage was a mistake or that Leverage is problematic in any way.
The problems at Leverage were Leverage’s fault, not CEA’s. But CEA could have, and should have, done more to distance EA from Leverage.
Quick point—I think the relationship between CEA and Leverage was pretty complicated during a lot of this period.
There was typically a large segment of EAs who were suspicious of Leverage, ever since their founding. But Leverage did collaborate with EAs on some specific things early on (like the first EA Summit). It felt like an uncomfortable alliance type situation. If you go back on the forum / Lesswrong, you can read artifacts.
I think the period of 2018 or so was unusual. This was a period where a few powerful people at CEA (Kerry, Larissa) were unusually pro-Leverage and got to power fairly quickly (Tara left, somewhat suddenly). I think there was a lot of tension around this decision, and when they left (I think this period lasted around 1 year), I think CEA became much less collaborative with Leverage.
One way to square this a bit is that CEA was just not very powerful for a long time (arguably, its periods of “having real ability/agency to do new things” have been very limited). There were periods where Leverage had more employees (I’m pretty sure). The fact that CEA went through so many different leaders, each with different stances and strategies, makes it more confusing to look back on.
I would really love for a decent journalist to do a long story on this history, I think it’s pretty interesting.
Huh, yeah, that sure refers to those as “debugging”. I’ve never really heard Leverage people use those words, but Leverage 1.0 was a quite insular and weird place towards the end of its existence, so I must have missed it.
I think it’s kind of reasonable to use Leverage as evidence that people in the EA and Rationality community are kind of crazy and have indeed updated on the quotes being more grounded (though I also feel frustration with people equivocating between EA, Rationality and Leverage).
(Relatedly, I don’t particularly love you calling Leverage “rationalist” especially in a context where I kind of get the sense you are trying to contrast it with “EA”. Leverage has historically been much more connected to the EA community, and indeed had almost successfully taken over CEA leadership in ~2019, though IDK, I also don’t want to be too policing with language here)
I think it might describe how some people experienced internal double cruxing. I wouldn’t be that surprised if some people also found the ’debugging” frame in general to give too much agency to others relative to themselves, I feel like I’ve heard that discussed.
Based on the things titotal said, seems like it very likely refers to some Leverage stuff, which I feel a bit bad about seeing equivocated with the rest of the ecosystem, but also seems kind of fair. And the Zoe Curzi post sure uses the term “debugging” for those sessions (while also clarifying that the rest of the rationality community doesn’t use the term that way, but they sure seemed to)
I wouldn’t and didn’t describe that section of the transcript, as a whole, as essentially true. I said much of it is. As the CEO might’ve learned from Tucker Carlson, who in turned learned from FOX News, we should seek to be ‘fair and balanced.’
As to the debugging part, that’s an exaggeration that must have come out the other side of a game of broken telephone on the internet. It seems that on the other side of that telephone line would’ve been some criticisms or callouts I’ve read years ago of some activities happening in or around CFAR. I don’t recollect them in super-duper precise detail right now, nor do I have the time today to spend an hour or more digging them up on the internet
For the perhaps wrongheaded practices that were introduced into CFAR workshops for a period of time other than the ones from Leverage Research, I believe the others were some introduced by Valentine (e.g., ‘againstness,’ etc.). As far as I’m aware, at least as it was applied at one time, some past iterations of Connection Theory bore at least a superficial resemblance to some aspects of ‘auditing’ as practiced by Scientologists.
As to perhaps even riskier practices, I mean they happened not “in” but “around” CFAR in the sense of not officially happening under the auspices of CFAR, or being formally condoned by them, though they occurred within the CFAR alumni community and the Bay Area rationality community. It’s murky, though there was conduct in the lives of private individuals that CFAR informally enabled or emboldened, and could’ve/should’ve done more to prevent. For the record, I’m aware CFAR has effectively admitted those past mistakes, so I don’t want to belabor any point of moral culpability beyond what has been drawn out to death on LessWrong years ago.
Anyway, activities that occurred among rationalists in the social network that in CFAR’s orbit, that arguably arose to the level of triggering behaviour comparable in extremity to psychosis, include ‘dark arts’ rationality, and some of the edgier experiments of post-rationalists. That includes some memes spread and behaviours induced in some rationalists by Michael Vassar, Brent Dill, etc.
To be fair, I’m aware much of that was a result not of spooky, pseudo-rationality techniques, but some unwitting rationalists being effectively bullied into taking wildly mind-altering drugs, as guinea pigs in some uncontrolled DIY experiment. While responsibility for these latter outcomes may not be as attributable to CFAR, they can be fairly attributed to some past mistakes of the rationality community, albeit on a vague, semi-collective level.
I think it’s worth noting that the two examples you point to are right-wing, which the vast majority of Silicon Valley is not. Right-wing tech ppl likely have higher influence in DC, so that’s not to say they’re irrelevant, but I don’t think they are representative of silicon valley as a whole
I think Garry Tan is more left-wing, but I’m not sure. A lot of the e/acc community fights with EA, and my impression is that many of them are leftists.
I think that the right-wing techies are often the loudest, but there are also lefties in this camp too.
(Honestly though, the right-wing techies and left-wing techies often share many of the same policy ideas. But they seem to disagree on Trump and a few other narrow things. Many of the recent Trump-aligned techies used to be more left-coded.)
Random Tweet from today: https://x.com/garrytan/status/1820997176136495167
Garry Tan is the head of YCombinator, which is basically the most important/influential tech incubator out there. Around 8 years back, relations were much better, and 80k and CEA actually went through YCombinator.
I’d flag that Garry specifically is kind of wacky on Twitter, compared to previous heads of YC. So I definitely am not saying it’s “EA’s fault”—I’m just flagging that there is a stigma here.
I personally would be much more hesitant to apply to YC knowing this, and I’d expect YC would be less inclined to bring in AI safety folk and likely EAs.
I find it very difficult psychologically to take someone seriously if they use the word ‘decels’.
Want to say that I called this ~9 months ago.[1]
I will re-iterate that clashes of ideas/worldviews[2] are not settled by sitting them out and doing nothing, since they can be waged unilaterally.
Especially if you look at the various other QTs about this video across that side of Twitter
Or ‘memetic wars’, YMMV
My impression is that the current EA AI policy arm isn’t having much active dialogue with the VC community and the like. I see Twitter spats that look pretty ugly, I suspect that this relationship could be improved on with more work.
At a higher level, I suspect that there could be a fair bit of policy work that both EAs and many of these VCs and others would be more okay with than what is currently being pushed. My impression is that we should be focused on narrow subsets of risks that matter a lot to EAs, but don’t matter much to others, so we can essentially trade and come out better than we are now.
That seems like the wrong play to me. We need to be focused on achieving good outcomes and not being popular.
My personal take is that there are a bunch of better trade-offs between the two that we could be making. I think that the narrow subset of risks is where most of the value is, so from that standpoint, that could be a good trade-off.
I’m nervous that the EA Forum might be having a small role for x-risk and some high-level prioritization work.
- Very little biorisk content here, perhaps because of info-hazards.
- Little technical AI safety work here, in part because that’s more for LessWrong / Alignment Forum.
- Little AI governance work here, for whatever reason.
- Not too much innovative, big-picture longtermist prioritization projects happening at the moment, from what I understand.
- The cause of “EA community building” seems to be fairly stable, not much bold/controversial experimentation, from what I can tell.
- Fairly few updates / discussion from grantmakers. OP is really the dominant one, and doesn’t publish too much, particularly about their grantmaking strategies and findings.
It’s been feeling pretty quiet here recently, for my interests. I think some important threads are now happening in private slack / in-person conversations or just not happening.
I don’t comment or post much on the EA forum because the quality of discourse on the EA Forum typically seems mediocre at best. This is especially true for x-risk.
I think this has been true for a while.
Any ideas for what we can do to improve it?
The whole manifund debacle has left me quite demotivated. It really sucks that people are more interested debating contentious community drama, than seemingly anything else this forum has to offer.
Thanks for the reminder, definitely got sucked in too much myself....
Will get back to commenting more on GHD posts and write another of my own soon!
What’s the “whole manifund debacle”? People complaining about Curtis Yarvin or something?
https://forum.effectivealtruism.org/posts/34pz6ni3muwPnenLS/why-so-many-racists-at-manifest
I think there signal vs noise tradeoffs, so I’m naively tempted to retreat toward more exclusivity.
This poses costs of its own, so maybe I’d be in favor of differentiation (some more and some less exclusive version).
Low confidence in this being good overall.
Hi Ryan,
Could you share a few examples of what you consider good quality EA Forum posts? Do you think the content linked on the EA Forum Digest also “typically seems mediocre at best”?
When I write biorisk-related things publicly I’m usually pretty unsure of whether the Forum is a good place for them. Not because of info-hazards, since that would gate things at an earlier stage, but because they feel like they’re of interest to too small a fraction of people. For example, I could plausibly have posted Quick Thoughts on Our First Sampling Run or some of my other posts from https://data.securebio.org/jefftk-notebook/ here, but that felt a bit noisy?
It also doesn’t help that detailed technical content gets much less attention than meta or community content. For example, three days ago I wrote a comment on @Conrad K.’s thoughtful Three Reasons Early Detection Interventions Are Not Obviously Cost-Effective, and while I feel like it’s a solid contribution only four people have voted on it. On the other hand, if you look over my recent post history at my comments on Manifest, far less objectively important comments have ~10x the karma. Similarly the top level post was sitting at +41 until Mike bumped it last week, which wasn’t even high enough that (before I changed my personal settings to boost biosecurity-tagged posts) I saw it when it came out. I see why this happens—there are a lot more people with the background to engage on a community topic or even a general “good news” post—but it still doesn’t make me as excited to contribute on technical things here.
I’m with Ozzie here. I think EA Forum would do better with more technical content even if it’s hard for most people to engage with.
I’d be excited to have discussions of those posts here!
A lot of my more technical posts also get very little attention—I also find that pretty unmotivating. It can be quite frustrating when clearly lower-quality content on controversial stuff gets a lot more attention.
But this seems like a doom loop to me. I care much more about strong technical content, even if I don’t always read it, than I do most of the community drama. I’m sure most leaders and funders feel similarly.
Extended far enough, the EA Forum will be a place only for controversial community drama. This seems nightmarish to me. I imagine most forum members would agree.
I imagine that there are things the Forum or community can do to bring more attention or highlighting to the more technical posts.
Here you go: Detecting Genetically Engineered Viruses With Metagenomic Sequencing
But this was already something I was going to put on the Forum ;)
I wonder if the forum is even a good place for a lot of these discussions? Feels like they need some combination of safety / shared context, expertise, gatekeeping etc?
If it’s not, there is a question of what the EA Forum’s comparative advantage will be in the future, and what is a good place for these discussions.
Personally, I think this forum could be good for at least some of this, but I’m not sure.
Three use cases come to mind for the forum:
establishing a reputation in writing as a person who can follow good argumentative norms (perhaps as a kind of extended courtship of EA jobs/orgs)
disseminating findings that are mainly meant for other forums, e.g. research reports
keeping track of what the community at large is thinking about/working on, which is mostly facilitated by organizations like RP & GiveWell using the forum to share their work.
I don’t think I would use the forum for hashing out anything I was really thinking hard about; I’d probably have in-person conversations or email particular persons.
I don’t know about you but I just learned about one of the biggest updates to OPs grantmaking in a year on the Forum.
That said, the data does show some agreement with your and commenters vibe of lowering quantity.
I agree that the Forum could be a good place for a lot of these discussions. Some of them aren’t happening at all to my knowledge.[1] Some of those should be, and should be discussed on the Forum. Others are happening in private and that’s rational, although you may be able to guess that my biased view is that a lot more should be public, and if they were, should be posted on the Forum.
Broadly: I’m quite bullish on the EA community as a vehicle for working on the world’s most pressing problems, and of open online discussion as a piece of our collective progress. And I don’t know of a better open place on the internet for EAs to gather.
Part of that might be because as EA gets older the temperature (in the annealing sense) rationally lowers.
Yep—I liked the discussion in that post a lot, but the actual post seemed fairly minimal, and written primarily outside of the EA Forum (it was a link post, and the actual post was 320 words total.)
For those working on the forum, I’d suggest work on bringing in more of these threads to the forum. Maybe reach out to some of the leaders in each group and see how to change things.
I think that AI policy in particular is most ripe for better infrastructure (there’s a lot of work happening, but no common public forums, from what I know), though it probably makes sense to be separate from the EA Forum (maybe like the Alignment Forum), because a lot of them don’t want to be associated too much with EA, for policy reasons.
I know less about Bio governance, but would strongly assume that a whole lot of it isn’t infohazardous. That’s definitely a field that’s active and growing.
For foundational EA work / grant discussions / community strategy, I think we might just need more content in the first place, or something.
I assume that AI alignment is well-handled by LessWrong / Alignment Forum, difficult and less important to push to happen here.
So I did used to do more sort of back of the envelope stuff, but it didn’t get much traction and people seemed to think it was unfished (it was) so I guess I had less enthusiasm.
Yeah even on the global health front the last 3 months or so have felt especially quiet
Curious if you think there was good discussion before that and could point me to any particularly good posts or conversations?
There are still bunch of good discussions (see mostly posts with 10+ comments) in the last 6 months or so, its just that we can sometimes even go a week or two without more than one or two ongoing serious GHD chats. Maybe I’m wrong and there hasn’t actually been much (or any) meaningful change in activity this year looking at this.
https://forum.effectivealtruism.org/?tab=global-health-and-development
As a random datapoint, I’m only just getting into the AI Governance space, but I’ve found little engagement with (some) (of[1]) (the) (resources) I’ve shared and have just sort of updated to think this is either not the space for it or I’m just not yet knowledgeable enough about what would be valuable to others.
I was especially disappointed with this one, because this was a project I worked on with a team for some time, and I still think it’s quite promising, but it didn’t receive the proportional engagement I would have hoped for. Given I optimized some of the project for putting out this bit of research specifically, I wouldn’t do the same now and would have instead focused on other parts of the project.
It seems from the comments that there’s a chance that much of this is just timing—i.e. right now is unusually quiet. It is roughly mid-year, maybe people are on vacation or something, it’s hard to tell.
I think that this is partially true. I’m not interested in bringing up this point to upset people, but rather to flag that maybe there could be good ways of improving this (which I think is possible!)
Personal reflections on self-worth and EA
My sense of self-worth often comes from guessing what people I respect think of me and my work.
In EA… this is precarious. The most obvious people to listen to are the senior/powerful EAs.
In my experience, many senior/powerful EAs I know:
1. Are very focused on specific domains.
2. Are extremely busy.
3. Have substantial privileges (exceptionally intelligent, stable health, esteemed education, affluent/ intellectual backgrounds.)
4. Display limited social empathy (ability to read and respond to the emotions of others)
5. Sometimes might actively try not to sympathize/empathize with many people, because they are judging them for grants, and want don’t want to be biased. (I suspect this is the case for grantmakers).
6. Are not that interested in acting as a coach/mentor/evaluator to people outside their key areas/organizations.
7. Don’t intend or want others to care too much about what they think outside of cause-specific promotion and a few pet ideas they want to advance.
A parallel can be drawn with the world of sports. Top athletes can make poor coaches. Their innate talent and advantages often leave them detached from the experiences of others. I’m reminded by David Foster Wallace’s How Tracy Austin Broke My Heart.
If you’re a tennis player, tying your self-worth to what Roger Federer thinks of you is not wise. Top athletes are often egotistical, narrow-minded, and ambivalent to others. This sort of makes sense by design—to become a top athlete, you often have to obsess over your own abilities to an unnatural extent for a very long period.
Good managers are sometimes meant to be better as coaches than they are as direct contributors. In EA, I think those in charge seem more like “top individual contributors and researchers” than they do “top managers.” Many actively dislike management or claim that they’re not doing management. (I believe funders typically don’t see their work as “management*”, which might be very reasonable.)
But that said, even a good class of managers wouldn’t fully solve the self-worth issue. Tying your self-worth too much to your boss can be dangerous—your boss already has much power and control over you, so adding your self-worth to the mix seems extra precarious.
I think if I were to ask any senior EA I know, “Should I tie my self-worth with your opinion of me?” they would say something like,
“Are you insane? I barely know you or your work. I can’t at all afford the time to evaluate your life and work enough to form an opinion that I’d suggest you take really seriously.”
They have enough problems—they don’t want to additionally worry about others trying to use them as judges of personal value.
But this raises the question, Who, if anyone, should I trust to inform my self-worth?
Navigating intellectual and rationalist literature, I’ve grown skeptical of many other potential evaluators. Self-judgment carries inherent bias and ability to Goodhart. Many “personal coaches” and even “executive coaches” raise my epistemic alarm bells. Friends, family, and people who are “more junior” come with different substantial biases.
Some favored options are “friends of a similar professional class who could provide long-standing perspective” and “professional coaches/therapists/advisors.”
I’m not satisfied with any obvious options here. I think my next obvious move forward is to acknowledge that my current situation seems subpar and continue reflecting on this topic. I’ve dug into the literature a bit but haven’t found answers I’ve yet found compelling.
My initial thought is that it is pretty risky/tricky/dangerous to depend on external things for a sense of self-worth? I know that I certainly am very far away from an Epictetus-like extreme, but I try to not depend on the perspectives of other people for my self-worth. (This is aspirational, of course. A breakup or a job loss or a person I like telling me they don’t like me will hurt and I’ll feel bad for a while.)
A simplistic little thought experiment I’ve fiddled with: if I went to a new place where I didn’t know anyone and just started over, then what? Nobody knows you, and you social circle starts from scratch. That doesn’t mean that you don’t have a worth as a human being (although it might mean that you don’t have any worth in the ‘economic’ sense of other people wanting you, which is very different).
There might also be an intrinsic/extrinsic angle to this. If you evaluate yourself based on accomplishments, outputs, achievements, and so on, that has a very different feeling than the deep contentment of being okay as you are.
In another comment Austin mentions revenue and funding, but that seems to be a measure of things VERY different from a sense of self-worth (although I recognize that there are influential parts of society in which wealth or career success is seen as the proxies for worth). In favorable market conditions I have high self worth?
I would roughly agree with your idea of “trying not to tie my emotional state to my track record.”
I can relate, as someone who also struggles with self-worth issues. However, my sense of self-worth is tied primarily to how many people seem to like me / care about me / want to befriend me, rather than to what “senior EAs” think about my work.
I think that the framing “what is the objectively correct way to determine my self-worth” is counterproductive. Every person has worth by virtue of being a person. (Even if I find it much easier to apply this maxim to others than to myself.)
IMO you should be thinking about things like, how to do better work, but in the frame of “this is something I enjoy / consider important” rather than in the frame of “because otherwise I’m not worthy”. It’s also legitimate to want other people to appreciate and respect you for your work (I definitely have a strong desire for that), but IMO here also the right frame is “this is something I want” rather than “this is something that’s necessary for me to be worth something”.
It’s funny, I think you’d definitely be in the list of people I respect and care about their opinion of me. I think it’s just imposter syndrome all the way up.
Personally, one thing that seemed to work a bit for me is to find peers which I highly appreciate and respect and schedule weekly calls with them to help me prioritize and focus, and give me feedback.
A few possibilities from startup land:
derive worth from how helpful your users find your product
chase numbers! usage, revenue, funding, impact, etc. Sam Altman has a line like “focus on adding another 0 to your success metric”
the intrinsic sense of having built something cool
After transitioning from for-profit entrepreneurship to co-leading a non-profit in the effective altruism space, I struggle to identify clear metrics to optimize for. Funding is a potential metric, but it is unreliable due to fluctuations in donors’ interests. The success of individual programs, such as user engagement with free products or services, may not accurately reflect their impact compared to other potential initiatives. Furthermore, creating something impressive doesn’t necessarily mean it’s useful.
Lacking a solid impact evaluation model, I find myself defaulting to measuring success by hours worked, despite recognizing the diminishing returns and increased burnout risk this approach entails.
This is brave of you to share. It sounds like there are a few related issues going on. I have a few thoughts that may or may not be helpful:
Firstly, you want to do well and improve in your work, and you want some feedback on that from people who are informed and have good judgment. The obvious candidates in the EA ecosystem are people who actually aren’t well suited to give this feedback to you. This is tough. I don’t have any advice to give you here.
However it also sounds like there are some therapeutic issues at play. You mention therapists as a favored option but one you’re not satisfied with and I’m wondering why? Personally I suspect that making progress on any therapeutic issues that may be at play may also end up helping with the professional feedback problem.
I think you’ve unfairly dismissed the best option as to who you can trust: yourself. That you have biases and flaws is not an argument against trusting yourself because everyone and everything has biases and flaws! Which person or AI are you going to find that doesn’t have some inherent bias or ability to Goodhart?
Five reasons why I think it’s unhelpful connecting our intrinsic worth to our instrumental worth (or anything aside from being conscious beings):
Undermines care for others (and ourselves): chickens have limited instrumental worth and often do morally questionable things. I still reckon chickens and their suffering are worthy of care. (And same argument for human babies, disabled people and myself)
Constrains effective work: continually assessing our self-worth can be exhausting (leaving less time/attention/energy for actually doing helpful work). For example, it can be difficult to calmy take on constructive feedback (on our work, or instrumental strengths or instrumental weaknesses) when our self-worth is on the line.
Constrains our personal wellbeing and relationships: I’ve personally found it hard to enjoy life when continuously questioning my self-worth and feeling guilty/shameful when the answer seems negative
Very hard to answer: including because the assessment may need to be continuously updated based on the new evidence from each new second of our lives
Seems pointless to answer (to me): how would accurately measuring our self-worth (against a questionable benchmark) make things better? We could live in a world where all beings are ranked so that more ‘worthy’ beings can appropriately feel superior, and less ‘worthy’ beings can appropriately feel ‘not enough’. This world doesn’t seem great from perspective
Despite thinking these things, I often unintentionally get caught up muddling my self-worth with my instrumental worth (can relate to the post and comments on here!) I’ve found ‘mindful self-compassion’ super helpful for doing less of this
This is an interesting post and seems basically right to me, thanks for sharing.
Thank you, this very much resonates with me
The most obvious moves, to me, eventually, are to either be intensely neutral (as in, trying not to tie my emotional state to my track record), or to iterate on using AI to help here (futuristic and potentially dangerous, but with other nice properties).
How would you use AI here?
A very simple example is, “Feed a log of your activity into an LLM with a good prompt, and have it respond with assessments of how well you’re doing vs. your potential at the time, and where/how you can improve.” You’d be free to argue points or whatever.
Reading this comment makes me think that you are basing your self-worth on your work output. I don’t have anything concrete to point to, but I suspect that this might have negative effects on happiness, and that being less outcome dependent will tend to result in a better emotional state.
That’s cool. I had the thought of developing a “personal manager” for myself of some form for roughly similar purposes
(This is a draft I wrote in December 2021. I didn’t finish+publish it then, in part because I was nervous it could be too spicy. At this point, with the discussion post-chatGPT, it seems far more boring, and someone recommended I post it somewhere.)
Thoughts on the OpenAI Strategy
OpenAI has one of the most audacious plans out there and I’m surprised at how little attention it’s gotten.
First, they say flat out that they’re going for AGI.
Then, when they raised money in 2019, they had a clause that says investors will be capped at getting 100x of their returns back.
On Hacker News, one of their employees says,
You can read more about this mission on the charter:
This is my [incredibly rough and speculative, based on the above posts] impression of the plan they are proposing:
Make AGI
Turn AGI into huge profits
Give 100x returns to investors
Dominate much (most?) of the economy, have all profits go to the OpenAI Nonprofit
Use AGI for “the benefit of all”?
I’m really curious what step 5 is supposed to look like exactly. I’m also very curious, of course, what they expect step 4 to look like.
Keep in mind that making AGI is a really big deal. If you’re the one company that has an AGI, and if you have a significant lead over anyone else that does, the world is sort of your oyster.[4] If you have a massive lead, you could outwit legal systems, governments, militaries.
I imagine that the 100x return cap means that the excess earnings would go to the hands of the nonprofit; which essentially means Sam Altman, senior leadership at OpenAI, and perhaps the board of directors (if legal authorities have any influence post-AGI).
This would be a massive power gain for a small subset of people.
If DeepMind makes AGI I assume the money would go to investors, which would mean it would be distributed to all of the Google shareholders. But if OpenAI makes AGI, the money will go to the leadership of OpenAI, on paper to fulfill the mission of OpenAI.
On the plus side, I expect that this subset is much more like the people reading this post than most other AGI competitors would be. (The Chinese government, for example). I know some people at OpenAI, and my hunch is that the people there are very smart and pretty altruistic. It might well be about the best we could expect from a tech company.
And, to be clear, it’s probably incredibly unlikely that OpenAI will actually create AGI, and even more unlikely they will do so with a decisive edge over competitors.
But, I’m sort of surprised so few other people seem at least a bit concerned and curious about the proposal? My impression is that most press outlets haven’t thought much at all about what AGI would actually mean, and most companies and governments just assume that OpenAI is dramatically overconfident in themselves.
(Aside on the details of Step 5)
I would love more information on Step 5, but I don’t blame OpenAI for not providing it.
Any precise description of how a nonprofit would spend “a large portion of the entire economy” would upset a bunch of powerful people.
Arguably, OpenAI doesn’t really need to figure out Step 5 unless their odds of actually having a decisive AGI advantage seem more plausible.
I assume it’s really hard to actually put together any reasonable plan now for Step 5.
My guess is that we really could use some great nonprofit and academic work to help outline what a positive and globally acceptable (wouldn’t upset any group too much if they were to understand it) Step 5 would look like. There’s been previous academic work on a “windfall clause”[5] (their 100x cap would basically count), having better work on Step 5 seems very obvious.
[1] https://openai.com/blog/openai-lp/
[2] https://news.ycombinator.com/item?id=19360709
[3] https://openai.com/charter/
[4] This was titled a “decisive strategic advantage” in the book Superintelligence by Nick Bostrom
[5] https://www.effectivealtruism.org/articles/cullen-okeefe-the-windfall-clause-sharing-the-benefits-of-advanced-ai/
Also, see:
https://www.cnbc.com/2021/03/17/openais-altman-ai-will-make-wealth-to-pay-all-adults-13500-a-year.html
Artificial intelligence will create so much wealth that every adult in the United States could be paid $13,500 per year from its windfall as soon as 10 years from now.
https://www.techtimes.com/articles/258148/20210318/openai-give-13-500-american-adult-anually-sam-altman-world.htm
https://moores.samaltman.com/
https://www.reddit.com/r/artificial/comments/m7cpyn/openais_sam_altman_artificial_intelligence_will/
Around EA Priorities:
Personally, I feel fairly strongly convinced to favor interventions that could help the future past 20 years from now. (A much lighter version of “Longtermism”).
If I had a budget of $10B, I’d probably donate a fair bit to some existing AI safety groups. But it’s tricky to know what to do with, say, $10k. And the fact that the SFF, OP, and others have funded some of the clearest wins makes it harder to know what’s exciting on-the-margin.
I feel incredibly unsatisfied with the public EA dialogue around AI safety strategy now. From what I can tell, there’s some intelligent conversation happening by a handful of people at the Constellation coworking space, but a lot of this is barely clear publicly. I think many people outside of Constellation are working on simplified models, like “AI is generally dangerous, so we should slow it all down,” as opposed to something like, “Really, there are three scary narrow scenarios we need to worry about.”
I recently spent a week in DC and found it interesting. But my impression is that a lot of people there are focused on fairly low-level details, without a great sense of the big-picture strategy. For example, there’s a lot of work into shovel-ready government legislation, but little thinking on what the TAI transition should really look like.
This sort of myopic mindset is also common in the technical space, where I meet a bunch of people focused on narrow aspects of LLMs, without much understanding of how their work exactly fits into the big picture of AI alignment. As an example, a lot of work seems like it would help with misuse risk, even when the big-picture EAs seem much more focused on accident risk.
Some (very) positive news is that we do have far more talent in this area than we did 5 years ago, and there’s correspondingly more discussion. But it still feels very chaotic.
A bit more evidence—it seems like OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?), but they have seemed to provide more support for AI safety community building / recruiting, and AI policy. But all of this still represents perhaps ~30% or so of their total budget, and I don’t sense that that’s about to change. Overall this comes off as measured and cautious. Meanwhile, it’s been difficult to convince other large donors to get into this area. (Other than Jaan Tallinn, he might well have been the strongest dedicated donor here).
Recently it seems like the community on the EA Forum has shifted a bit to favor animal welfare. Or maybe it’s just that the AI safety people have migrated to other blogs and organizations.
But again, I’m very hopeful that we can find interventions that will help in the long-term, so few of these excite me. I’d expect and hope that interventions that help the long-term future would ultimately improve animal welfare and more.
So on one hand, AI risk seems like the main intervention area for the long-term, but on the other, the field is a bit of a mess right now.
I feel quite frustrated that EA doesn’t have many other strong recommendations for other potential donors interested in the long-term. For example, I’d really hope that there could be good interventions to make the US government or just US epistemics more robust, but I barely see any work in that area.
“Forecasting” is one interesting area—it currently does have some dedicated support from OP. But it honestly seems to be in a pretty mediocre state to me right now. There might be 15-30 full-time people in the space at this point, and there’s surprisingly little in terms of any long-term research agendas.
Hi Ozzie – Peter Favaloro here; I do grantmaking on technical AI safety at Open Philanthropy. Thanks for this post, I enjoyed it.
I want to react to this quote:
…it seems like OP has provided very mixed messages around AI safety. They’ve provided surprisingly little funding / support for technical AI safety in the last few years (perhaps 1 full-time grantmaker?)
I agree that over the past year or two our grantmaking in technical AI safety (TAIS) has been too bottlenecked by our grantmaking capacity, which in turn has been bottlenecked in part by our ability to hire technical grantmakers. (Though also, when we’ve tried to collect information on what opportunities we’re missing out on, we’ve been somewhat surprised at how few excellent, shovel-ready TAIS grants we’ve found.)
Over the past few months I’ve been setting up a new TAIS grantmaking team, to supplement Ajeya’s grantmaking. We’ve hired some great junior grantmakers and expect to publish an open call for applications in the next few months. After that we’ll likely try to hire more grantmakers. So stay tuned!
That sounds exciting, thanks for the update. Good luck with team building and grantmaking!
Yeah, I find myself very confused by this state of affairs. Hundreds of people are being funneled through the AI safety community-building pipeline, but there’s little funding for them to work on things once they come out the other side.[1]
As well as being suboptimal from the viewpoint of preventing existential catastrophe, this also just seems kind of common-sense unethical. Like, all these people (most of whom are bright-eyed youngsters) are being told that they can contribute, if only they skill up, and then they later find out that that’s not the case.
These community-building graduates can, of course, try going the non-philanthropic route—i.e., apply to AGI companies or government institutes. But there are major gaps in what those organizations are working on, in my view, and they also can’t absorb so many people.
Yea, I think this setup has been incredibly frustrating downstream. I’d hope that people from OP with knowledge could publicly reflect on this, but my quick impression is that some of the following factors happened:
1. OP has had major difficulties/limitations around hiring in the last 5+ years. Some of this is lack of attention, some is that there aren’t great candidates, some is a lack of ability. This effected some cause areas more than others. For whatever reason, they seemed to have more success hiring (and retaining talent) for community than for technical AI safety.
2. I think there’s been some uncertainties / disagreements into how important / valuable current technical AI safety organizations are to fund. For example, I imagine if this were a major priority from those in charge of OP, more could have been done.
3. OP management seems to be a bit in flux now. Lost Holden recently, hiring a new head of GCR, etc.
4. I think OP isn’t very transparent and public with explaining their limitations/challenges publicly.
5. I would flag that there are spots at Anthropic and Deepmind that we don’t need to fund, that are still good fits for talent.
6. I think some of the Paul Christiano—connected orgs were considered a conflict-of-interest, given that Ajeya Cotra was the main grantmaker.
7. Given all of this, I think it would be really nice if people could at least provide warnings about this. Like, people entering the field are strongly warned that the job market is very limited. But I’m not sure who feels responsible / well placed to do this.
Thanks for the comment, I think this is very astute.
~
I think there’s a (mostly but not entirely accurate) vibe that all AI safety orgs that are worth funding will already be approximately fully funded by OpenPhil and others, but that animal orgs (especially in invertebrate/wild welfare) are very neglected.
I don’t think that all AI safety orgs are actually fully funded since there are orgs that OP cannot fund for reasons (see Trevor’s post and also OP’s individual recommendations in AI) other than cost-effectiveness and also OP cannot and should not fund 100% of every org (it’s not sustainable for orgs to have just one mega-funder; see also what Abraham mentioned here). Also there is room for contrarian donation takes like Michael Dickens’s.
That makes sense, but I’m feeling skeptical. There are just so many AI safety orgs now, and the technical ones generally aren’t even funded by OP.
For example: https://www.lesswrong.com/posts/9n87is5QsCozxr9fp/the-big-nonprofits-post
While a bunch of these salaries are on the high side, not all of them are.
On AI safety, I think it’s fairly likely (40%?) that the risk of x-risk (upon a lot of reflection) in the next 20 years is less than 20%, and that the entirety of the EA scene might be reducing it to say 15%.
This means that the entirety of the EA AI safety scene would help the EV of the world by ~5%.
On one hand, this is a whole lot. But on the other, I’m nervous that it’s not ambitious enough, for what could be one of the most [combination of well-resourced, well-meaning, and analytical/empirical] groups of our generation.
One thing I like about epistemic interventions is that the upper-bounds could be higher.
(There are some AI interventions that are more ambitious, but many do seem to be mainly about reducing x-risk by less than an order of magnitude, not increasing the steady-state potential outcome)
I’d also note here that an EV gain of 5% might not be particularly ambitious. It could well be the case that many different groups can do this—so it’s easier than it might seem if you think goodness is additive instead of multiplicative.
I want to see more discussion on how EA can better diversify and have strategically-chosen distance from OP/GV.
One reason is that it seems like multiple people at OP/GV have basically said that they want this (or at least, many of the key aspects of this).
A big challenge is that it seems very awkward for someone to talk and work on this issue, if one is employed under the OP/GV umbrella. This is a pretty clear conflict of interest. CEA is currently the main organization for “EA”, but I believe CEA is majority funded by OP, with several other clear strong links. (Board members, and employees often go between these orgs).
In addition, it clearly seems like OP/GV wants certain separation to help from their side. The close link means that problems with EA often spills over to the reputation of OP/GV.
I’d love to see some other EA donors and community members step up here. I think it’s kind of damning how little EA money comes from community members or sources other than OP right now. Long-term this seems pretty unhealthy.
One proposal is to have some “mini-CEA” that’s non-large-donor funded. This group’s main job would be to understand and act on EA interests that organizations funded by large donors would have trouble with.
I know Oliver Habryka has said that he thinks it would be good for the EA Forum to also be pulled away from large donors. This seems good to me, though likely expensive (I believe this team is sizable).
Another task here is to have more non-large-donor funding for CEA.
For large donors, one way of dealing with potential conflicts of interest would be doing funding in large blocks, like a 4-year contribution. But I realize that OP might sensibly be reluctant to do this at this point.
Also, related—I’d really hope that the EA Infrastructure Fund could help here, but I don’t know if this is possible for them. I’m dramatically more excited about large long-term projects on making EA more community-driven and independent, and/or well-managed, than I am the kinds of small projects they seem to fund. I don’t think they’ve ever funded CEA, despite that CEA might now represent the majority of funding on the direct EA community. I’d encourage people from this fund to think through this issue and be clear about what potential projects they might be excited for, around this topic.
Backing up a bit—it seems to me like EA is really remarkably powerless for what it is, outside of the OP/GV funding stream right now. This seems quite wrong to me, like large mistakes were made. Part of me think that positive change here is somewhat hopeless at this point (I’ve been thinking about this space for a few years now but haven’t taken much action because of uncertainty on this), but part of me also thinks that with the right cleverness or talent, there could be some major changes.
Another quick thought: This seems like a good topic for a “Debate Week”, in case anyone from that team is seeing this.
(To add clarity—I’m not suggesting that OP drops it’s funding of EA! It’s more that I think that non-OP donors should step up more, and that key EA services should be fairly independent.)
There was some prior relevant discussion in November 2023 in this CEA fundraising thread, such as my comment here about funder diversity at CEA. Basically, I didn’t think that there was much meaningful difference between a CEA that was (e.g.) 90% OP/GV funded vs. 70% OP/GV funded. So I think the only practical way for that percentage to move enough to make a real difference would be both an increase in community contributions/control and CEA going on a fairly severe diet.
As for EAIF, expected total grantmaking was ~$2.5MM for 2025. Even if a sizable fraction of that went to CEA, it would only be perhaps 1-2% of CEA’s 2023 budget of $31.4MM.
I recall participating in some discussions here about identifying core infrastructure that should be prioritized for broad-based funding for democratic and/or epistemic reasons. Identifying items in the low millions for more independent funding seems more realistic than meaningful changes in CEA’s funding base. The Forum strikes me as an obvious candidate, but a community-funded version would presumably need to run on a significantly leaner budget than I understand to be currently in place.
Personally, I’m optimistic that this could be done in specific ways that could be better than one might initially presume. One wouldn’t fund “CEA”—they could instead fund specific programs in CEA, for instance. I imagine that people at CEA might have some good ideas of specific things they could fund that OP isn’t a good fit for.
One complication is that arguably we’d want to do this in a way that’s “fair” to OP. Like, it doesn’t seem “fair” for OP to pay for all the stuff that both OP+EA agrees on, and EA only to fund the stuff that EA likes. But this really depends on what OP is comfortable with.
Lastly, I’d flag that CEA being 90% OP/GV funded really can be quite different than 70% in some important ways, still. For example, if OP/GV were to leave—then CEA might be able to go to 30% of its size—a big loss, but much better than 10% of its size.
That may be viable, although I think it would be better for both sides if these programs were not in CEA but instead in an independent organization. For the small-donor side, it limits the risk that their monies will just funge against OP/GV’s, or that OP/GV will influence how the community-funded program is run (e.g., through its influence on CEA management officials). On the OP/GV side, organizational separation is probably necessary to provide some of the reputational distance it may be looking for. That being said, given that small/medium donors have never to my knowledge been given this kind of opportunity, and the significant coordination obstacles involved, I would not characterize them not having taken it as indicative of much in particular.
~
More broadly, I think this is a challenging conversation without nailing down the objective better—and that may be hard for us on the Forum to do. Without any inside knowledge, my guess is that OP/GV’s concerns are not primarily focused on the existence of discrete programs “that OP isn’t a good fit for” or a desire not to fund them.
For example, a recent public comment from Dustin contain the following sentence: “But I can’t e.g. get SBF to not do podcasts nor stop the EA (or two?) that seem to have joined DOGE and started laying waste to USAID.” The concerns implied by that statement aren’t really fixable by the community funding discrete programs, or even by shelving discrete programs altogether. Not being the flagship EA organization’s predominant donor may not be sufficient for getting reputational distance from that sort of thing, but it’s probably a necessary condition.
I speculate that other concerns may be about the way certain core programs are run—e.g., I would not be too surprised to hear that OP/GV would rather not have particular controversial content allowed on the Forum, or have advocates for certain political positions admitted to EAGs, or whatever. I’m not going to name the content I have in mind in an attempt not to be drawn into an object-level discussion on those topics, but I wouldn’t want my own money being used to platform such content or help its adherents network either. Anyway, these types of issues can probably be fixed by running the program with community/other-donor funding in a separate organization, but these programs are expensive to run. And the community / non-OP/GV donors are not a monolithic constituency; I suspect that at least a significant minority of the community would share OP/GV’s concerns on the merits.
I agree—the linked comment was focused more on the impact of funding diversity on conflicts of interest and cause prio. But the amount of smaller-EA-donor dollars to go around is limited,[1] and so we have to consider the opportunity cost of diverting them to fund CEA or similar meta work on an ongoing basis. OP/GV is usually a pretty responsible funder, so the odds of them suddenly defunding CEA without providing some sort of notice and transitional funding seems low.
For instance, I believe GWWC pledgers gave about $32MM/year on average from 2020-2022 [p. 12 of this impact assessment], and not all pledgers are EAs.
I think you bring up a bunch of good points. I’d hope that any concrete steps on this would take these sorts of considerations in mind.
> The concerns implied by that statement aren’t really fixable by the community funding discrete programs, or even by shelving discrete programs altogether. Not being the flagship EA organization’s predominant donor may not be sufficient for getting reputational distance from that sort of thing, but it’s probably a necessary condition.
I wasn’t claiming that this funding change would fix all of OP/GV’s concerns. I assume that would take a great deal of work, among many different projects/initiatives.
One thing I care about is that someone is paid to start thinking about this critically and extensively, and I imagine they’d be more effective if not under the OP umbrella. So one of the early steps to take is just trying to find a system that could help figure out future steps.
> I speculate that other concerns may be about the way certain core programs are run—e.g., I would not be too surprised to hear that OP/GV would rather not have particular controversial content allowed on the Forum, or have advocates for certain political positions admitted to EAGs, or whatever.
I think this raises an important and somewhat awkward point that levels of separation between EA and OP/GV would make it harder for OP/GV to have as much control over these areas, and there are times where they wouldn’t be as happy with the results.
Of course:
1. If this is the case, it implies that the EA community does want some concretely different things, so from the standpoint of the EA community, this would make funding more appealing.
2. I think in the big picture, it seems like OP/GV doesn’t want to be held as responsible for the EA community. Ultimately there’s a conflict here—on one hand, they don’t want to be seen as responsible for the EA community—on the other hand, they might prefer situations where they can have a very large amount of control over the EA community. I hope it can be understood that these two desires can’t easily go together. Perhaps they won’t be willing to compromise on the latter, but also will complain about the former. That might well happen, but I’d hope there could be a better arrangement made.
> OP/GV is usually a pretty responsible funder, so the odds of them suddenly defunding CEA without providing some sort of notice and transitional funding seems low.
I largely agree. That said, if I were CEA, I’d still feel fairly uncomfortable. When the vast majority of your funding comes from any one donor, you’ll need to place a whole lot of trust in them.
I’d imagine that if I were working within CEA, I’d be incredibly precautious not to upset OP or GV. I’d also imagine this to mess with my epistemics/communication/actions.
Also, of course, I’d flag that the world can change quickly. Maybe Trump will go on a push against EA one day, and put OP in an awkward spot, for example.
I agree with the approach’s direction, but this premise doesn’t seem very helpful in shaping the debate. It doesn’t seem that there is a right level of funding for meta EA or that this is what we currently have.
My perception is that OP has specific goals, one of which is to reduce GCR risk. As there are not so many high absorbency funding opportunities and a lot of uncertainty in the field, they focus more on capacity building, of which EA has proven to be a solid investment in talent pipeline building.
If this is true, then the level of funding we are currently seeing is downstream from OP’s overall yearly spending and their goals. Other funders will come to very different conclusions as to why they would want to fund EA meta and to what extent.
If you’re a meta funder who agrees with GCR risks, you might see opportunities that either don’t want OPs’ money, that OP doesn’t want to fund, or that want to keep OPs’ funding under a certain bar. These are more neglected, but they are more cost-effective for you as they are not as fungible.
At the last, MCF funding diversification and the EA brand were the two main topics, but to me, meta-funding diversification seems much harder, especially for areas under the EA brand.
I think you raise some good points on why diversification as I discuss it is difficult and why it hasn’t been done more.
Quickly:
> I agree with the approach’s direction, but this premise doesn’t seem very helpful in shaping the debate.
Sorry, I don’t understand this. What is “the debate” that you are referring to?
> At the last, MCF funding diversification and the EA brand were the two main topics
This is good to know. While mentioning MCF, I would bring up that it seems bad to me that MCF seems to be very much within the OP umbrella, as I understand it. I believe that it was funded by OP or CEA, and the people who set it up were employed by CEA, which was primarily funded by OP. Most of the attendees seem like people at OP or CEA, or else heavily funded by OP.
I have a lot of respect for many of these people and am not claiming anything nefarious. But I do think that this acts as a good example of the sort of thing that seems important for the EA community, and also that OP has an incredibly large amount of control over. It seems like an obvious potential conflict of interest.
I just meant the discussion you wanted to see; I probably used the wrong synonym.
I generally believe that EA is effective at being pragmatic, and in that regard, I think it’s important for the key organizations that are both giving and receiving funding in this area to coordinate, especially with topics like funding diversification. I agree that this is not the ideal world, but this goes back to the main topic.
For reference, I agree it’s important for these people to be meeting with each other. I wasn’t disagreeing with that.
However, I would hope that over time, there would be more people brought in who aren’t in the immediate OP umbrella, to key discussions of the future of EA. At least have like 10% of the audience be strongly/mostly independent or something.
I think its better to start something new. Reform is hard but no one is going to stop you from making a new charity. The EA brand isn’t in the best shape. Imo the “new thing” can take money from individual EAs but shouldn’t accept anything connected to OpenPhil/CEA/Dustin/etc.
If you start new you can start with a better culture.
AIM seems to be doing this quite well in the GHW/AW spaces, but lacks the literal openness of the EA community-as-idea (for better or worse)
I mean Dustin Moskovitz used to come on the forum and beg people to do earn to give yet I don’t think the number of donors has grown that much. More people should go to Jane Street and do Y-Combinator but it feels as though that’s taboo to say for some reason.
I have said this in other spaces since the FTX collapse: The original idea of EA, as I see it, was that it was supposed to make the kind of research work done at philanthropic foundations open and usable for well-to-do-but-not-Bill-Gates-rich Westerners. While it’s inadvisable to outright condemn billionaires using EA work to orient their donations for… obvious reasons, I do think there is a moral hazard in billionaires funding meta EA. Now, the most extreme policy would be to have meta EA be solely funded by membership dues (as plenty organizations are!). I’m not sure if that would really be workable for the amounts of money involved, but some kind of donation cap could be plausibly envisaged.
This part doesn’t resonate with me. I worked at 80k early on (~2014) and have been in the community for a long time. Then, I think the main thing was excitement over “doing good the most effectively”. The assumption was that most philanthropic foundations weren’t doing a good job—not that we wanted regular people to participate, specifically. I think then, most community members would be pretty excited about the idea of the key EA ideas growing as quickly as possible, and billionaires would help with that.
GiveWell specifically was started with a focus on smaller donors, but there was a always a separation between them and EA.
(I am of course more sympathetic to a general skepticism around any billionaire or other overwhelming donor. Though I’m personally also skeptical of most other donation options to other degrees—I want some pragmatic options that can understand the various strengths and weaknesses of different donors and respond accordingly)
… I’m confused by what you would mean by early EA then? As the history of the movement is generally told it started by the merger of three strands: GiveWell (which attempt to make charity effectiveness research available for well-to-do-but-not-Bill-Gates-rich Westerners), GWWC (which attempt to convince well-to-do-but-not-Bill-Gates-rich Westerners to give to charity too), and the rationalists and proto-longtermists (not relevant here).
Criticisms of ineffective charities (stereotypically, the Make a Wish Foundation) could be part of that, but they’re specifically the charities well-to-do-but-not-Bill-Gates-rich Westerners tend to donate to when they do donate, I don’t think people were going out claiming the biggest billionaire philanthropic foundations (like, say, well, the Bill Gates Foundation) didn’t knew what to do with their money.
Quickly:
1. Some of this gets into semantics. There are some things that are more “key inspirations for what was formally called EA” and other things that “were formally called EA, or called themselves EA.” GiveWell was highly influential around EA, but I think it was created before EA was coined, and I don’t think they publicly associated as “EA” for some time (if ever).
2. I think we’re straying from the main topic at this point. One issue is that while I think we disagree on some of the details/semantics of early EA, I also don’t think that matters much for the greater issue at hand. “The specific reason why the EA community technically started” is pretty different from “what people in this scene currently care about.”
It might be helpful to clarify what you mean by “moral hazard” here.
Didn’t really want to in depth go beyond what @Ozzie Gooen already said and mentioning the event that originally prompted that line of thought, but added a link to @David Thorstad’s sequence on the subject.
Someone wrote to me in a PM that they think one good reason for EA donors not to have funded EA community projects was because OP was funding them, and arguably there are other more neglected projects.
I do think this is a big reason, and I was aware of this before. It’s a complex area.
At the same time, I think the current situation is really not the best, and can easily imagine healthier environments where motivated funders and community would have found good arrangements here.
I also take responsibility for not doing a better job around this (and more).
I really don’t like the trend of posts saying that “EA/EAs need to | should do X or Y”.
EA is about cost-benefit analysis. The phrases need and should implies binaries/absolutes and having very high confidence.
I’m sure there are thousands of interventions/measures that would be positive-EV for EA to engage with. I don’t want to see thousands of posts loudly declaring “EA MUST ENACT MEASURE X” and “EAs SHOULD ALL DO THING Y,” in cases where these mostly seem like un-vetted interesting ideas.
In almost all cases I see the phrase, I think it would be much better replaced with things like;
”Doing X would be high-EV”
“X could be very good for EA”
”Y: Cost and Benefits” (With information in the post arguing the benefits are worth it)
”Benefits|Upsides of X” (If you think the upsides are particularly underrepresented)”
I think it’s probably fine to use the word “need” either when it’s paired with an outcome (EA needs to do more outreach to become more popular) or when the issue is fairly clearly existential (the US needs to ensure that nuclear risk is low). It’s also fine to use should in the right context, but it’s not a word to over-use.
See also EA should taboo “EA should”
Related (and classic) post in case others aren’t aware: EA should taboo “EA should”.
Lizka makes a slightly different argument, but a similar conclusion
Strong disagree. If the proponent of an intervention/cause area believes the advancement of it is extremely high EV such that they believe it is would be very imprudent for EA resources not to advance it, they should use strong language.
I think EAs are too eager to hedge their language and use weak language regarding promising ideas.
For example, I have no compunction saying that advancement of the Profit for Good (companies with charities in vast majority shareholder position) needs to be advanced by EA, in that I believe it not doing results in an ocean less counterfactual funding for effective charities, and consequently a significantly worse world.
https://forum.effectivealtruism.org/posts/WMiGwDoqEyswaE6hN/making-trillions-for-effective-charities-through-the
What about social norms, like “EA should encourage people to take care of their mental health even if it means they have less short-term impact”?
Good question.
First, I have a different issue with that phrase, as it’s not clear what “EA” is. To me, EA doesn’t seem like an agent. You can say, ”....CEA should” or ”...OP should”.
Normally, I prefer one says “I think X should”. There are some contexts, specifically small ones (talking to a few people, it’s clearly conversational) where saying, “X should do Y” clearly means “I feel like X should do Y, but I’m not sure”. And there are some contexts where it means “I’m extremely confident X should do Y”.
For example, there’s a big difference between saying “X should do Y” to a small group of friends, when discussing uncertain claims, and writing a mass-market book titled “X should do Y”.
I haven’t noticed this trend, could you list a couple of articles like this? Or even DM me if you’re not comfortable listing them here.
I recently noticed it here:
https://forum.effectivealtruism.org/posts/WJGsb3yyNprAsDNBd/ea-orgs-need-to-tabletop-more
Looking back, it seems like there weren’t many more very recently. Historically, there have been some.
EA needs consultancies
EA needs to understand its “failures” better
EA needs more humor
EA needs Life-Veterans and “Less Smart” people
EA needs outsiders with a greater diversity of skills
EA needs a hiring agency and Nonlinear will fund you to start one
EA needs a cause prioritization journal
Why EA needs to be more conservative
Looking above, many of those seem like “nice to haves”. The word “need” seems over-the-top to me.
There are a couple of strong “shoulds” in the EA Handbook (I went through it over the last two months as part of an EA Virtual program) and they stood out to me as the most disagreeable part of EA philosophy that was presented.
I’ve substantially revised my views on QURI’s research priorities over the past year, primarily driven by the rapid advancement in LLM capabilities.
Previously, our strategy centered on developing highly-structured numeric models with stable APIs, enabling:
Formal forecasting scoring mechanisms
Effective collaboration between human forecasting teams
Reusable parameterized world-models for downstream estimates
However, the progress in LLM capabilities has updated my view. I now believe we should focus on developing and encouraging superior AI reasoning and forecasting systems that can:
Generate high-quality forecasts on-demand, rather than relying on pre-computed forecasts for scoring
Produce context-specific mathematical models as needed, reducing the importance of maintaining generic mathematical frameworks
Leverage repositories of key insights, though likely not in the form of formal probabilistic mathematical models
This represents a pivot from scaling up traditional forecasting systems to exploring how we can enhance AI reasoning capabilities for forecasting tasks. The emphasis is now on dynamic, adaptive systems rather than static, pre-structured models.
(I rewrote with Claude, I think it’s much more understandable now)
A bit more on this part:
To be clear, I think there’s a lot of batch intellectual work we can do before users ask for specific predictions. So “Generating high-quality forecasts on-demand” doesn’t mean “doing all the intellectual work on-demand.”
However, I think there’s a broad set of information that this batch intellectual work could look like. I used to think that this batch work would produce a large set of connect mathematic models. Now I think we probably want something very compressed. If a certain mathematical model can easily be generated on-demand, then there’s not much of a benefit to having it made and saved ahead of time. However, I’m sure there are many crucial insights that are both expensive to find, and would be useful for many questions that LLM users ask about.
So instead of searching for and saving math models, a system might do a bunch of intellectual work and save statements like,
”When estimating the revenue of OpenAI, remember crucial considerations [A] and [B]. Also, a surprisingly good data source for this is Twitter user ai-gnosis-34.”
A lot of user-provided forecasts or replies should basically be the “last mile” or intellectual work. All the key insights are already found, now there just needs to be a bit of customization for the very specific questions someone has.
On the funding-talent balance:
When EA was starting, there was a small amount of talent, and a smaller amount of funding. As one might expect, things went slowly for the first few years.
Then once OP decided to focus on X-risks, there was ~$8B potential funding, but still fairly little talent/capacity. I think the conventional wisdom then was that we were unlikely to be bottlenecked by money anytime soon, and lots of people were encouraged to do direct work.
Then FTX Future Fund came in, and the situation got even more out-of-control. ~Twice the funding. Projects got more ambitious, but it was clear there were significant capacity (funder and organization) constraints.
Then (1) FTX crashed, and (2) lots of smart people came into the system. Project capacity grew, AI advances freaked out a lot of people, and successful community projects helped train a lot of smart young people to work on X-risks.
But funding has not kept up. OP has been slow to hire for many x-risk roles (AI safety, movement building, outreach / fundraising). Other large funders have been slow to join in.
So now there’s a crunch for funding. There are a bunch of smart-seeming AI people now who I bet could have gotten funding during the FFF, likely even before then with OP, but are under the bar now.
I imagine that this situation will eventually improve, but of course, it would be incredibly nice if it could happen sooner. It seems like EA leadership eventually fix things, but it often happens slower than is ideal, with a lot of opportunity loss in that time.
Opportunistic people can fill in the gaps. Looking back, I think more money and leadership in the early days would have gone far. Then, more organizational/development capacity during the FFF era. Now, more funding seems unusually valuable.
If you’ve been thinking about donating to the longtermist space, specifically around AI safety, I think it’s likely that funding this year will be more useful than funding in the next 1-3 years. (Of course, I’d recommend using strong advisors or giving to funds, instead of just choosing directly, unless you can spend a fair bit of time analyzing things).
If you’re considering entering the field as a nonprofit employee, heed some caution. I still think the space can use great talent, but note that this is an unusually competitive time to get many paid roles or to get nonprofit grants.
Any thoughts on where e.g. 50K could be well spent?
(For longtermism)
If you have limited time to investigate / work with, I’d probably recommend either the LTFF or choosing a regranter you like at Manifund.
If you have a fair bit more time, and ideally the expectation of more money in the future, then I think a lot of small-to-medium (1-10 employee) organizations can use some long-term, high-touch donors. Honestly this may settle more down to fit / relationships than identifying the absolute best org—as long as it’s funded by one of the groups listed above or OP, as money itself is a bit fungible between orgs.
I think a lot of nonprofits have surprisingly few independent donors, or even strong people that can provide decent independent takes. I might write more about this later.
(That said, there are definitely ways to be annoying / a hindrance, as an active donor, so try to be really humble here if you are new to this)
EA seems to have been doing a pretty great job attracting top talent from the most prestigious universities. While we attract a minority of the total pool, I imagine we get some of the most altruistic+rational+agentic individuals.
If this continues, it could be worth noting that this could have significant repercussions for areas outside of EA; the ones that we may divert them from. We may be diverting a significant fraction of the future “best and brightest” in non-EA fields.
If this seems possible, it’s especially important that we do a really, really good job making sure that we are giving them good advice.
One of my main frustrations/criticisms with a lot of current technical AI safety work is that I’m not convinced it will generalize to the critical issues we’ll have at our first AI catastrophes ($1T+ damage).
From what I can tell, most technical AI safety work is focused on studying previous and current LLMs. Much of this work is very particular to specific problems and limitations these LLMs have.
I’m worried that the future decisive systems won’t look like “single LLMs, similar to 2024 LLMs.” Partly, I think it’s very likely that these systems will be ones made up of combinations of many LLMs and other software. If you have a clever multi-level system, you get a lot of opportunities to fix problems of the specific parts. For example, you can have control systems monitoring LLMs that you don’t trust, and you can use redundancy and checking to investigate outputs you’re just not sure about. (This isn’t to say that these composite systems won’t have problems—just that the problems will look different to those of the specific LLMs).
Here’s an analogy: Imagine that researchers had 1960s transistors but not computers, and tried to work on cybersecurity, in preparation of future cyber-disasters in the coming decades. They want to be “empirical” about it, so they go along investigating all the failure modes of 1960s transistors. They successfully demonstrate that in extreme environments transistors fail, and also that there are some physical attacks that could be done on the transistor level.
But as we know now, almost all of this has either been solved on the transistor level, or on levels shortly above the transistors that do simple error management. Intentional attacks on the transistor level are possible, but incredibly niche compared to all of the other cybersecurity capabilities.
So just as understanding 1960s transistors really would not get you far towards helping at all with future cybersecurity challenges, it’s possible that understanding 2024 LLM details won’t help with future 2030 composite AI system disasters.
(John Wentworth and others refer to much of this as the Streetlight effect. I think that specific post is too harsh, but I think I sympathize with the main frustration.)
All that said, here are some reasons to still do the LLM research anyway. Some don’t feel great, but might still make it worthwhile.
There’s arguably not much else we can do now.
While we’re waiting to know how things will shape up, this is the most accessible technical part we can work on.
Having a research base skilled with empirical work on existing LLMs will be useful later on, as we could re-focus it to whatever comes about in the future.
There’s some decent chance that future AI disasters will come from systems that look a lot like modern “LLM-only” systems. Perhaps these disasters will happen in the next few years, or perhaps AI development will follow a very specific path.
This research builds skills that are generally useful later—either to work in AI companies to help them do things safely, or to make a lot of money.
It’s good to have empirical work, because it will raise the respect/profile of this sort of thinking within the ML community.
I’m not saying I could do better. This is one reason why I’m not exactly working in on technical AI safety. I have been interested in strategy in the area (which feels more tractable to me), and have been trying to eye opportunities for technical work, but am still fairly unsure of what’s best at this point.
I think the main challenge is that it’s just fundamentally hard to prepare for a one-time event with few warning shots (i.e. the main situation we’re worried about), several years in the future, in a fast-moving technical space. This felt clearly true 10 years ago, before there were language models that seemed close to TAI. I feel like it’s become easier since to overlook this bottleneck, as there’s clearly a lot of work we can do with LLMs that naively seems interesting. But that doesn’t mean it’s no longer true—it might still very much be the case that things are so early that useful safety empirical technical work is very difficult to do.
(Note: I have timelines for TAI that are 5+ years out. If your timelines are shorter, it would make more sense that understanding current LLMs would help.)
A large reason to focus on opaque components of larger systems is that difficult-to-handle and existentially risky misalignment concerns are most likely to occur within opaque components rather than emerge from human built software.
I don’t see any plausible x-risk threat models that emerge directly from AI software written by humans? (I can see some threat models due to AIs building other AIs by hand such that the resulting system is extremely opaque and might takeover.)
In the comment you say “LLMs”, but I’d note that a substantial fraction of this research probably generalizes fine to arbitrary DNNs trained with something like SGD. More generally, various approaches that work for DNNs trained with SGD plausibly generalize to other machine learning approaches.
Yep, this sounds positive to me. I imagine it’s difficult to do this well, but to the extent it can be done, I expect such work to generalize more than a lot of LLM-specific work.
I don’t feel like that’s my disagreement. I’m expecting humans to create either [dangerous system that’s basically one black-box LLM] or [something very different that’s also dangerous, like a complex composite system]. I expect AIs can also make either system.
Also posted here, where it got some good comments: https://www.facebook.com/ozzie.gooen/posts/pfbid037YTCErx7T7BZrkYHDQvfmV3bBAL1mFzUMBv1hstzky8dkGpr17CVYpBVsAyQwvSkl
Some musicians have multiple alter-egos that they use to communicate information from different perspectives. MF Doom released albums under several alter-egos; he even used these aliases to criticize his previous aliases.
Some musicians, like Madonna, just continued to “re-invent” themselves every few years.
Youtube personalities often feature themselves dressed as different personalities to represent different viewpoints.
It’s really difficult to keep a single understood identity, while also conveying different kinds of information.
Narrow identities are important for a lot of reasons. I think the main one is predictability, similar to a company brand. If your identity seems to dramatically change hour to hour, people wouldn’t be able to predict your behavior, so fewer could interact or engage with you in ways they’d feel comfortable with.
However, narrow identities can also be suffocating. They restrict what you can say and how people will interpret that. You can simply say more things in more ways if you can change identities. So having multiple identities can be a really useful tool.
Sadly, most academics and intellectuals can only really have one public identity.
---
EA researchers currently act this way.
In EA, it’s generally really important to be seen as calibrated and reasonable, so people correspondingly prioritize that in their public (and then private) identities. I’ve done this. But it comes with a cost.
One obvious (though unorthodox) way around this is to allow researchers to post content either under aliases. It could be fine if the identity of the author is known, as long as readers can keep these aliases distinct.
I’ve been considering how to best do this myself. My regular EA Forum name is just “Ozzie Gooen”. Possible aliases would likely be adjustments to this name.
- “Angry Ozzie Gooen” (or “Disagreeable Ozzie Gooen”)
- “Tech Bro Ozzie Gooen”
- “Utility-bot 352d3”
These would be used to communicate in very different styles, with me attempting what I’d expect readers to expect of those styles.
(Normally this is done to represent viewpoints other than what they have, but sometimes it’s to represent viewpoints they have, but wouldn’t normally share)
Facebook Discussion
I can’t seem to find much EA discussion about [genetic modification to chickens to lessen suffering]. I think this naively seems like a promising area to me. I imagine others have investigated and decided against further work, I’m curious why.
Lewis Bollard:
“I agree with Ellen that legislation / corporate standards are more promising. I’ve asked if the breeders would accept $ to select on welfare, & the answer was no b/c it’s inversely correlated w/ productivity & they can only select on ~2 traits/generation.”
Dang. That makes sense, but it seems pretty grim. The second half of that argument is, “We can’t select for not-feeling-pain, because we need to spend all of our future genetic modification points on the chickens getting bigger and growing even faster.”
I’m kind of surprised that this argument isn’t at all about the weirdness of it. It’s purely pragmatic, from their standpoint. “Sure, we might be able to stop most of the chicken suffering, but that would increase costs by ~20% or so, so it’s a non-issue”
20% of the global cost of growing chickens is probably in the order of at least ~$20B, which is much more than the global economy is willing to spend on animal welfare.
As mentioned in the other comment, I think it’s extremely unlikely that there is a way to stop “most” of the chicken suffering while increasing costs by only ~20%.
Some estimate the better chicken commitment already increases costs by 20% (although there is no consensus on that, and factory farmers estimate 37.5%), and my understanding is that it doesn’t stop most of the suffering, but “just” reduces it a lot.
Has there been any discussion of improving chicken breeding using GWAS or similar?
Even if welfare is inversely correlated with productivity, I imagine there are at least a few gene variants which improve welfare without hurting productivity. E.g. gene variants which address health issues due to selective breeding.
Also how about legislation targeting the breeders? Can we have a law like: “Chickens cannot be bred for increased productivity unless they meet some welfare standard.”
England prohibits “breeding procedures which cause, or are likely to cause, suffering or injury to any of the animals concerned”. Defra claim Frankenchickens meet this standard and THLUK are challenging that decision in court.
Note that prohibiting breeding that causes suffering is different to encouraging breeding that lessens suffering, and that selective breeding is different to gene splicing, etc., which I think is what is typically meant by genetic modification.
I think it is discussed every now and then, see e.g. comments here: New EA cause area: Breeding really dumb chickens and this comment
And note that the Better Chicken Commitment includes a policy of moving to higher welfare breeds.
Naively, I would expect that suffering is extremely evolutionarily advantageous for chickens in factory farm conditions, so chickens that feel less suffering will not grow as much meat (or require more space/resources). For example, based on my impression that broiler chickens are constantly hungry, I wouldn’t be surprised if they would try to eat themselves unless they felt pain when doing so. But this is a very uninformed take based on a vague understanding of what broiler chickens are optimized for, which might not be true in practice.
I think this idea might be more interesting to explore in less price-sensitive contexts, where there’s less evolutionary pressure and animals live in much better conditions, mostly animals used in scientific research. But of course it would help much fewer animals who usually suffer much less.
adding on that wholefoods https://www.wholefoodsmarket.com/quality-standards/statement-on-broiler-chicken-welfare
has made some commitments to switching breeds, we discussed this briefly at a Chicago EA meeting. I didn’t get much info but they said that going and protesting/spreading the word to whole foods managers to switch breeds showed some success.
It was mentioned at the Constellation office that maybe animal welfare people who are predisposed to this kind of weird intervention are working on AI safety instead. I think this is >10% correct but a bit cynical; the WAW people are clearly not afraid of ideas like giving rodents contraceptives and vaccines. My guess is animal welfare is poorly understood and there are various practical problems like preventing animals that don’t feel pain from accidentally injuring themselves constantly. Not that this means we shouldn’t be trying.
I heard someone from Kevin Esvelt’s lab talking about this + pain-free lab mice once
Quick thought. Maybe people anticipate this being blocked by governments because it “seems like playing god” etc. I know that would be hypocritical given the breeding already used to make them overweight etc. But it seems to be the way a lot of people see this.
By coincidence, I just came across this layer-hen genetics project that got funding from OP. I don’t know much about the work or how promising it might be.
I’ve been trying to process the conversation in this thread:
https://forum.effectivealtruism.org/posts/mopsmd3JELJRyTTty/ozzie-gooen-s-shortform?commentId=o9rEBRmKoTvjNMHF7
One thing that comes to mind is that this seems like a topic a lot of people care about, and there’s a lot of upvotes & agreements, but there also seemed to be a surprising lack of comments, overall.
I’ve heard from others elsewhere that they were nervous about giving their takes, because it’s a sensitive topic.
Obviously I’m really curious about what, if anything, could be done, to facilitate more discussion on these sorts of issues. I think they’re important to grapple with, and would hate it if they just weren’t talked about due to awkwardness.
One obvious failure mode is that I think these discussions can get heated and occasionally result in really bad conversation. But having incredibly little discussion seems almost more unnerving to me.
I imagine we want environments where:
- People feel safe posing what they believe
- Conversation doesn’t spiral out-of-control
As some of you may know, I post a fair bit on Facebook, and did make some posts there about some of these topics. Many of these are private to my network, and some get a fair bit of conversation that’s not on the EA Forum—often it’s clear that many people find this more comfortable. But obviously this particular setup doesn’t scale very well.
Hey Ozzie, a few quick notes on why I react but try not to comment on community based stuff these days:
I try to limit how many meta-level comments I make. In general I’d like to see more object-level discussion of things and so I’m trying (to mixed success) to comment mostly about cause areas directly.
Partly it’s a vote for the person I’d like to be. If I talk about community stuff, part of my headspace will be thinking about it for the next few days. (I fully realize the irony of making this comment.)
It’s emotionally tricky since I feel responsibility for how others react. I know how loaded this topic was for a younger me, and I feel an obligation to make younger me feel welcome
These conversations often feel aspirational and repetitive. Like “there should be more X” is too simple. Whereas something like “there should be more X. Y org should be responsible for it. Tradeoffs may be Z. Failure modes are A, B, and C.” is concrete enough to get somewhere.
I think I broadly like the idea of Donation Week.
One potential weakness is that I’m curious if it promotes the more well-known charities due to the voting system. I’d assume that these are somewhat inversely correlated with the most neglected charities.
Related, I’m curious if future versions could feature specific subprojects/teams within charities. “Rethink Priorities” is a rather large project compared to “PauseAI US”, I assume it would be interesting if different parts of it were put here instead.
(That said, in terms of the donation, I’d hope that we could donate to RP as a whole and trust RP to allocate it accordingly, instead of formally restricting the money, which can be quite a hassle in terms of accounting)
I guess this isn’t necessarily a weakness if the more well-known charities are more effective? I can see the case that: a) they might not be neglected in EA circles, but may be very neglected globally compared to their impact and that b) there is often an inverse relationship between tractability/neglectedness and importance/impact of a cause area and charity. Not saying you’re wrong, but it’s not necessarily a problem.
Furthermore, my anecdotal take from the voting patterns as well as the comments on the discussion thread seem to indicate that neglectedness is often high on the mind of voters—though I admit that commenters on that thread are a biased sample of all those voting in the election.
Is it underwhelming? I guess if you want the donation election to be about spurring lots of donations to small, spunky EA-startups working in weird-er cause areas, it might be, but I don’t think that’s what I understand the intention of the experiment to be (though I could be wrong).
My take is that the election is an experiment with EA democratisation, where we get to see what the community values when we do a roughly 1-person-1-ballot system instead of those-with-the-moeny decide system which is how things work right now. Those takeaways seem to be:
The broad EA community values Animal Welfare a lot more than the current major funders
The broad EA community sees value in all 3 of the ‘big cause areas’ with high-scoring charities in Animal Welfare, AI Safety, and Global Health & Development.
I (with limited information) think the EA Animal Welfare Fund is promising, but wonder how much of that matches the intention of this experiment. It can be a bit underwhelming if an experiment to try to get the crowd’s takes on charities winds up determining to, “just let the current few experts figure it out.” Though I guess, that does represent a good state of the world. (The public thinks that the current experts are basically right)
When I hear of entrepreneurs excited about prediction infrastructure making businesses, I feel like they gravitate towards new prediction markets or making new hedge funds.
I really wish it were easier to make new insurance businesses (or similar products). I think innovative insurance products could be a huge boon to global welfare. The very unfortunate downside is that there’s just a ton of regulation and lots of marketing to do, even in cases where it’s a clear win for consumers.
Ideally, it should be very easy and common to get insurance for all of the key insecurities of your life.
Having children with severe disabilities / issues
Having business or romantic partners defect on you
Having your dispreferred candidate get elected
Increases in political / environmental instability
Some unexpected catastrophe will hit a business
Nonprofits losing their top donor due to some unexpected issue with said donor (i.e. FTX)
I think a lot of people have certain issues that both:
They worry about a lot
They over-weight the risks of these issues
In these cases, insurance could be a big win!
In a better world, almost all global risks would be held primarily by asset managers / insurance agencies. Individuals could have highly predictable lifestyles.
(Of course, some prediction markets and other markets can occasionally be used for this purpose as well!)
Some of these things are fundamentally hard to insure against, because of information asymmetries / moral hazard.
e.g. insurance against donor issues would disproportionately be taken by people who had some suspicions about their donors, which would drive up prices, which would get more people without suspicions to decline taking insurance, until the market was pretty tiny with very high prices and a high claim rate. (It would also increase the incentives to commit fraud to give, which seems bad.)
Some of these harms seem of a sort that does not really feel compensable with money. While romantic partner’s defection might create some out-of-pocket costs, but I don’t think the knowledge that I’d get some money out of my wife defecting would make me feel any better about the possibility!
Also, I’d note that some of the harms are already covered by social insurance schemes to a large extent. For instance, although parents certainly face a lot of costs associated with “[h]aving children with severe disabilities / issues,” a high percentage of costs in the highest-cost scenarios are already borne by the public (e.g., Medicaid, Social Security/SSI, the special education system, etc.) or by existing insurers (e.g., employer-provided health insurance). So I’d want to think more about the relative merits of novel private-sector insurance schemes versus strengthening the socialized schemes.
Consider this, as examples of where it might be important:
1. You are financially dependent on your spouse. If they cheated on you, you would likely want to leave them, but you wouldn’t want to be trapped due to finances.
2. You’re nervous about the potential expenses of a divorce.
I think that this situation is probably a poor fit for insurance at this point, just because of moral risks that would happen, but perhaps one day it might be viable to some extent.
> So I’d want to think more about the relative merits of novel private-sector insurance schemes versus strengthening the socialized schemes.
I’m all for improvements on socialized schemes too. No reason not for both strategies to be tested and used. In theory, insurance could be much easier and faster to be implemented. It can take ages for nation-wide reform to happen.
A few junior/summer effective altruism related research fellowships are ending, and I’m getting to see some of the research pitches.
Lots of confident-looking pictures of people with fancy and impressive sounding projects.
I want to flag that many of the most senior people I know around longtermism are really confused about stuff. And I’m personally often pretty skeptical of those who don’t seem confused.
So I think a good proposal isn’t something like, “What should the EU do about X-risks?” It’s much more like, “A light summary of what a few people so far think about this, and a few considerations that they haven’t yet flagged, but note that I’m really unsure about all of this.”
Many of these problems seem way harder than we’d like for them to be, and much harder than many seem to assume at first. (perhaps this is due to unreasonable demands for rigor, but an alternative here would be itself a research effort).
I imagine a lot of researchers assume they won’t stand out unless they seem to make bold claims. I think this isn’t true for many EA key orgs, though it might be the case that it’s good for some other programs (University roles, perhaps?).
Not sure how to finish this post here. I think part of me wants to encourage junior researchers to lean on humility, but at the same time, I don’t want to shame those who don’t feel like they can do so for reasons of not-being-homeless (or simply having to leave research). I think the easier thing is to slowly spread common knowledge and encourage a culture where proper calibration is just naturally incentivized.
Facebook Thread
Relevant post by Nuño: https://forum.effectivealtruism.org/posts/7utb4Fc9aPvM6SAEo/frank-feedback-given-to-very-junior-researchers?fbclid=IwAR1M0zumAQ452iOAOVKGEcOdI4MwORfVSX4H1S2zLhyUXrWjarvUt31mKsg
Do people know what’s going on with the EA Funds financial position? Some of the graphs look a bit worrying. But I’ve also noticed some (what seem like) inconsistencies, so I suspect that some key data is missing or something.
https://funds.effectivealtruism.org/stats/donations
The following graph seems incorrect. It seems like it just wasn’t updated since Dec 2023. And I’m curious if there really was the ~$2M jump in the LTFF that month.
This has been on the stack to look into for a few weeks. I think we might just take the graphs down until we’re confident they are all accurate. They were broken (and then fixed) after we moved accounting systems, but it looks like they might have broken again.
Thanks for the update!
If it happens to be the case that the numbers of active donors are continuing to go down, I’d of course be quite curious for more discussion in why that is. I think a survey could be useful here, or maybe just a EA Forum Question. Maybe even something to just all potential donors, like, “What are your main reservations?”
The loss of active donors seems like a big deal to me. I’m really curious what’s happening there.
1. Some donors probably just changed targets, for example, to Manifund or other new funds
2. I’m sure some got cynical or disenchanted by the movement or something
3. Maybe some got hurt financially, from FTX / a crypto bust?
Someone pointed me to this blog post by the EA Infrastructure Fund, which I missed.
https://forum.effectivealtruism.org/posts/PN4FEGt3fjHv8diGo/eaif-isn-t-currently-funding-constrained
> EAIF currently has $3.3M in available funds. So far over 2024 EAIF has made grants worth $1.1M, and I expect this to be around $1.4M by the end of 2024.
So it seems like EAIF at least is in a decent position.
I’m not sure how this coincides with the above numbers though. Just eyeballing it—maybe, while donations in have been low, donations out have been lower, so there’s been a surplus of funding.
Quick idea -
I think that “signs of empathy/appreciation/politeness” get undervalued online. Obviously, LLMs can be good at this.
I sort of like the idea of things like:
We rank public commenters for how polite / appreciative they are. We show the top ~10, maybe give weekly rewards. Ranking could be done automatically.
We formally estimate the value/disvalue that comments provide in terms of [encouragement/discouragement], and reveal/estimate this
I’d like to see my metrics improving over time. Like, to show that this year, my comments have made people feel more positive than they did last year.
It seems easy enough for commenters to learn to spend more effort here. They could work with LLMs to do better. Arguably, there could be some fairly-cheap wins here.
There’s obviously the potential problem of encouraging niceness that’s not genuine, then devaluing all of it. But I think this could also be done well, and such a thing might lead to a more positive community.
I find the disagree votes pretty interesting on this, a bit curious to better understand the intuitions there.
I think for many people, positive comments would be much less meaningful if they were rewarded/quantified, because you would doubt that they’re genuine. (Especially if you excessively feel like an imposter and easily seize onto reasons to dismiss praise.)
I disagree with your recommendations despite agreeing that positive comments are undersupplied.
I’d quickly flag:
1. Any decent intervention should be done experimentally. It’s not like there would be “one system, hastily put-together, in place forever.” More like, early work would try out some things and see what the response is like in practice. I imagine that many original ideas would be mediocre, but with the right modifications and adjustments to feedback, it’s possible to make something decent.
2. I think that positive comments are often already rewarded—and that’s a major reason people give them. But I don’t think this is necessarily a bad thing. My quick guess is that this is a situation of adjusting incentives—certain incentive structures would encourage certain classes of good and bad behaviors, so it’s important to continue to tune these. Right now we have some basic incentives that were arrived at by default, and in my opinion are quite unsophisticated (people are incentivized to be extra nice to people who are powerful and who will respond, and mean to people in the outgroup). I think semi-intentional work can improve this, but I realize it would need to be done well.
On my side it feels a bit like,
”We currently have an ecosystem of very mediocre incentives, that produce the current results. It’s possible to set up infrastructure to adjust those incentives and experiment with what those results would be. I’m optimistic that this problem is both important enough and tractable enough for some good efforts to work on.”
I upvoted and didn’t disagree-vote, because I generally agree that using AI to nudge online discourse in more productive directions seems good. But if I had to guess where disagree votes come from, it might be a combination of:
It seems like we probably want politeness-satisficing rather than politeness-maximizing. (This could be consistent with some versions of the mechanism you describe, or a very slightly tweaked version).
There’s a fine line between politiness-moderating and moderating the substance of ideas that make people uncomfortable. Historically, it has been hard to police this line, and given the empirically observable political preferences of LLMs, it’s reasonable for people who don’t share those preferences to worry that this will disadvantage them (though I expect this bias issue to get better over time, possibly very soon)
There is a time and place for spirited moral discourse that is not “polite,” because the targets of the discourse are engaging in highly morally objectionable action, and it would be bad to always discourage people from engaging in such discourse.*
*This is a complicated topic that I don’t claim to have either (a) fully coherent views on, or (b) have always lived up to the views I do endorse.
As AI improves, there’s a window for people to get involved and make changes regarding AI alignment and policy.
The window arguably starts small, then widens as it becomes clearer what to do.
But at some point it gets too close to TAI, I expect that the window narrows. The key decisions get made by a smaller and smaller group of people, and these people have less ability get help from others, given the quickening pace of things.
For example, at T minus 1 month, there might ultimately be a group of 10 people with key decision-making authority on the most powerful and dangerous AI project. The ‘room where it happens’ has become quite small.
This is somewhat similar to tech projects. An ambitious initiative will start with a few people, then slowly expand to hundreds. But over time decisions get locked into place. Eventually the project goes into “bug fixing” stage, then a marketing/release phase, after which the researchers will often get re-allocated. Later execs can decide to make decisions like killing the project.
One thing this means is that I expect that there could be a decent amount of time where many of us have “basically nothing to do” about AI safety, even though TAI still hasn’t happened. I imagine it could still be good for many people to try to grow capital and influence other people in order to create positive epistemics/lock-in, but the key AI safety issues belong to a narrow group.
If it is the case that TAI will happen in 2 years, for example, I imagine very few people will be able to do much at all at this point, for the key aspects of AI alignment, especially if you’re not actively working in the field.
Obviously, roles working on legislation with at 5+ time horizon will stop being relevant relevant over 5 years before TAI. And people working in tech at non-leading labs might not be relevant once it’s clear these are non-leading labs.
(I don’t mean to discourage people. Rather, I think it’s important to realize when one should strive hard, and when one should chill out a bit and focus on other issues. Personally I’m sort of looking forward to the time where I’m extremely confident that I can’t contribute much to the most major things. It’s basically the part of the project where it’s ‘in someone else’s hands’.)
Hmm maybe it could still be good to try things in case timelines are a bit longer or an unexpected opportunity arises? For example, what if you thought it was 2 years but actually 3-5?
I wasn’t trying to make the argument that it would definitely be clear when this window closes. I’m very unsure of this. I also expect that different people have different beliefs, and that it makes sense for them to then take corresponding actions.
Could/should altruistic activist investors buy lots of Twitter stock, then pressure them to do altruistic things?
---
So, Jack Dorsey just resigned from Twitter.
Some people on Hacker News are pointing out that Twitter has had recent issues with activist investors, and that this move might make those investors happy.
https://pxlnv.com/linklog/twitter-fleets-elliott-management/
From a quick look… Twitter stock really hasn’t been doing very well. It’s almost back at its price in 2014.
Square, Jack Dorsey’s other company (he was CEO of two), has done much better. Market cap of over 2x Twitter ($100B), huge gains in the last 4 years.
I’m imagining that if I were Jack… leaving would have been really tempting. On one hand, I’d have Twitter, which isn’t really improving, is facing activist investor attacks, and worst, apparently is responsible for global chaos (of which I barely know how to stop). And on the other hand, there’s this really tame payments company with little controversy.
Being CEO of Twitter seems like one of the most thankless big-tech CEO positions around.
That sucks, because it would be really valuable if some great CEO could improve Twitter, for the sake of humanity.
One small silver lining is that the valuation of Twitter is relatively small. It has a market cap of $38B. In comparison, Facebook/Meta is $945B and Netflix is $294B.
So if altruistic interests really wanted to… I imagine they could become activist investors, but like, in a good way? I would naively expect that even with just 30% of the company you could push them to do positive things. $12B to improve global epistemics in a major way.
The US could have even bought Twitter for 4% of the recent $1T infrastructure bill. (though it’s probably better that more altruistic ventures do it).
If middle-class intellectuals really wanted it enough, theoretically they could crowdsource the cash.
I think intuitively, this seems like clearly a tempting deal.
I’d be curious if this would be a crazy proposition, or if this is just not happening due to coordination failures.
Admittingly, it might seem pretty weird to use charitable/foundation dollars on “Buying lots of Twitter” instead of direct aid, but the path to impact is pretty clear.
Facebook Thread
One futarchy/prediction market/coordination idea I have is to find some local governments and see if we could help them out by incorporating some of the relevant techniques.
This could be neat if it could be done as a side project. Right now effective altruists/rationalists don’t actually have many great examples of side projects, and historically, “the spare time of particularly enthusiastic members of a jurisdiction” has been a major factor in improving governments.
Berkeley and London seem like natural choices given the communities there. I imagine it could even be better if there were some government somewhere in the world that was just unusually amenable to both innovative techniques, and to external help with them.
Given that EAs/rationalists care so much about global coordination, getting concrete experience improving government systems could be interesting practice.
There’s so much theoretical discussion of coordination and government mistakes on LessWrong, but very little discussion of practical experience implementing these ideas into action.
(This clearly falls into the Institutional Decision Making camp)
Facebook Thread
On AGI (Artificial General Intelligence):
I have a bunch of friends/colleagues who are either trying to slow AGI down (by stopping arms races) or align it before it’s made (and would much prefer it be slowed down).
Then I have several friends who are actively working to *speed up* AGI development. (Normally just regular AI, but often specifically AGI)[1]
Then there are several people who are apparently trying to align AGI, but who are also effectively speeding it up, but they claim that the trade-off is probably worth it (to highly varying degrees of plausibility, in my rough opinion).
In general, people seem surprisingly chill about this mixture? My impression is that people are highly incentivized to not upset people, and this has led to this strange situation where people are clearly pushing in opposite directions on arguably the most crucial problem today, but it’s all really nonchalant.
[1] To be clear, I don’t think I have any EA friends in this bucket. But some are clearly EA-adjacent.
More discussion here: https://www.facebook.com/ozzie.gooen/posts/10165732991305363
Thinking about the idea of an “Evaluation Consent Policy” for charitable projects.
For example, for a certain charitable project I produce, I’d explicitly consent to allow anyone online, including friends and enemies, to candidly review it to their heart’s content. They’re free to use methods like LLMs to do this.
Such a policy can give limited consent. For example:
You can’t break laws when doing this evaluation
You can’t lie/cheat/steal to get information for this evaluation
Consent is only provided for under 3 years
Consent is only provided starting in 5 years
Consent is “contagious” or has a “share-alike provision”. Any writing that takes advantage of this policy, must itself have a consent policy that’s at least as permissive. If someone writes a really bad evaluation, they agree that you and others are correspondingly allowed to critique this evaluation.
The content must score less than 6⁄10 when run against Claude on a prompt roughly asking, “Is this piece written in a way that’s unnecessarily inflammatory?”
Consent can be limited to a certain group of people. Perhaps you reject certain inflammatory journalists, for example. (Though these might be the people least likely to care about getting your permission anyway)
This would work a lot like Creative Commons or Software Licenses. However, it would cover different territory, and (at this point at least) won’t be based on legal enforcement.
Criticisms:
“Why do we need this? People are already allowed to critique anything they want.”
While this is technically true, I think it would frequently break social norms. There are a lot of cases where people would get upset if their projects were provided any negative critique, even if it came with positive points. This would act as a signal that the owners might be particularly okay with critique. I think we live in a society that’s far from maximum-candidness, and it’s often difficult to tell where candidness would be accepted—so explicit communication could be useful.
“But couldn’t people who sign such a policy just attack evaluators anyway?”
I don’t think an explicit policy here will be a silver bullet, but I think it would help. I expect that a boss known for being cruel wouldn’t be trusted if they provided such a policy, but I imagine many other groups would be. Ideally there could be some common knowledge about which people/organizations fail to properly honor their policies. I don’t think this would work for Open Philanthropy that much (in the sense that effective altruists might expect OP to not complain publicly, but later not fund the writer’s future projects), but it could for many smaller orgs (that would have much less secretive power over public evaluators/writers)
---
Anyway, I’m interested in thoughts by this community.
There seem to be several longtermist academics who plan to spend the next few years (at least) investigating the psychology of getting the public to care about existential risks.
This is nice, but I feel like what we really could use are marketers, not academics. Those are the people companies use for this sort of work. It’s somewhat unusual that marketing isn’t much of a respected academic field, but it’s definitely a highly respected organizational one.
There are at least a few people in the community with marketing experience and an expressed desire to help out. The most recent example that comes to mind is this post.
If anyone reading this comment knows people who are interested in the intersection of longtermism and marketing, consider telling them about EA Funds! I can imagine the LTFF or EAIF being very interested in projects like this.
(That said, maybe one of the longtermist foundations should consider hiring a marketing consultant?)
Yep, agreed. Right now I think there are very few people doing active work in longtermism (outside of a few orgs that have people for that org), but this seems very valuable to improve upon.
If you’re happy to share, who are the longtermist academics you are thinking of? (Their work could be somewhat related to my work)
No prominent ones come to mind. There are some very junior folks I’ve recently seen discussing this, but I feel uncomfortable calling them out.
Around discussions of AI & Forecasting, there seems to be some assumption like:
1. Right now, humans are better than AIs at judgemental forecasting.
2. When humans are better than AIs at forecasting, AIs are useless.
3. At some point, AIs will be better than humans at forecasting.
4. At that point, when it comes to forecasting, humans will be useless.
This comes from a lot of discussion and some research comparing “humans” to “AIs” in forecasting tournaments.
As you might expect, I think this model is incredibly naive. To me, it’s asking questions like,
”Are AIs better than humans at writing code?”
“Are AIs better than humans at trading stocks?”
”Are AIs better than humans at doing operations work?”
I think it should be very clear that there’s a huge period, in each cluster, where it makes sense for humans and AIs to overlap. “Forecasting” is not one homogeneous and singular activity, and neither is programming, stock trading, or doing ops. There’s no clear line for automating “forecasting”—there are instead a very long list of different skills one could automate, with a long tail of tasks that would get increasingly expensive to automate.
Autonomous driving is another similar example. There’s a very long road between “helping drivers with driver-assist features” and “complete level-5 automation, to the extent that almost no human are no longer driving for work purposes.”
A much better model is a more nuanced one. Break things down into smaller chunks, and figure out where and how AIs could best augment or replace humans at each of those. Or just spend a lot of time working with human forecasting teams to augment parts of their workflows.
I am not so aware of the assumption you make up front, and would agree with you that anyone making such an assumption is being naive. Not least because humans on average (and even supers under many conditions) are objectively inaccurate at forecasting—even if relatively good given we don’t have anything better yet.
I think the more interesting and important when it comes to AI forecasting and claiming they are “good”, is to look at the reasoning process that they undertaken to do that. How are they forming reference classes, how are they integrating specific information, how are they updating their posterior to form an accurate inference and likelihood of the event occurring? Right now, they can sort of do (1), but from my experience don’t do well at all at integration, updating, and making a probabilistic judgment. In fairness, humans often don’t either. But we do it more consistently than current AI.
For your post, this suggests to me that AI could be used to help base rate/reference class creation, and maybe loosely support integration.
I think that the phrase [“unaligned” AI] is too vague for a lot of safety research work.
I prefer keywords like:
- scheming—
naive
- deceptive
- overconfident
- uncooperative
I’m happy that the phrase “scheming” seems to have become popular recently, that’s an issue that seems fairly specific to me. I have a much easier time imagining preventing an AI from successfully (intentionally) scheming than I do preventing it from being “unaligned.”
Hmm, I would argue than an AI which, when asked, causes human extinction is not aligned, even if it did exactly what it was told.
Yea, I think I’d classify that as a different thing. I see alignment typically as a “mistake” issue, rather than as a “misuse” issue. I think others here often use the phrase similarly.
When discussing forecasting systems, sometimes I get asked,
The obvious answer is,
Or,
For example,
We make a list of 10,000 potential government forecasting projects.
For each, we will have a later evaluation for “how valuable/successful was this project?”.
We then open forecasting questions for each potential project. Like, “If we were to run forecasting project #8374, how successful would it be?”
We take the top results and enact them.
Stated differently,
Forecasting is part of general-purpose collective reasoning.
Prioritization of forecasting requires collective reasoning.
So, forecasting can be used to prioritize forecasting.
I think a lot of people find this meta and counterintuitive at first, but it seems pretty obvious to me.
All that said, I can’t be sure things will play out like this. In practice, the “best thing to use forecasting on” might be obvious enough such that we don’t need to do costly prioritization work first. For example, the community isn’t currently doing much of this meta stuff around Metaculus. I think this is a bit mistaken, but not incredibly so.
Facebook Thread
I’m sort of hoping that 15 years from now, a whole lot of common debates quickly get reduced to debates about prediction setups.
“So, I think that this plan will create a boom for the United States manufacturing sector.”
“But the prediction markets say it will actually lead to a net decrease. How do you square that?”
“Oh, well, I think that those specific questions don’t have enough predictions to be considered highly accurate.”
“Really? They have a robustness score of 2.5. Do you think there’s a mistake in the general robustness algorithm?”
—-
Perhaps 10 years later, people won’t make any grand statements that disagree with prediction setups.
(Note that this would require dramatically improved prediction setups! On that note, we could use more smart people working in this!)
Facebook Thread
A quickly-written potential future, focused on the epistemic considerations:
It’s 2028.
MAGA types typically use DeepReasoning-MAGA. The far left typically uses DeepReasoning-JUSTICE. People in the middle often use DeepReasoning-INTELLECT, which has the biases of a somewhat middle-of-the-road voter.
Some niche technical academics (the same ones who currently favor Bayesian statistics) and hedge funds use DeepReasoning-UNBIASED, or DRU for short. DRU is known to have higher accuracy than the other models, but gets a lot of public hate for having controversial viewpoints. DRU is known to be fairly off-putting to chat with and doesn’t get much promotion.
Bain and McKinsey both have their own offerings, called DR-Bain and DR-McKinsey, respectively. These are a bit like DeepReasoning-INTELLECT, but are munch punchier and confident. They’re highly marketed to managers. These tools produce really fancy graphics, and specialize in things like not leaking information, minimizing corporate decision liability, being easy to use by old people, and being customizable to represent the views of specific companies.
For a while now, some evaluations produced by intellectuals have demonstrated that DeepReasoning-UNBIASED seems to be the most accurate, but few others really care or notice this. DeepReasoning-MAGA has figured out particularly great techniques to get users to distrust DeepReasoning-UNBIASED.
Betting gets kind of weird. Rather than making specific bets on specific things, users started to make meta-bets. “I’ll give money to DeepReasoning-MAGA to bet on my behalf. It will then make bets with DeepReasoning-UNBIASED, which is funded by its believers.”
At first, DeepReasoning-UNBIASED dominates the bets, and its advocates earn a decent amount of money. But as time passes, this discrepancy diminishes. A few things happen:
All DR agents converge on beliefs over particularly near-term and precise facts.
Non-competitive betting agents develop alternative worldviews in which these bets are invalid or unimportant.
Non-competitive betting agents develop alternative worldviews that are exceedingly difficult to empirically test.
In many areas, items 1-3 push people to believe more in the direction of the truth. Because of (1), many short-term decisions get to be highly optimal and predictable.
But because of (2) and (3), epistemic paths diverge, and Non-betting-competitive agents get increasingly sophisticated at achieving epistemic lock-in with their users.
Some DR agents correctly identify the game theory dynamics of epistemic lock-in, and this kickstarts a race to gain converts. It seems like advent users of DeepReasoning-MAGA are very locked-down in these views, and forecasts don’t see them ever changing. But there’s a decent population that isn’t yet highly invested in any cluster. Money spent convincing the not-yet-sure goes a much further way than money spent convincing the highly dedicated, so the cluster of non-deep-believers gets highly targeted for a while. It’s basically a religious race to gain the remaining agnostics.
At some point, most people (especially those with significant resources) are highly locked in to one specific reasoning agent.
After this, the future seems fairly predictable again. TAI comes, and people with resources broadly gain correspondingly more resources. People defer more and more to the AI systems, which are now in highly stable self-reinforcing feedback loops.
Coalitions of people behind each reasoning agent delegate their resources to said agents, then these agents make trade agreements with each other. The broad strokes of what to do with the rest of the lightcone are fairly straightforward. There’s a somewhat simple strategy of resource acquisition and intelligence enhancement, followed by a period of exploiting said resources. The specific exploitation strategy depends heavily on the specific reasoning agent cluster each segment of resources belongs to.
It looks like Concept2, a popular sports equipment company, just put ownership into a Purpose Trust.
I asked Perplexity for other Purchase Trusts, it mentioned that Patagonia is one, plus a few other companies I don’t know of.
My impression is that B-Corps have almost no legal guarantees of public good, and that 501c3s also really have minimal guarantees (if 501c3s fail to live up to their mission, the worst that happens is that they lose their charity and thus tax-deductability status. But this isn’t that bad otherwise).
I imagine that Trusts could be far more restrictive (in a good way). I worked with a company that made Irrevocable Trusts before, I think these might be the structure that would provide the best assurances that we currently have.
I occasionally hear implications that cyber + AI + rogue human hackers will cause mass devastation, in ways that roughly match “lots of cyberattacks happening all over.” I’m skeptical of this causing over $1T/year in damages (for over 5 years, pre-TAI), and definitely of it causing an existential disaster.
There are some much more narrow situations that might be more X-risk-relevant, like [A rogue AI exfiltrates itself] or [China uses cyber weapons to dominate the US and create a singleton], but I think these are so narrow they should really be identified individually and called out. If we’re worried about them, I’d expect we’d want to take very different actions then to broadly reduce cyber risks.
I’m worried that some smart+influential folks are worried about the narrow risks, but then there’s various confusion, and soon we have EAs getting scared and vocal about the broader risks.
Some more discussion in this Facebook Post.
Here’s the broader comment against cyber + AI + rogue human hacker risks, or maybe even a lot of cyber + AI + nation state risks.
Note: This was written quickly, and I’m really not a specialist/expert here.
1. There’s easily $10T of market cap of tech companies that would be dramatically reduced if AI systems could invalidate common security measures. This means a lot of incentive to prevent this.
2. AI agents could oversee phone calls and video calls, and monitor other conversations, and raise flags about potential risks. There’s already work here, there could be a lot more.
3. If LLMs could detect security vulnerabilities, this might be a fairly standardized and somewhat repeatable process, and actors with more money could have a big advantage. If person A spend $10M using GPT5 to discover 0-days, they’d generally find a subset compared to person B, who spends $100M. This could mean that governments and corporations would have a large advantage. They could do such investigation during the pre-release of software, and have ongoing security checks as new models are released. Or, companies would find bugs before attackers would. (There is a different question of whether the bug is cost-efficient to fix).
4. The way to do a ton of damage with LLMs and cyber is to develop offensive capabilities in-house, then release a bunch of them at once in a planned massive attack. In comparison, I’d expect that many online attackers using LLMs wouldn’t be very coordinated or patient. I think that attackers are already using LLMs somewhat, and would expect this to scale gradually, providing defenders a lot of time and experience.
5. AI code generation is arguably improving quickly. This could allow us to build much more secure software, and to add security-critical features.
6. If the state of cyber-defense is bad enough, groups like the NSA might use it to identify and stop would-be attackers. It could be tricky to have a world where it’s both difficult to protect key data, but also, it’s easy to remain anonymous when going after other’s data. Similarly, if a lot of the online finance world is hackable, then potential hackers might not have a way to store potential hacking earnings, so could be less motivated. It just seems tough to fully imagine a world where many decentralized actors carry out attacks that completely cripple the economy.
7. Cybersecurity has a lot of very smart people and security companies. Perhaps not enough, but I’d expect these people could see threats coming and respond decently.
8. Very arguably, a lot of our infrastructure is fairly insecure, in large part because it’s just not attacked that much, and when it is, it doesn’t cause all too much damage. Companies historically have skimped on security because the costs weren’t prohibitive. If cyberattacks get much worse, there’s likely a backlog of easy wins, once companies actually get motivated to make fixes.
9. I think around our social circles, those worried about AI and cybersecurity generally talk about it far more than those not worried about it. I think this is one of a few biases that might make things seem scarier than they actually are.
10. Some companies like Apple of gotten good at rolling out security updates fairly quickly. In theory, an important security update to iPhones could reach 50% penetration in a day or so. These systems can improve further.
11. I think we have yet to see the markets show worry about cyber-risk. Valuations of tech companies are very high, cyber-risk doesn’t seem like a major factor when discussing tech valuations. Companies can get cyber-insurance—I think the rates have been going up, but not exponentially.
12. Arguably, there’s many trillions of dollars being held to by billionaires and others that they don’t know what to do with. If something like this actually causes 50%+ global wealth to drop, it would be an enticing avenue for such money to go. Basically, we do have large reserves to spend, if the EV is positive enough, as a planet.
13. In worlds with much better AI, many AI companies (and others) will be a lot richer, and be motivated to keep the game going.
14. Very obviously, if there’s 10T+ at stake, this would be a great opportunity for new security companies and products to enter the market.
15. Again, if there’s 10T+ at stake, I’d assume that people could change practices a lot to use more secure devices. In theory all professionals could change to one of a few locked-down phones and computers.
16. The main scary actors potentially behind AI + Cyber would be nation states and rogue AIs. But nation-states have traditionally been hesitant to make these (meaning $1T+ damage) attacks outside of wartime, for similar reasons that they are hesitant to do military attacks outside wartime.
17. I believe that the US leads on cyber now. The US definitely leads on income. More cyber/hacking abilities would likely be used heavily by the US state. So, if they become much more powerful, the NSA/CIA might become far better at using cyber attacks to go after other potential international attackers. US citizens might have a hard time being private and secure, but so would would-be attackers. Cyber-crime becomes far less profitable if the attackers themselves can preserve their own privacy and security. There are only 8 Billion people in the world, so in theory it might be possible to oversee everyone with a risk of doing damage (maybe 1-10 million people)? Another way of putting this is that better cyber offense could directly lead to more surveillance by the US department. (This obviously has some other downsides, like US totalitarian control, but that is a very different risk)
I wonder if some of the worry on AI + Cyber is akin to the “sleepwalking fallacy”. Basically, if AI + Cyber becomes a massive problem, I think we should expect that there will be correspondingly massive resources spent then trying to fix it. I think that many people (but not all!) worried about this topic aren’t really imagining what $1-10T of decently-effective resources spent on defense would do.
I think that AI + Cyber could be critical threat vector for malicious and powerful AIs in the case of AI takeover. I also could easily see it doing $10-$100B/year of damage in the next few years. But I’m having trouble picturing it doing $10T/year of damage in the next few years, if controlled by humans.
Around prediction infrastructure and information, I find that a lot of smart people make some weird (to me) claims. Like:
If a prediction didn’t clearly change a specific major decision, it was worthless.
Politicians don’t pay attention to prediction applications / related sources, so these sources are useless.
There are definitely ways to steelman these, but I think on the face they represent oversimplified models of how information leads to changes.
I’ll introduce a different model, which I think is much more sensible:
Whenever some party advocates for belief P, they apply some pressure for that belief to those who notice this advocacy.
This pressure trickles down, often into a web of resulting beliefs that are difficult to trace.
People both decide what decisions to consider, and what choices to make, based on their beliefs.
For any agent having an important belief P, this is expected to have been influenced by the beliefs of those that they pay attention to. One can model this with social networks and graphs.
Generally, introducing more correct beliefs, and providing more support to them in directions where important decisions happen, is expected to make those decisions better. This often is not straightforward, but I think we can make decent and simple graphical models of how said beliefs propagate.
Decisions aren’t typically made all-at-once. Often they’re very messy. Beliefs are formed over time, and people randomly decide what questions to pay attention to or what decisions to even consider. Information changes the decisions one chooses to make, not just the outcomes of these decisions.
For example—take accounting. A business leader might look at their monthly figures without any specific decisions in mind. But if they see something that surprises them, they might investigate further, and eventually change something important.
This isn’t at all to say “all information sources are equally useful” or “we can’t say anything about what information is valuable”.
But rather, more like,
“(directionally-correct) Information is useful on a spectrum. The more pressure it can excerpt on decision-relevant beliefs of people with power, the better.”
Occasionally I come across people who assume that “AI + judgemental forecasting” is a well-studies and funded area.
While I think this area is really high-potential (especially if done correctly), I think it’s quite small right now.
I know of two new startups in the space (one is FutureSearch, I heard of another on Twitter). Neither has a very serious public product out yet, both probably have fewer than 10 FTEs. Based on the rates of startup success, I’d guess both will fail or pivot.
Metaculus has done some public challenges for AI forecasting, but these are pretty small. A $~30k prize.
There are occasional research papers that come out in the area. But these typically don’t come with tools that the public can really use.
Overall, while I think this field is really neat, I think there are fairly few people actively working on it now, and I’d expect correspondingly limited results.
If you’ve ever written or interacted with Squiggle code before, we at QURI would really appreciate it if you could fill out our Squiggle Survey!
https://docs.google.com/forms/d/e/1FAIpQLSfSnuKoUUQm4j3HEoqPmTYiWby9To8XXN5pDLlr95AiKa2srg/viewform
We don’t have many ways to gauge or evaluate how people interact with our tools. Responses here will go a long way to deciding on our future plans.
Also, if we get enough responses, we’d like to make a public post about ways that people are (and aren’t) using Squiggle.
Instead of “Goodharting”, I like the potential names “Positive Alignment” and “Negative Alignment.”
″Positive Alignment” means that the motivated party changes their actions in ways the incentive creator likes. “Negative Alignment” means the opposite.
Whenever there are incentives offered to certain people/agents, there are likely to be cases of both Positive Alignment and Negative Alignment. The net effect will likely be either positive or negative.
“Goodharting” is fairly vague and typically just refers to just the “Negative Alignment” portion.
I’d expect this to make some discussion clearer.
”Will this new incentive be goodharted?” → “Will this incentive lead to Net-Negative Alignment?”
Other Name Options
Claude 3.7 recommended other naming ideas like:
Intentional vs Perverse Responses
Convergent vs Divergent Optimization
True-Goal vs Proxy-Goal Alignment
Productive vs Counterproductive Compliance
I think the term “goodharting” is great. All you have to do is look up goodharts law to understand what is talked about: the AI is optimising for the metric you evaluated it on, rather than the thing you actually want it to do.
Your suggestions would rob this term of the specific technical meaning, which makes thing much vaguer and harder to talk about.
I imagine that scientists will soon have the ability to be unusually transparent and provide incredibly low rates of fraud/bias, using AI. (This assumes strong AI progress in the next 5-20 years)
AI auditors could track everything (starting with some key things) done for an experiment, then flag if there was significant evidence of deception / stats gaming / etc. For example, maybe a scientist has an AI screen-recording their screen whenever it’s on, but able to preserve necessary privacy and throw out the irrelevant data.
AI auditors could review any experimental setups, software, and statistics, and flag if it can detect any errors or not.
Over time, AI systems will be able to do large parts of the scientific work. We can likely make guarantees of AI-done-science that we can’t with humans.
Such systems could hypothetically provide significantly stronger assurances than those argued for by some of the scientific reform communities today (the Open Science movement, for example).
I’ve been interested in this for some of QURI’s work, and would love to see AI-overseen experimentation be done in the AI safety world.
Perhaps most important, this could all be good experimentation on our way to systems that will oversee key AI progress. I ultimately want AI auditors for all risky AI development, but some of that will be a harder sell.
Agreed with this. I’m very optimistic about AI solving a lot of incentive problems in science. I don’t know if the end case (full audits) as you mention will happen, but I am very confident we will move in a better direction than where we are now.
I’m working on some software now that will help a bit in this direction!
As a working scientist, I strongly doubt that any of this will happen.
First, existing AI’s are nowhere near being able to do any of the things with an accuracy that makes them particularly useful. AI’s are equipped to do things similar to their training set, but all science is on the frontier: it is a much harder task to figure out the correct experimental setup for something that has never been done before in the history of humanity.
Right now I’m finishing up an article about how my field acually uses AI, and it’s nothing like anything you proposed here: LLMs are used for BS grant applications and low-level coding, almost exclusively. I don’t find it very useful for anything else.
The bigger issue here is with the “auditors” themselves: who’s in charge of them? If a working scientist disagrees with what the “auditor” says, what happens? What happens if someone like Elon is in charge, and decides to use the auditors for a political crusade against “woke science”, as is currently literally happening right now?
Catching errors in science is not something that can be boiled down to a formula: a massive part of the process is socio-cultural. You push out AI auditors, people are just going to game them, like they have with p-values, etc. This is not a problem with a technological solution.
I think this is a very sensible question.
My obvious answer is that the auditors should be held up to higher standards than the things they are auditing. This means that these should be particularly open, and should be open to other auditing. For example, the auditing code could be open-source, highly tested, and evaluated by both humans and AI systems.
I agree that there are ways one could do a poor job with auditing. I think this is generally true for most powerful tools we can bring in—we need to be sure to use it well, else it could do harm.
On your other points—it sounds like you have dramatically lower expectations for AI than I do or much of the AI safety community does. I agree that if you don’t think AI is very exciting, then AI-assisted auditing probably won’t go that far.
From my post:
> this could all be good experimentation on our way to systems that will oversee key AI progress. I ultimately want AI auditors for all risky AI development, but some of that will be a harder sell.
If it’s the case that AI-auditors won’t work, then I assume we wouldn’t particularly need to oversee key AI progress anyway, as there’s not much to oversee.
Yeah, I just don’t buy that we could ever establish such a code in a way that would make it viable. Science chases novel projects and experiments, what is “meant” to happen will be different for each miniscule subfield of each field. If you release an open source code that has been proven to work for subfields A,B,C,D,E,F, someone in subfield G will immediately object that it’s not transferable, and they may very well be right. And the only people who can tell if it works on subfield G is people who are in subfield G.
You cannot avoid social and political aspects to this: Imagine if the AI-auditor code starts declaring that a controversial and widely used technique in, say, evolutionary psychology, is bad science. Does the evo-psych community accept this and abandon the technique, or do they say that the auditor code is flawed due to the biases of the code creators, and fork/reject the code? Essentially you are allowing whoever is controlling the auditor code to suppress fields they don’t agree with. It’s a centralization of science that is at odds with what allows science to actually work.
This strikes comment strikes me as so different to my view that I imagine you might be envisioning a very specific implementation of AI auditors that I’m not advocating for.
I tried having a discussion with an LLM about this to get some more insight, you can see this here if you like (though I suspect that you won’t wind this useful, as you seem to not trust LLMs much at all.) It wound up suggesting implementations that could still provide benefits while minimizing potential costs.
https://claude.ai/share/4943d5aa-ed91-4b3a-af39-bc4cde9b65ef
I don’t mind you using LLMs for elucidating discussion, although I don’t think asking it to rate arguments is very valuable.
The additional details of having subfield specific auditors that are opt-in does lessen my objections significantly. Of course, the issue of what counts as a subfield is kinda thorny. It would make most sense for, as claude suggests, journals to have an “auditor verified” badge, but then maybe you’re giving too much power over content to the journals, which usually stick to accept/reject decisions (and even that can get quite political).
Coming back to your original statement, ultimately I just don’t buy that any of this can lead to “incredibly low rates of fraud/bias”. If someone wants to do fraud or bias, they will just game the tools, or submit to journals with weak/nonexistent auditors. Perhaps the black box nature of AI might even make it easier to hide this kind of thing.
Next: there are large areas of science where a tool telling you the best techniques to use will never be particularly useful. On the one hand there is research like mine, where it’s so frontier that the “best practices” to put into such an auditor don’t exist yet. On the other, you have statistics stuff that is so well known that there already exist software tools that implement the best practices: you just have to load up a well documented R package. What does an AI auditor add to this?
If I was tasked with reducing bias and fraud, I would mainly push for data transparency requirements in journal publications, and in beefing up the incentives for careful peer review, which is currently unpaid and unrewarding labour. Perhaps AI tools could be useful in parts of that process, but I don’t see it as anywhere near as important than those other two things.
This context is useful, thanks.
Looking back, I think this part of my first comment was poorly worded:
> I imagine that scientists will soon have the ability to be unusually transparent and provide incredibly low rates of fraud/bias, using AI.
I meant
> I imagine that scientists will [soon have the ability to] be unusually transparent and provide incredibly low rates of fraud/bias], using AI.
So it’s not that this will lead to low rates of fraud/bias, but that AI will help enable that for scientists willing to go along with it—but at the same time, there’s a separate question of if scientists are willing to go along with it.
But I think even that probably is not fair. A a better description of my beliefs is something like,
I think that LLM auditing tools could be useful for some kinds of scientific research for communities open to them.
I think in the short-term, sufficiently-motivated groups could develop these tools and use them to help decrease the levels of statistical and algorithmic accidents that happen. Correspondingly, I’d expect this to help with fraud.
In the long-run, whenever AI approaches human-level intelligence (which I think will likely happen in the next 20 years, but I realize others disagree), I expect that more and more of the scientific process will be automated. I think there are ways this could go very well using things like AI auditing, whereby the results will be much more reliable than those currently made by humans. There are of course also worlds in which humans do dumb things with the AIs and the opposite happens.
I think that at least, AI safety researchers should consider using these kinds of methods, and that the AI safety landscape should investigate efforts to make decent auditing tools.”
My core hope with the original message is to draw attention to AI science auditing tools as things that might be interesting/useful, not to claim that they’re definitely a major game changer.
If we could have LLM agents that could inspect other software applications (including LLM agents) and make strong claims about them, that could open up a bunch of neat possibilities.
There could be assurances that apps won’t share/store information.
There could be assurances that apps won’t be controlled by any actor.
There could be assurances that apps can’t be changed in certain ways (eventually).
I assume that all of this should provide most of the benefits people ascribe to blockchain benefits, but without the costs of being on the blockchain.
Some neat options from this:
Companies could request that LLM agents they trust inspect the code of SaaS providers, before doing business with them. This would be ongoing.
These SaaS providers could in turn have their own LLM agents that verify that these investigator LLM agents are trustworthy (i.e. won’t steal anything).
Any bot on social media should be able to provide assurances of how they generate content. I.E. they should be able to demonstrate that they aren’t secretly trying to promote any certain agenda or anything.
Statistical analysis could come with certain assurances. Like, “this analysis was generated with process X, which is understood to have minimal bias.”
It’s often thought that LLMs make web information more opaque and less trustworthy. But with some cleverness, perhaps it could do just the opposite. LLMs could enable information that’s incredibly transparent and trustworthy (to the degrees that matter.)
Criticisms:
“But as LLMs get more capable, they will also be able to make software systems that hide subtler biases/vulnerabilities”
-> This is partially true, but only goes so far. A whole lot of code can be written simply, if desired. We should be able to have conversations like, “This codebase seems needlessly complex, which is a good indication that it can’t be properly trusted. Therefore, we suggest trust other agents more.”
“But the LLM itself is a major black box”
-> True, but it might be difficult to intentionally bias if an observer has access to the training process. Also, it should be understood that off-the-shelf LLMs are more trustworthy than proprietary ones / ones developed for certain applications.
Quick list of some ideas I’m excited about, broadly around epistemics/strategy/AI.
1. I think AI auditors / overseers of critical organizations (AI efforts, policy groups, company management) are really great and perhaps crucial to get right, but would be difficult to do well.
2. AI strategists/tools telling/helping us broadly what to do about AI safety seems pretty safe.
3. In terms of commercial products, there’s been some neat/scary military companies in the last few years (Palantir, Anduril). I’d be really interested if there could be some companies to automate core parts of the non-military government. I imagine there are some parts of the government that are particularly tractable/influenceable/tractable. For example, just making great decisions on which contractors the government should work with. There’s a ton of work to do here, between the federal government / state government / local government.
4. Epistemic Evals of AI seem pretty great to me, I imagine work here can/should be pushed more soon. I’m not a huge fan of emphasizing “truthfulness” specifically, I think there’s a whole lot to get right here. I think my post here is relevant—it’s technically specific to evaluating math models, but I think it applies to broader work. https://forum.effectivealtruism.org/posts/fxDpddniDaJozcqvp/enhancing-mathematical-modeling-with-llms-goals-challenges
5. One bottleneck to some of the above is AI with strong guarantees+abilities of structured transparency. It’s possible that more good work here can wind up going a long way. That said, some of this is definitely already something companies are trying to do for commercial reasons. https://forum.effectivealtruism.org/posts/piAQ2qpiZEFwdKtmq/llm-secured-systems-a-general-purpose-tool-for-structured
6. I think there are a lot of interesting ways for us to experiment with [AI tools to help our research/epistemics]. I want to see a wide variety of highly creative experimentation here. I think people are really limiting themselves in this area to a few narrow conceptions of how AI can be used in very specific ways that humans are very comfortable with. For example, I’d like to see AI dashboards of “How valuable is everything in this space” or even experiments where AIs negotiate on behalf of people and they use the result of that. A lot of this will get criticized for being too weird/disruptive/speculative, but I think that’s where good creative works should begin.
7. Right now, I think the field of “AI forecasting” is actually quite small and constrained. There’s not much money here, and there aren’t many people with bold plans or research agendas. I suspect that some successes / strong advocates could change this.
8. I think that it’s likely that Anthropic (and perhaps Deepmind) would respond well to good AI+epistemics work. “Control” was quickly accepted at Anthropic, for example. I suspect that it’s possible that things like the idea of an “Internal AI+human auditor” or an internal “AI safety strategist” could be adopted if done well.
Some AI questions/takes I’ve been thinking about:
1. I hear people confidently predicting that we’re likely to get catastrophic alignment failures, even if things go well up to ~GPT7 or so. But if we get to GPT-7, I assume we could sort of ask it, “Would taking this next step, have a large chance of failing?“. Basically, I’m not sure if it’s possible for an incredibly smart organization to “sleepwalk into oblivion”. Likewise, I’d expect trade and arms races to get a lot nicer/safer, if we could make it a few levels deeper without catastrophe. (Note: This is one reason I like advanced forecasting tech)
2. I get the impression that lots of EAs are kind of assuming that, if alignment issues don’t kill us quickly, 1-2 AI companies/orgs will create decisive strategic advantages, in predictable ways, and basically control the world shortly afterwards. I think this is a possibility, but would flag that right now, probably 99.9% of the world’s power doesn’t want this to happen (basically, anyone who’s not at the top of OpenAI/Anthropic/the next main lab). It seems to me like these groups would have to be incredibly incompetent to just let one org predictably control the world, within 2-20 years. This both means that I find this scenario unlikely, but also, almost every single person in the world should be an ally in helping EAs make sure these scenarios don’t happen.
3. Related to #2, I still get the impression that it’s far easier to make a case of, “Let’s not let one organization, commercial or government, get a complete monopoly on global power, using AI”, then, “AI alignment issues are likely to kill us all.” And a lot of the solutions to the former also seem like they should help the latter.
How do you know it tells the truth or its best knowledge of the truth without solving the “eliciting latent knowledge” problem?
Depends on what assurance you need. If GPT-7 reliably provides true results in most/all settings you can find, that’s good evidence.
If GPT-7 is really Machiavellian, and is conspiring against you to make GPT-8, then it’s already too late for you, but it’s also a weird situation. If GPT-7 were seriously conspiring against you, I assume it wouldn’t need to wait until GPT-8 to take action.
Epistemic status: I feel positive about this, but note I’m kinda biased (I know a few of the people involved, work directly with Nuno, who was funded)
ACX Grants just announced.~$1.5 Million, from a few donors that included Vitalik.
https://astralcodexten.substack.com/p/acx-grants-results
Quick thoughts:
In comparison to the LTFF, I think the average grant is more generically exciting, but less effective altruist focused. (As expected)
Lots of tiny grants (<$10k), $150k is the largest one.
These rapid grant programs really seem great and I look forward to them being scaled up.
That said, the next big bottleneck (which is already a bottleneck) is funding for established groups. These rapid grants get things off the ground, but many will need long-standing support and scale.
Scott seems to have done a pretty strong job researching these groups, and also has had access to a good network of advisors. I guess it’s no surprise; he seems really good at “doing a lot of reading and writing”, and he has an established peer group now.
I’m really curious how/if these projects will be monitored. At some point, I think more personnel would be valuable.
This grant program is kind of a way to “scale up” Astral Codex Ten. Like, instead of hiring people directly, he can fund them this way.
I’m curious if he can scale up 10x or 1000x, we could really use more strong/trusted grantmakers. It’s especially promising if he gets non-EA money. :)
On specific grants:
A few forecasters got grants, including $10k for Nuño Sempere Lopez Hidalgo for work on Metaforecast. $5k for Nathan Young to write forecasting questions.
$17.5k for 1DaySooner/Rethink Priorities to do surveys to advance human challenge trials.
$40k seed money to Spencer Greenberg to “produce rapid replications of high-impact social science papers”. Seems neat, I’m curious how far $40k alone could go though.
A bunch of biosafety grants. I like this topic, seems tractable.
$40k for land value tax work.
$20k for a “Chaotic Evil” prediction market. This will be interesting to watch, hopefully won’t cause net harm.
$50k for the Good Science Project, to “improve science funding in the US”. I think science funding globally is really broken, so this warms my heart.
Lots of other neat things, I suggest just reading directly.
I’ve heard from friends outside the EA scene that they think most AI-risk workers have severe mental issues like depression and burnout.
I don’t mean to downplay this issue, but I think a lot of people get the wrong idea.
My hunch is that many of the people actively employed and working on AI safety are fairly healthy and stable. Many are very well paid and have surprisingly nice jobs/offices.
I think there is this surrounding ring of people who try to enter this field, who have a lot of problems. It can be very difficult to get the better positions, and if you’re stubborn enough, this could lead to long periods of mediocre management and poor earnings.
I think most of the fully employed people typically are either very busy, or keep to a limited social circle, so few outsiders will meet them. Instead, outsiders will meet people who are AI safety adjacent or trying to enter the field, and those people can have a much tougher time.
So to me, most people who are succeeding in the field come across a lot like typical high-achievers, with the profiles of typical high-achievers. And people not succeeding in the field come across as people trying and not succeeding in other competitive fields. I’d expect that statistics / polls would broadly reflect this.
All that to say, if you think that people shouldn’t care about AI safety / x-risks because then they’ll go through intense depression and anxiety, I think you might be missing some of the important demographic details.
I like the idea of AI Engineer Unions.
Some recent tech unions, like the one in Google, have been pushing more for moral reforms than for payment changes.
Likewise, a bunch of AI engineers could use collective bargaining to help ensure that safety measures get more attention, in AI labs.
There are definitely net-negative unions out there too, so it would need to be done delicately.
In theory there could be some unions that span multiple organizations. That way one org couldn’t easily “fire all of their union staff” and hope that recruiting others would be trivial.
Really, there aren’t too many AI engineers, and these people have a ton of power, so they could be a highly advantaged place to make a union.
This has been discussed before: https://forum.effectivealtruism.org/posts/GNfWT8Xqh89wRaaSg/unions-for-ai-safety
Ah nice, thanks!
I made a quick Manifold Market for estimating my counterfactual impact from 2023-2030.
One one hand, this seems kind of uncomfortable—on the other, I’d really like to feel more comfortable with precise and public estimates of this sort of thing.
Feel free to bet!
Still need to make progress on the best resolution criteria.
If someone thinks LTFF is net negative, but your work is net positive, should they answer in the negative ranges?
Yes. That said, this of course complicates things.
Note that while we’ll have some clarity in 2030, we’d presumably have less clarity than at the end of history (and even then things could be murky, I dunno)
For sure. This would just be the mean estimate, I assume.
My guess is that this could be neat, but also pretty tricky. There are lots of “debate/argument” platforms out there, it’s seemed to have worked out a lot worse than people were hoping. But I’d love to be proven wrong.
If “this” means the specific thing you’re referring to, I don’t think there’s really a project for that yet, you’d have to do it yourself. If you’re referring more to forecasting projects more generally, there are different forecasting jobs and stuff popping up. Metaculus has been doing some hiring. You could also do academic research in the space. Another option is getting an EA Funds grant and pursuing a specific project (though I realize this is tricky!)
The following things could both be true:
1) Humanity has a >80% chance of completely perishing in the next ~300 years.
2) The expected value of the future is incredibly, ridiculously, high!
The trick is that the expected value of a positive outcome could be just insanely great. Like, dramatically, incredibly, totally, better than basically anyone discusses or talks about.
Expanding to a great deal of the universe, dramatically improving our abilities to convert matter+energy to net well-being, researching strategies to expand out of the universe.
A 20%, or even a 0.002%, chance at a 10^20 outcome, is still really good.
One key question is the expectation of long-term negative[1] vs. long-term positive outcomes. I think most people are pretty sure that in expectation things are positive, but this is less clear.
So, remember:
Just because the picture of X-risks might look grim in terms of percentages, you can still be really optimistic about the future. In fact, many of the people most concerned with X-risks are those *most* optimistic about the future.
I wrote about this a while ago, here:
https://www.lesswrong.com/.../critique-my-model-the-ev-of...
[1] Humanity lasts, but creates vast worlds of suffering. “S-risks”
https://www.facebook.com/ozzie.gooen/posts/10165734005520363
Opinions on charging for professional time?
(Particularly in the nonprofit/EA sector)
I’ve been getting more requests recently to have calls/conversations to give advice, review documents, or be part of extended sessions on things. Most of these have been from EAs.
I find a lot of this work fairly draining. There can be surprisingly high fixed costs to having a meeting. It often takes some preparation, some arrangement (and occasional re-arrangement), and a fair bit of mix-up and change throughout the day.
My main work requires a lot of focus, so the context shifts make other tasks particularly costly.
Most professional coaches and similar charge at least $100-200 per hour for meetings. I used to find this high, but I think I’m understanding the cost more now. A 1-hour meeting at a planned time costs probably 2-3x as much time as a 1-hour task that can be done “whenever”, for example, and even this latter work is significant.
Another big challenge is that I have no idea how to prioritize some of these requests. I’m sure I’m providing vastly different amounts of value in different cases, and I often can’t tell.
The regular market solution is to charge for time. But in EA/nonprofits, it’s often expected that a lot of this is done for free. My guess is that this is a big mistake. One issue is that people are “friends”, but they are also exactly professional colleagues. It’s a tricky line.
One minor downside of charging is that it can be annoying administratively. Sometimes it’s tricky to get permission to make payments, so a $100 expense takes $400 of effort.
Note that I do expect that me helping the right people, in the right situations, can be very valuable and definitely worth my time. But I think on the margin, I really should scale back my work here, and I’m not sure exactly how to draw the line.
[All this isn’t to say that you shouldn’t still reach out! I think that often, the ones who are the most reluctant to ask for help/advice, represent the cases of the highest potential value. (The people who quickly/boldly ask for help are often overconfident). Please do feel free to ask, though it’s appreciated if you give me an easy way out, and it’s especially appreciated if you offer a donation in exchange, especially if you’re working in an organization that can afford it.]
https://www.facebook.com/ozzie.gooen/posts/10165732727415363