At Risk of violating @Linch’s principle “Assume by default that if something is missing in EA, nobody else is going to step up.”, I think it would be valuable to have a well researched estimate of the counterfactual value of getting investment from different investors (whether for profit or donors).
For example in global health, we could make GiveWell the baseline, as I doubt whether there is a ll funding source where switching as less impact, as the money will only ever be shifted from something slightly less effective. For example if my organisation received funding from GiveWell, we might only make slightly better use of that money than where it would otherwise have gone, and we’re not going to be increasing the overall donor pool either.
Who knows, for-profit investment dollars could be 10x −100x more counterfactually impactful than GiveWell, which could mean a for-profit company trying to do something good could plausibly be 10-100x less effective than a charity and still doing as much counterfactual good overall? Or is this a stretch?
This would be hard to estimate but doable, and must have been done at least on a casual scale by some people.
Examples ( and random guesses) of counterfactual comparisons of the value of each dollar given by a particular source might be something like....
I’m admittedly a bit more confused by your fleshed-out example with random guesses than I was when I read your opening sentence, as it went in a different direction than I expected (using multipliers instead of subtracting the value of the next-best alternative use of funds), so maybe we’re thinking about different things. I also didn’t understand what you meant by this when I tried to flesh it out myself with some (made-up) numbers:
Who knows, for-profit investment dollars could be 10x −100x more counterfactually impactful than GiveWell, which could mean a for-profit company trying to do something good could plausibly be 10-100x less effective than a charity and still doing as much counterfactual good overall?
If it helps, GiveWell has a spreadsheet summarising their analysis of the counterfactual value of other actors’ spending. From there and a bit of arithmetic, you can figure out their estimates of the value generated by $1 million in spending from the following sources, expressed in units of doubling consumption for one person for a year:
Government health spending: 5,639 units (this is also GW’s assumed value generated by philanthropic actors’s spending on VAS and vitamin A capsule procurement in other countries)
Suppose there’s an org (“EffectiveHealth”) that could create 50k units worth of benefits given $1M in funding. If it came from GW with their 33.5k units counterfactual value, then a grant to EffectiveHealth from GW would be creating 16.5k more units of benefits than otherwise, while if it came from domestic govt spending (7.5x lower value at 5k per $1M) it’d create 45k more units of benefits. A hypothetical for-profit investor whose funding generates 10x less good than domestic govt spending (500 units) would get you not 10 x 45k = 450k more units of benefits, but 49.5k units. And if EffectiveHealth got funding from OP it would actually be net-negative by −20k units.
(maybe you meant something completely different, in which case apologies!)
Questions would just be finding out what the cost-effectiveness of money from current individual donors, philanthropies, Gates Foundation was, then calculating the difference if it went to the most cost-effective rganisations.
Reflections on “Status Handcuffs” over one’s career
(This was edited using Claude)
Having too much professional success early on can ironically restrict you later on. People typically are hesitant to go down in status when choosing their next job. This can easily mean that “staying in career limbo” can be higher-status than actually working. At least when you’re in career limbo, you have a potential excuse.
This makes it difficult to change careers. It’s very awkward to go from “manager of a small team” to “intern,” but that can be necessary if you want to learn a new domain, for instance.
The EA Community Context
In the EA community, some aspects of this are tricky. The funders very much want to attract new and exciting talent. But this means that the older talent is in an awkward position.
The most successful get to take advantage of the influx of talent, with more senior leadership positions. But there aren’t too many of these positions to go around. It can feel weird to work on the same level or under someone more junior than yourself.
Pragmatically, I think many of the old folks around EA are either doing very well, or are kind of lost/exploring other avenues. Other areas allow people to have more reputable positions, but these are typically not very EA/effective areas. Often E2G isn’t very high-status in these clusters, so I think a lot of these people just stop doing much effective work.
Similar Patterns in Other Fields
This reminds me of law firms, which are known to have “up or out” cultures. I imagine some of this acts as a formal way to prevent this status challenge—people who don’t highly succeed get fully kicked out, in part because they might get bitter if their career gets curtailed. An increasingly narrow set of lawyers continue on the Partner track.
I’m also used to hearing about power struggles for senior managers close to retirement at big companies, where there’s a similar struggle. There’s a large cluster of highly experienced people who have stopped being strong enough to stay at the highest levels of management. Typically these people stay too long, then completely leave. There can be few paths to gracefully go down a level or two while saving face and continuing to provide some amount of valuable work.
But around EA and a lot of tech, I think this pattern can happen much sooner—like when people are in the age range of 22 to 35. It’s more subtle, but it still happens.
Finding Solutions
I’m very curious if it’s feasible for some people to find solutions to this. One extreme would be, “Person X was incredibly successful 10 years ago. But that success has faded, and now the only useful thing they could do is office cleaning work. So now they do office cleaning work. And we’ve all found a way to make peace with this.”
Traditionally, in Western culture, such an outcome would be seen as highly shameful. But in theory, being able to find peace and satisfaction from something often seen as shameful for (what I think of as overall-unfortunate) reasons could be considered a highly respectable thing to do.
Perhaps there could be a world where [valuable but low-status] activities are identified, discussed, and later turned to be high-status.
The EA Ideal vs. Reality
Back to EA. In theory, EAs are people who try to maximize their expected impact. In practice, EA is a specific ideology that typically has a limited impact on people (at least compared to strong Religious groups, for instance). I think that the EA scene has demonstrated success at getting people to adjust careers (in circumstances where it’s fairly cheap and/or favorable to do so), and has created an ecosystem that rewards people for certain EA behaviors. But at the same time, people typically feature with a great deal of non-EA constraints that must be continually satisfied for them to be productive; money, family, stability, health, status, etc.
Personal Reflection
Personally, every few months I really wonder what might make sense for me. I’d love to be the kind of person who would be psychologically okay doing the lowest-status work for the youngest or lowest-status people. At the same time, knowing myself, I’m nervous that taking a very low-status position might cause some of my mind to feel resentment and burnout. I’ll continue to reflect on this.
I’ve just ran into this, so excuse a bit of grave digging. As someone who has entered the EA community with prior career experience I disagree with your premise
“It’s very awkward to go from “manager of a small team” to “intern,” but that can be necessary if you want to learn a new domain, for instance.”
To me this kind of situation just shouldn’t happen. It’s not a question of status, it’s a question of inefficiency. If I have managerial experience and the organization I’d be joining can only offer me the exact same job they’d be offering to a fresh grad, then they are simply wasting my potential. I’d be better off at a place which can appreciate what I bring and the organization would be better off with someone who has a fresher mind and less tempting alternatives.
IMO the problem is not with the fact that people are unwilling to take a step down. The problem is with EA orgs unwilling or unable to leverage the transferrable skills of experienced professionals, forcing them into entry-level positions instead.
I agree with you. I think in EA this is especially the case because much of the community-building work is focused on universities/students, and because of the titling issue someone else mentioned. I don’t think someone fresh out of uni should be head of anything, wah. But the EA movement is young and was started by young people, so it’ll take a while for career-long progression funnels to develop organically.
Thanks for writing this, this is also something I have been thinking about and you’ve expressed it more eloquently.
One thing I have thought might be useful is at times showing restraint with job titling. I’ve observed cases where people have had a title for example Director in a small org or growing org, and in a larger org this role might be a coordinator, lead, admin.
I’ve thought at times this doesn’t necessarily set people up for long term career success as the logical career step in terms of skills and growth, or a career shift, often is associated with a lower sounding title. Which I think decreases motivation to take on these roles.
At the same time I have seen people, including myself, take a decrease in salary and title, in order to shift careers and move forward.
A related issue I have actually encountered is something like “but you seem overqualified for this role we are hiring for”. Even if previously successful people wanted to take a “less prestigious” role, they might encounter real problems in doing so. I hope the EA eco system might have some immunity to this though—as hopefully the mission alignment will be strong enough evidence of why such a person might show interest in a “lower” role.
As a single data point: seconded. I’ve explicitly been asked by interviewers (in a job interview) why I left a “higher title job” for a “lower title job,” with the implication that it needed some special justification. I suspect there have also been multiple times in which someone looking at my resume saw that transition, made an assumption about it, and choose to reject me. (although this probably happens with non-EA jobs more often than EA jobs, as the “lower title role” was with a well-known EA organization)
Pragmatically, I think many of the old folks around EA are either doing very well, or are kind of lost/exploring other avenues. Other areas allow people to have more reputable positions, but these are typically not very EA/effective areas. Often E2G isn’t very high-status in these clusters, so I think a lot of these people just stop doing much effective work.
I haven’t really noticed this happening very much empirically, but I do think the effect you are talking about is quite intuitive. Have you seen many cases of this that you’re confident are correct (e.g. they aren’t lost for other reasons like working on non-public projects or being burnt out)? No need to mention specific names.
In theory, EAs are people who try to maximize their expected impact. In practice, EA is a light ideology that typically has a limited impact on people. I think that the EA scene has demonstrated success at getting people to adjust careers (in circumstances where it’s fairly cheap and/or favorable to do so)
This seems incorrect to me, in absolute terms. By the standards of ~any social movement, EAs are very sacrificial and focused on increasing their impact. I suspect you somewhat underrate how rare it is outside of EA to be highly committed to ~any non-self-serving principles seriously enough to sacrifice significant income and change careers, particularly in new institutions/movements.
Have you seen many cases of this that you’re confident are correct (e.g. they aren’t lost for other reasons like working on non-public projects or being burnt out)? No need to mention specific names.
I’m sure that very few of these are explained by “non-public projects”.
I’m unsure about burnout. I’m not sure where the line is between “can’t identify high-status work to do” and burnout. I expect that the two are highly correlated. My guess is that they don’t literally think of it as “I’m low status now”, instead I’d expect them to feel emotions like resentment / anger / depression. But I’d also expect that if we could change the status lever, other negative feelings would go away. (I think that status is a big deal for people! Like, status means you have a good career, get to be around people you like, etc)
> I suspect you somewhat underrate how rare it is outside of EA to be highly committed to ~any non-self-serving principles seriously enough to sacrifice significant income and change careers.
I suspect we might have different ideologies in mind to compare to, and correspondingly, that we’re not disagreeing much.
I think that a lot of recently-popular movements like BLM or even MAGA didn’t change the average lifestyle of the median participant much at all, though much of this is because they are far larger.
But religious groups are far more intense, for example. Or maybe take dedicated professional specialties like ballet or elite music, which can require intense sacrifices.
The recent rise of AI Factory/Neocloud companies like CoreWeave, Lambda and Crusoe strikes me as feverish and financially unsound. These companies are highly overleveraged, offering GPU access as a commodity to a monopsony. Spending vast amounts of capex on a product that must be highly substitutive to compete with hyperscalers on cost strikes me as an unsustainable business model in the long term. The association of these companies with the ‘AI Boom’ could cause collateral reputation damage to more reputable firms if these Neoclouds go belly up.
And, if you’re in a race-to-the-bottom price war on margins, fewer resources are devoted to risk management and considerate deployment. A recent SemiAnalysis article confirms the “long tail” of these companies that lack even basic ISO27001 or SOC 2 certs.
I should say that the Sovereign Neocloud model seems to alleviate both the financial and x-risk concerns; however, that’s mostly relevant outside the US and China.
I’d be excited to see 1-2 opportunistic EA-rationalist types looking into where marginal deregulation is a bottleneck to progress on x-risk/GHW, circulating 1-pagers among experts in these areas, and then pushing the ideas to DOGE/Mercatus/Executive Branch. I’m thinking things like clinical trials requirements for vaccines, UV light, anti-trust issues facing companies collaborating on safety and security, maybe housing (though I’m not sure which are bottlenecked by federal action). For most of these there’s downside risk if the message is low fidelity, the issue becomes polarized, or priorities are poorly set, hence collaborating with experts. I doubt there’s that much useful stuff to be done here, but marginal deregulation looks very easy right now and looks good to strike while the iron is hot.
Thought these quotes from Holden’s old (2011) GW blog posts were thought-provoking, unsure to what extent I agree. In In defense of the streetlight effect he argued that
If we focus evaluations on what can be evaluated well, is there a risk that we’ll also focus on executing programs that can be evaluated well? Yes and no.
Some programs may be so obviously beneficial that they are good investments even without high-quality evaluations available; in these cases we should execute such programs and not evaluate them.
But when it comes to programs that where evaluation seems both necessary and infeasible, I think it’s fair to simply de-emphasize these sorts of programs, even if they might be helpful and even if they address important problems. This reflects my basic attitude toward aid as “supplementing people’s efforts to address their own problems” rather than “taking responsibility for every problem clients face, whether or not such problems are tractable to outside donors.” I think there are some problems that outside donors can be very helpful on and others that they’re not well suited to helping on; thus, “helping with the most important problem” and “helping as much as possible” are not at all the same to me.
(I appreciate the bolded part, especially as something baked into GW’s approach and top recs by $ moved.)
[Project AK-47′s emotionally appealing pitch to donors is] an extreme example of a style of argument common to nonprofits: point to a problem so large and severe (and the world has many such problems) that donors immediately focus on that problem – feeling compelled to give to the organization working on addressing it – without giving equal attention to the proposed solution, how much it costs, and how likely it is to work. …
Many of the donors we hear from are passionately committed to fighting global warming because it’s the “most pressing problem,” or to a particular disease because it affected them personally – even while freely admitting that they know nothing about the most promising potential solutions. I ask these donors to consider the experience related by William Easterly:
I am among the many who have tried hard to find the answer to the question of what the end of poverty requires of foreign aid. I realized only belatedly that I was asking the question backward … the right way around [is]: What can foreign aid do for poor people? (White Man’s Burden pg 11)
As a single human being, your powers are limited. As a donor, you’re even more limited – you’re not giving your talent or your creativity, just your money. This creates a fundamentally different challenge from identifying the problem you care most about, and can lead to a completely different answer.
The truth is that you may not be able to do anything to help address the root causes of poverty or cure cancer or solve the global energy crisis.* But you probably can save a life, and insisting on giving to the “biggest problem” could be passing up that chance.
LLMs seem more like low-level tools to me than direct human interfaces.
Current models suffer from hallucinations, sycophancy, and numerous errors, but can be extremely useful when integrated into systems with redundancy and verification.
We’re in a strange stage now where LLMs are powerful enough to be useful, but too expensive/slow to have rich scaffolding and redundancy. So we bring this error-prone low-level tool straight to the user, for the moment, while waiting for the technology to improve.
Using today’s LLM interfaces feels like writing SQL commands directly instead of using a polished web application. It’s functional if that’s all you have, but it’s probably temporary.
Imagine what might happen if/when LLMs are 1000x faster and cheaper.
Then, answering a question might involve:
Running ~100 parallel LLM calls with various models and prompts
Using aggregation layers to compare responses and resolve contradictions
Identifying subtasks and handling them with specialized LLM batches and other software
Big picture, I think researchers might focus less on making sure any one LLM call is great, and more that these broader setups can work effectively.
(I realize this has some similarities to Mixture of Experts)
I guess orgs need to be more careful about who they hire as forecasting/evals researchers in light of a recently announced startup.
Sometimes things happen, but three people at the same org...
This is also a massive burning of the commons. It is valuable for forecasting/evals orgs to be able to hire people with a diversity of viewpoints in order to counter bias. It is valuable for folks to be able to share information freely with folks at such forecasting orgs without having to worry about them going off and doing something like this.
However, this only works if those less worried about AI risks who join such a collaboration don’t use the knowledge they gain to cash in on the AI boom in an acceleratory way. Doing so undermines the very point of such a project, namely, to try to make AI go well. Doing so is incredibly damaging to trust within the community.
Now let’s suppose you’re an x-risk funder considering whether to fund their previous org. This org does really high-quality work, but the argument for them being net-positive is now significantly weaker. This is quite likely to make finding future funding harder for them.
This is less about attacking those three folks and more just noting that we need to strive to avoid situations where things like this happen in the first place. This requires us to be more careful in terms of who gets hired.
There’s been some discussions on the EA forum along the lines of “why do we care about value alignment shouldn’t we just hire who can best do the job”. My answer to that is that it’s myopic to only consider what happens whilst they’re working for you. Hiring someone or offering them an opportunity empowers them, you need to consider whether they’re someone who you want to empower[1].
Admittedly, this isn’t quite the same as value alignment. Suppose someone were diligent, honest, wise and responsible. You might want to empower them even if their views were extremely different from yours. Stronger: even if their views were the opposite in many ways. But in the absence of this, value alignment matters.
If you only hire people who you believe are intellectually committed to short AGI timelines (and who won’t change their minds given exposure to new evidence and analysis) to work in AGI forecasting, how can you do good AGI forecasting?
One of the co-founders of Mechanize, who formerly worked at Epoch AI, says he thinks AGI is 30 to 40 years away. That was in this video from a few weeks ago on Epoch AI’s YouTube channel.
He and one of his co-founders at Mechanize were recently on Dwarkesh Patel’s podcast (note: Dwarkesh Patel is an investor in Mechanize) and I didn’t watch all of it but it seemed like they were both arguing for longer AGI timelines than Dwarkesh believes in.
I also disagree with the shortest AGI timelines and found it refreshing that within the bubble of people who are fixated on near-term AGI, at least a few people expressed a different view.
I think if you restrict who you hire to do AGI forecasting based on strong agreement with a predetermined set of views, such as short AGI timelines and views on AGI alignment and safety, then you will just produce forecasts that re-state the views you already decided were the correct ones while you were hiring.
I wasn’t suggesting only hiring people who believe in short-timelines. I believe that my original post adequately lays out my position, but if any points are ambiguous, feel free to request clarification.
I don’t know how Epoch AI can both “hire people with a diversity of viewpoints in order to counter bias” and ensure that your former employees won’t try to “cash in on the AI boom in an acceleratory way”. These seem like incompatible goals.
I think Epoch has to either:
Accept that people have different views and will have different ideas about what actions are ethical, e.g., they may view creating an AI startup focused on automating labour as helpful to the world and benign
or
Only hire people who believe in short AGI timelines and high AGI risk and, as a result, bias its forecasts towards those conclusions
Presumably there are at least some people who have long timelines, but also believe in high risk and don’t want to speed things up. Or people who are unsure about timelines, but think risk is high whenever it happens. Or people (like me) who think X-risk is low* and timelines very unclear, but even a very low X-risk is very bad. (By very low, I mean like at least 1 in 1000, not 1 in 1x10^17 or something. I agree it is probably bad to use expected value reasoning with probabilities as low as that.)
I think you are pointing at a real tension though. But maybe try to see it a bit from the point of view of people who think X-risk is real enough and raised enough by acceleration that acceleration is bad. It’s hardly going to escape their notice that projects at least somewhat framed as reducing X-risk often end up pushing capabilities forward. They don’t have to be raging dogmatists to worry about this happening again, and it’s reasonable for them to balance this risk against risks of echo chambers when hiring people or funding projects.
*I’m less surely merely catastrophic biorisk from human misuse is low sadly.
Why don’t we ask ChatGPT? (In case you’re wondering, I’ve read every word of this answer and I fully endorse it, though I think there are better analogies that the journalism example ChatGPT used).
Hopefully, this clarifies a possible third option (one that my original answer was pointing at).
I think there is a third option, though it’s messy and imperfect. The third option is to:
Maintain epistemic pluralism at the level of research methods and internal debate, while being selective about value alignment on key downstream behaviors.
In other words:
You hire researchers with a range of views on timelines, takeoff speeds, and economic impacts, so long as they are capable of good-faith engagement and epistemic humility.
But you also have clear social norms, incentives, and possibly contractual commitments around what counts as harmful conflict of interest — e.g., spinning out an acceleratory startup that would directly undermine the mission of your forecasting work.
This requires drawing a distinction between research belief diversity and behavioral alignment on high-stakes actions. That’s tricky! But it’s not obviously incoherent.
The key mechanism that makes this possible (if it is possible) is something like:
“We don’t need everyone to agree on the odds of doom or the value of AGI automation in theory. But we do need shared clarity on what types of action would constitute a betrayal of the mission or a dangerous misuse of privileged information.”
So you can imagine hiring someone who thinks timelines are long and AGI risk is overblown but who is fully on board with the idea that, given the stakes, forecasting institutions should err on the side of caution in their affiliations and activities.
This is analogous to how, say, journalists might disagree about political philosophy but still share norms about not taking bribes from the subjects they cover.
Caveats and Challenges:
Enforceability is hard. Noncompetes are legally dubious in many jurisdictions, and “cash in on the AI boom” is vague enough that edge cases will be messy. But social signaling and community reputation mechanisms can still do a lot of work here.
Self-selection pressure remains. Even if you say you’re open to diverse views, the perception that Epoch is “aligned with x-risk EAs” might still screen out applicants who don’t buy the core premises. So you risk de facto ideological clustering unless you actively fight against that.
Forecasting bias could still creep in via mission alignment filtering. Even if you welcome researchers with divergent beliefs, if the only people willing to comply with your behavioral norms are those who already lean toward the doomier end of the spectrum, your epistemic diversity might still collapse in practice.
Summary:
The third option is:
Hire for epistemic virtue, not belief conformity, while maintaining strict behavioral norms around acceleratory conflict of interest.
It’s not a magic solution — it requires constant maintenance, good hiring processes, and clarity about the boundaries between “intellectual disagreement” and “mission betrayal.” But I think it’s at least plausible as a way to square the circle.”
So, you want to try to lock in AI forecasters to onerous and probably illegal contracts that forbid them from founding an AI startup after leaving the forecasting organization? Who would sign such a contract? This is even worse than only hiring people who are intellectually pre-committed to certain AI forecasts. Because it goes beyond a verbal affirmation of their beliefs to actually attempting to legally force them to comply with the (putative) ethical implications of certain AI forecasts.
If the suggestion is simply promoting “social norms” against starting AI startups, well, that social norm already exists to some extent in this community, as evidenced by the response on the EA Forum. But if the norm is too weak, it won’t prevent the undesired outcome (the creation of an AI startup), and if the norm is too strong, I don’t see how it doesn’t end up selecting forecasters for intellectual conformity. Because non-conformists would not want to go along with such a norm (just like they wouldn’t want to sign a contract telling them what they can and can’t do after they leave the forecasting company).
I agree that we need to be careful about who we are empowering.
“Value alignment” is one of those terms which has different meanings to different people. For example, the top hit I got on Google for “effective altruism value alignment” was a ConcernedEAs post which may not reflect what you mean by the term. Without knowing exactly what you mean, I’d hazard a guess that some facets of value alignment are pretty relevant to mitigating this kind of risk, and other facets are not so important. Moreover, I think some of the key factors are less cognitive or philosophical than emotional or motivational (e.g., a strong attraction toward money will increase the risk of defecting, a lack of self-awareness increases the risk of motivated reasoning toward goals one has in a sense repressed).
So, I think it would be helpful for orgs to consider what elements of “value alignment” are of particular importance here, as well as what other risk or protective factors might exist outside of value alignment, and focus on those specific things.
Also, it is worrying if the optimists easily find financial opportunities that depend on them not changing their minds. Even if they are honest and have the best of intentions, the disparity in returns to optimism is epistemically toxic.
I’d like to suggest a little bit more clarity here. The phrases you use refer to some knowledge that isn’t explicitly stated here. “in light of a recently announced startup” and “three people at the same org” make sense to someone who already knows the context of what you are writing about, but it is confusing to a reader who doesn’t have the same background knowledge that you do.
Once upon a time, some people were arguing that AI might kill everyone, and EA resources should address that problem instead of fighting Malaria.
So OpenPhil poured millions of dollars into orgs such as EpochAI (they got 9 million).
Now 3 people from EpochAI created a startup to provide training data to help AI replace human workers.
Some people are worried that this startup increases AI capabilities, and therefore increases the chance that AI will kill everyone.
100 percent agree. I dont understand the entire post because I don’t know the context. I don’t think alluding to something helps, better to say it explicitly.
I’m not sure how to best write about these on the EA Forum / LessWrong. They feel too technical and speculative to gain much visibility.
But I’m happy for people interested in the area to see them. Like with all things, I’m eager for feedback.
Here’s a brief summary of them, written by Claude.
---
1. AI-Assisted Auditing
A system where AI agents audit humans or AI systems, particularly for organizations involved in AI development. This could provide transparency about data usage, ensure legal compliance, flag dangerous procedures, and detect corruption while maintaining necessary privacy.
2. Consistency Evaluations for Estimation AI Agents
A testing framework that evaluates AI forecasting systems by measuring several types of consistency rather than just accuracy, enabling better comparison and improvement of prediction models. It’s suggested to start with simple test sets and progress to adversarial testing methods that can identify subtle inconsistencies across domains.
3. AI for Epistemic Impact Estimation
An AI tool that quantifies the value of information based on how it improves beliefs for specific AIs. It’s suggested to begin with narrow domains and metrics, then expand to comprehensive tools that can guide research prioritization, value information contributions, and optimize information-seeking strategies.
4. Multi-AI-Critic Document Comments & Analysis
A system similar to “Google Docs comments” but with specialized AI agents that analyze documents for logical errors, provide enrichment, and offer suggestions. This could feature a repository of different optional open-source agents for specific tasks like spot-checking arguments, flagging logical errors, and providing information enrichment.
5. Rapid Prediction Games for RL
Specialized environments where AI agents trade or compete on predictions through market mechanisms, distinguishing between Information Producers and Consumers. The system aims to both evaluate AI capabilities and provide a framework for training better forecasting agents through rapid feedback cycles.
6. Analytics on Private AI Data
A project where government or researcher AI agents get access to private logs/data from AI companies to analyze questions like: How often did LLMs lie or misrepresent information? Did LLMs show bias toward encouraging user trust? Did LLMs employ questionable tactics for user retention? This addresses the limitation that researchers currently lack access to actual use logs.
7. Prediction Market Key Analytics Database
A comprehensive analytics system for prediction markets that tracks question value, difficulty, correlation with other questions, and forecaster performance metrics. This would help identify which questions are most valuable to specific stakeholders and how questions relate to real-world variables.
8. LLM Resolver Agents
A system for resolving forecasting questions using AI agents with built-in desiderata including: triggering experiments at specific future points, deterministic randomness methods, specified LLM usage, verifiability, auditability, proper caching/storing, and sensitivity analysis.
9. AI-Organized Information Hubs
A platform optimized for AI readers and writers where systems, experts, and organizations can contribute information that is scored and filtered for usefulness. Features would include privacy levels, payment proportional to information value, and integration of multiple file types.
I’m not sure how to word this properly, and I’m uncertain about the best approach to this issue, but I feel it’s important to get this take out there.
Yesterday, Mechanize was announced, a startup focused on developing virtual work environments, benchmarks, and training data to fully automate the economy. The founders include Matthew Barnett, Tamay Besiroglu, and Ege Erdil, who are leaving (or have left) Epoch AI to start this company.
I’m very concerned we might be witnessing another situation like Anthropic, where people with EA connections start a company that ultimately increases AI capabilities rather than safeguarding humanity’s future. But this time, we have a real opportunity for impact before it’s too late. I believe this project could potentially accelerate capabilities, increasing the odds of an existential catastrophe.
I’ve already reached out to the founders on X, but perhaps there are people more qualified than me who could speak with them about these concerns. In my tweets to them, I expressed worry about how this project could speed up AI development timelines, asked for a detailed write-up explaining why they believe this approach is net positive and low risk, and suggested an open debate on the EA Forum. While their vision of abundance sounds appealing, rushing toward it might increase the chance we never reach it due to misaligned systems.
I personally don’t have a lot of energy or capacity to work on this right now, nor do I think I have the required expertise, so I hope that others will pick up the slack. It’s important we approach this constructively and avoid attacking the three founders personally. The goal should be productive dialogue, not confrontation.
Does anyone have thoughts on how to productively engage with the Mechanize team? Or am I overreacting to what might actually be a beneficial project?
Two of the Mechanize co-founders were on Dwarkesh Patel’s podcast recently to discuss AGI timelines, among other things: https://youtu.be/WLBsUarvWTw
(Note: Dwarkesh Patel is listed on Mechanize’s website as an investor. I don’t know if this is disclosed in the podcast.)
I’ve only watched the first 45 minutes, but it seems like these two co-founders think AGI is decades away (e.g. one of them says 30-40 years). Dwarkesh seems to believe AGI will come much sooner and argues with them about this.
The situation doesn’t seem very similar to Anthropic. Regardless of whether you think Anthropic is good or bad (I think Anthropic is very good, but I work at Anthropic, so take that as you will), Anthropic was founded with the explicitly altruistic intention of making AI go well. Mechanize, by contrast, seems to mostly not be making any claims about altruistic motivations at all.
What concerns are there that you think the mechanize founders haven’t considered? I haven’t engaged with their work that much, but it seems like they have been part of the AI safety debate for years now, with plenty of discussion on this Forum and elsewhere (e.g. I can’t think of many AIS people that have been as active on this Forum as @Matthew_Barnett has been for the last few years). I feel like they have communicated their models and disagreements a (more than) fair amount already, so I don’t know what you would expect to change in further discussions?
You make a fair point, but what other tool do we have than our voice? I’ve read Matthew’s last post and skimmed through others. I see some concerning views, but I can also understand how he arrives at them. But what puzzles me often with some AI folks is the level of confidence needed to take such high-stakes actions. Why not err on the side of caution when the stakes are potentially so high?
Perhaps instead of trying to change someone’s moral views, we could just encourage taking moral uncertainty seriously? I personally lean towards hedonic act utilitarianism, yet I often default to ‘common sense morality’ because I’m just not certain enough.
I don’t have strong feelings on know how to best tackle this. I won’t have good answers to any questions. I’m just voicing concern and hoping others with more expertise might consider engaging constructively.
“AIs doing Forecasting”[1] has become a major part of the EA/AI/Epistemics discussion recently.
I think a logical extension of this is to expand the focus from forecasting to evaluation.
Forecasting typically asks questions like, “What will the GDP of the US be in 2026?”
Evaluation tackles partially-speculative assessments, such as:
“How much economic benefit did project X create?”
“How useful is blog post X?”
I’d hope that “evaluation” could function as “forecasting with extra steps.” The forecasting discipline excels at finding the best epistemic procedures for uncovering truth[2]. We want to maintain these procedures while applying them to more speculative questions.
Evaluation brings several additional considerations:
We need to identify which evaluations to run from a vast space of useful and practical options.
Evaluations often disrupt the social order, requiring skillful management.
Determining the best ways to “resolve” evaluations presents greater challenges than resolving forecast questions.
I’ve been interested in this area for 5+ years but struggled to draw attention to it—partly because it seems abstract, and partly because much of the necessary technology wasn’t quite ready.
We’re now at an exciting point where creating LLM apps for both forecasting and evaluation is becoming incredibly affordable. This might be a good time to spotlight this area.
There’s a curious gap now where we can, in theory, envision a world with sophisticated AI evaluation infrastructure, yet discussion of this remains limited. Fortunately, researchers and enthusiasts can fill this gap, one sentence at a time.
[1] As opposed to [Forecasting About AI], which is also common here. [2] Or at least, do as good a job as we can.
“Chief of Staff” models from a long-time Chief of Staff
I have served in Chief of Staff or CoS-like roles to three leaders of CEA (Zach, Ben and Max), and before joining CEA I was CoS to a member of the UK House of Lords. I wrote up some quick notes on how I think about such roles for some colleagues, and one of them suggested they might be useful to other Forum readers. So here you go:
Chief of Staff means many things to different people in different contexts, but the core of it in my mind is that many executive roles are too big to be done by one person (even allowing for a wider Executive or Leadership team, delegation to department leads, etc). Having (some parts of) the role split/shared between the principal and at least one other person increases the capacity and continuity of the exec function.
Broadly, I think of there being two ways to divide up these responsibilities (using CEO and CoS as stand-ins, but the same applies to other principal/deputy duos regardless of titles):
Split the CEO’s role into component parts and assign responsibility for each part to CEO or CoS
Example: CEO does fundraising; CoS does budgets
Advantages: focus, accountability
Share the CEO’s role with both CEO and CoS actively involved in each component part
Example: CEO speaks to funders based on materials prepared by CoS; CEO assigns team budget allocations which are implemented by CoS
Advantages: flex capacity, gatekeeping
Some things to note about these approaches:
In practice, it’s inevitably some combination of the two, but I think it’s really important to be intentional and explicit about what’s being split and what’s being shared
Failure to do this causes confusion, dropped balls, and duplication of effort
Sharing is especially valuable during the early phases of your collaboration because it facilitates context-swapping and model-building
I don’t think you’d ever want to get all the way or too far towards split, because then you functionally have one more department-lead-equivalent, and you lose a lot of the benefits in terms of flex capacity and especially continuity
Both approaches depend on trust, and maximising them depends on an unusually high degree of trust
CEO trusting CoS to act on their behalf
In turn, this depends on trusting their judgement, and in particular trusting their judgement of when it’s appropriate to act unilaterally and when it’s appropriate to get input/approval from CEO
Others trusting that CoS is empowered to and capable of acting on CEO’s behalf
Doesn’t work if CEO and CoS disagree or undermine each other’s decisions in view of others, or if others expect CoS decisions to be overturned by CEO
It being easier to burn credibility than to build it is something close to an iron law, which means CoS should tread carefully while establishing the bounds of their delegated authority
It’s not a seniority thing: an Executive Assistant having responsibility for scheduling is an example of splitting the role; a Managing Director doing copyedits for the CEO’s op-ed is an example of sharing the role
I don’t think the title “CoS” matters, but I do think maximising the benefits of both models requires the deputy to have a title that conveys that they both represent and can act unilaterally on behalf of the principal to some meaningful degree
Managing Director and Chief of Staff do this; Project Manager and Exec Assistant do not
The one thing that matters more for this than anything else is setting up an EA hub in a low cost of living area with decent visa options. The thing that matters second most is setting up group houses in high cost of living cities with good networking opportunities.
I used to feel so strongly about effective altruism. But my heart isn’t in it anymore.
I still care about the same old stuff I used to care about, like donating what I can to important charities and trying to pick the charities that are the most cost-effective. Or caring about animals and trying to figure out how to do right by them, even though I haven’t been able to sustain a vegan diet for more than a short time. And so on.
But there isn’t a community or a movement anymore where I want to talk about these sorts of things with people. That community and movement existed, at least in my local area and at least to a limited extent in some online spaces, from about 2015 to 2017 or 2018.
These are the reasons for my feelings about the effective altruist community/movement, especially over the last one or two years:
-The AGI thing has gotten completely out of hand. I wrote a brief post here about why I strongly disagree with near-term AGI predictions. I wrote a long comment here about how AGI’s takeover of effective altruism has left me disappointed, disturbed, and alienated. 80,000 Hours and Will MacAskill have both pivoted to focusing exclusively or almost exclusively on AGI. AGI talk has dominated the EA Forum for a while. It feels like AGI is what the movement is mostly about now, so now I just disagree with most of what effective altruism is about.
-The extent to which LessWrong culture has taken over or “colonized” effective altruism culture is such a bummer. I know there’s been at least a bit of overlap for a long time, but ten years ago it felt like effective altruism had its own, unique culture and nowadays it feels like the LessWrong culture has almost completely taken over. I have never felt good about LessWrong or “rationalism” and the more knowledge and experience of it I’ve gained, the more I’ve accumulated a sense of repugnance, horror, and anger toward that culture and ideology. I hate to see that become what effective altruism is like.
-The stories about sexual harassment are so disgusting. They’re really, really bad and crazy. And it’s so annoying how many comments you see on EA Forum posts about sexual harassment that make exhausting, unempathetic, arrogant, and frankly ridiculous statements, if not borderline incomprehensible in some cases. You see these stories of sexual harassment in the posts and you see evidence of the culture that enables sexual harassment in the comments. Very, very, very bad. Not my idea of a community I can wholeheartedly feel I belong to.
-Kind of a similar story with sexism, racism, and transphobia. The level of underreaction I’ve seen to instances of racism has been crazymaking. It’s similar to the comments under the posts about sexual harassment. You see people justifying or downplaying clearly immoral behaviour. It’s sickening.
-A lot of the response to the Nonlinear controversy was disheartening. It was disheartening to see how many people were eager to enable, justify, excuse, downplay, etc. bad behaviour. Sometimes aggressively, arrogantly, and rudely. It was also disillusioning to see how many people were so… easily fooled.
-Nobody talks normal in this community. At least not on this forum, in blogs, and on podcasts. I hate the LessWrong lingo. To the extent the EA Forum has its own distinct lingo, I probably hate that too. The lingo is great if you want to look smart. It’s not so great if you want other people to understand what the hell you are talking about. In a few cases, it seems like it might even be deliberate obscurantism. But mostly it’s just people making poor choices around communication and writing style and word choice, maybe for some good reasons, maybe for some bad reasons, but bad choices either way. I think it’s rare that writing with a more normal diction wouldn’t enhance people’s understanding of what you’re trying to say, even if you’re only trying to communicate with people who are steeped in the effective altruist niche. I don’t think the effective altruist sublanguage is serving good thinking or good communication.
-I see a lot of interesting conjecture elevated to the level of conventional wisdom. Someone in the EA or LessWrong or rationalist subculture writes a creative, original, evocative blog post or forum post and then it becomes a meme, and those memes end up taking on a lot of influence over the discourse. Some of these ideas are probably promising. Many of them probably contain at least a grain of truth or insight. But they become conventional wisdom without enough scrutiny. Just because an idea is “homegrown” it takes on the force of a scientific idea that’s been debated and tested in peer-reviewed journals for 20 years, or a widely held precept of academic philosophy. That seems just intellectually the wrong thing to do and also weirdly self-aggrandizing.
-An attitude I could call “EA exceptionalism”, where people assert that people involved in effective altruism are exceptionally smart, exceptionally wise, exceptionally good, exceptionally selfless, etc. Not just above the average or median (however you would measure that), but part of a rare elite and maybe even superior to everyone else in the world. I see no evidence this is true. (In these sorts of discussions, you also sometimes see the lame argument that effective altruism is definitionally the correct approach to life because effective altruism means doing the most good and if something isn’t doing the most good, then it isn’t EA. The obvious implication of this argument is that what’s called “EA” might not be true EA, and maybe true EA looks nothing like “EA”. So, this argument is not a defense of the self-identified “EA” movement or community or self-identified “EA” thought.)
-There is a dark undercurrent to some EA thought, along the lines of negative utilitarianism, anti-natalism, misanthropy, and pessimism. I think there is a risk of this promoting suicidal ideation because it basically is suicidal ideation.
-Too much of the discourse seems to revolve around how to control people’s behaviours or beliefs. It’s a bit too House of Cards. I recently read about the psychologist Kurt Lewin’s study on the most effective ways to convince women to use animal organs (e.g. kidneys, livers, hearts) in their cooking during meat shortages during World War II. He found that a less paternalistic approach that showed more respect for the women’s was more effective in getting them to incorporate animal organs into their cooking. The way I think about this is: you didn’t have to be manipulated to get to the point where you are in believing what you believe or caring this much about this issue. So, instead of thinking of how to best manipulate people, think about how you got to the point where you are and try to let people in on that in an honest, straightforward way. Not only is this probably more effective, it’s also more moral and shows more epistemic humility (you might be wrong about what you believe and that’s one reason not to try to manipulate people into believing it).
-A few more things but this list is already long enough.
Put all this together and the old stuff I cared about (charity effectiveness, giving what I can, expanding my moral circle) is lost in a mess of other stuff that is antithetical to what I value and what I believe. I’m not even sure the effective altruism movement should exist anymore. The world might be better off if it closed down shop. I don’t know. It could free up a lot of creativity and focus and time and resources to work on other things that might end up being better things to work on.
I still think there is value in the version of effective altruism I knew around 2015, when the primary focus was on global poverty and the secondary focus was on animal welfare, and AGI was on the margins. That version of effective altruism is so different from what exists today — which is mostly about AGI and has mostly been taken over by the rationalist subculture — that I have to consider those two different things. Maybe the old thing will find new life in some new form. I hope so.
Thanks for sharing this, while I personally believe the shift in focus on AI is justified (I also believe working on animal welfare is more impactful than global poverty), I can definitely sympathize with many of the other concerns you shared and agree with many of them (especially LessWrong lingo taking over, the underreaction to sexism/racism, and the Nonlinear controversy not being taken seriously enough). While I would completely understand in your situation if you don’t want to interact with the community anymore, I just want to share that I believe your voice is really important and I hope you continue to engage with EA! I wouldn’t want the movement to discourage anyone who shares its principles (like “let’s use our time and resources to help others the most”), but disagrees with how it’s being put into practice, from actively participating.
I don’t think people dropped the ball here really, people were struggling honestly to take accusations of bad behaviour seriously without getting into witch hunt dynamics.
Good point, I guess my lasting impression wasn’t entirely fair to how things played out. In any case, the most important part of my message is that I hope he doesn’t feels discouraged from actively participating in EA.
I’d distinguish here between the community and actual EA work. The community, and especially its leaders, have undoubtedly gotten more AI-focused (and/or publicly admittted to a degree of focus on AI they’ve always had) and rationalist-ish. But in terms of actual altruistic activity, I am very uncertain whether there is less money being spent by EAs on animal welfare or global health and development in 2025 than there was in 2015 or 2018. (I looked on Open Phil’s website and so far this year it seems well down from 2018 but also well up from 2015, but also 2 months isn’t much of a sample.) Not that that means your not allowed to feel sad about the loss of community, but I am not sure we are actually doing less good in these areas than we used to.
Yes, this seems similar to how I feel: I think the major donor(s) have re-prioritized, but am not so sure how many people have switched from other causes to AI. I think EA is more left to the grassroots now, and the forum has probably increased in importance. As long as the major donors don’t make the forum all about AI—then we have to create a new forum! But as donors change towards AI, the forum will inevitable see more AI content. Maybe some functions to “balance” the forum posts so one gets representative content across all cause areas? Much like they made it possible to separate out community posts?
The key objection I always have to starting new charities, as Charity Entrepreneurship used to focus on is that I feel is money usually not the bottleneck? I mean, we already have a ton of amazing ideas of how to use more funds, and if we found new ones, it may be very hard to reduce the uncertainty sufficiently to be able to make productive decisions. What do you think Ambitious Impact ?
A new organization can often compete for dollars that weren’t previously available to an EA org—such as government or non-EA foundation grants that are only open to certain subject areas.
Hot take, but political violence is bad and will continue to be bad in the foreseeable near-term future. That’s all I came here to say folks, have a great rest of your day.
Back in October 2024, I tried to test various LLM Chatbots with the question:
”Is there a way to convert a correlation to a probability while preserving the relationship 0 = 1/n?”
Years ago, I came up with an unpublished formula that does just that:
p(r) = (n^r * (r + 1)) / (2^r * n)
So I was curious if they could figure it out. Alas, back in October 2024, they all made up formulas that didn’t work.
Yesterday, I tried the same question on ChatGPT and, while it didn’t get it quite right, it came, very, very close. So, I modified the question to be more specific:
”Is there a way to convert a correlation to a probability while preserving the relationships 1 = 1, 0 = 1/n, and −1 = 0?”
This time, it came up with a formula that was different and simpler than my own, and… it actually works!
I tried this same prompt with a bunch of different LLM Chatbots and got the following:
Correct on the first prompt:
GPT4o, Claude 3.7
Correct after explaining that I wanted a non-linear, monotonic function:
Gemini 2.5 Pro, Grok 3
Failed:
DeepSeek-V3, Mistral Le Chat, QwenMax2.5, Llama 4
Took too long thinking and I stopped it:
DeepSeek-R1, QwQ
All the correct models got some variation of:
p(r) = ((r + 1) / 2)^log2(n)
This is notably simpler and arguably more elegant than my earlier formula. It also, unlike my old formula, has an easy to derive inverse function.
So yeah. AI is now better than me at coming up with original math.
Per Bloomberg, the Trump administration is considering restricting the equivalency determination for 501(c)3s as early as Tuesday. The equivalency determination allows for 501(c)3s to regrant money to foreign, non-tax-exempt organisations while maintaining tax-exempt status, so long as an attorney or tax practitioner claims the organisation is equivalent to a local tax-exempt one.
I’m not an expert on this, but it sounds really bad. I guess it remains to be seen if they go through with it.
Regardless, the administration is allegedly also preparing to directly strip environmental and political (i.e. groups he doesn’t like, not necessarily just any policy org) non-profits of their tax exempt status. In the past week, he’s also floated trying to rescind the tax exempt status of Harvard. From what I understand, such an Executive Order is illegal under U.S. law (to whatever extent that matters anymore), unless Trump instructs the State Department to designate them foreign terrorist organisations, at which point all their funds are frozen too.
Is anyone in the U.S. savvy with how to deduct from your taxes the value of stocks which have been donated to eligible charities? The stocks have been held for decades with a very high value and capital gains. Would love help as my tax guy hasn’t seen this before.
You can calculate the “Fair Market Value” of the stock(s) you donated by averaging the highest and lowest price of that stock on the day you donated it. I used this page to find that, but you can replace “MSFT” in the URL with whatever stock it is you sold.
At Risk of violating @Linch’s principle “Assume by default that if something is missing in EA, nobody else is going to step up.”, I think it would be valuable to have a well researched estimate of the counterfactual value of getting investment from different investors (whether for profit or donors).
For example in global health, we could make GiveWell the baseline, as I doubt whether there is a ll funding source where switching as less impact, as the money will only ever be shifted from something slightly less effective. For example if my organisation received funding from GiveWell, we might only make slightly better use of that money than where it would otherwise have gone, and we’re not going to be increasing the overall donor pool either.
Who knows, for-profit investment dollars could be 10x −100x more counterfactually impactful than GiveWell, which could mean a for-profit company trying to do something good could plausibly be 10-100x less effective than a charity and still doing as much counterfactual good overall? Or is this a stretch?
This would be hard to estimate but doable, and must have been done at least on a casual scale by some people.
Examples ( and random guesses) of counterfactual comparisons of the value of each dollar given by a particular source might be something like....
1. GiveWell 1x
2. Gates Foundation 3x
3. Individual donors NEW donations 10x
4. Indivudal donors SHIFTING donations. 5x
5. Non EA-Aligned foundations 8x
6. Climate funding 5x
7. For-profit investors. 20x
Or this might be barking up the wrong tree, not sure (and I have mentioned it before)
I’m admittedly a bit more confused by your fleshed-out example with random guesses than I was when I read your opening sentence, as it went in a different direction than I expected (using multipliers instead of subtracting the value of the next-best alternative use of funds), so maybe we’re thinking about different things. I also didn’t understand what you meant by this when I tried to flesh it out myself with some (made-up) numbers:
If it helps, GiveWell has a spreadsheet summarising their analysis of the counterfactual value of other actors’ spending. From there and a bit of arithmetic, you can figure out their estimates of the value generated by $1 million in spending from the following sources, expressed in units of doubling consumption for one person for a year:
Government social security spending: 2,625 units
Government education spending: 2,834 units
Donating to GiveDirectly’s cash transfer program (1x cash for reference): 3,355 units
Government spending on deworming: 3,525 units
Government spending on malaria, VAS, and immunization programs: 5,057 units (this is also what GW uses as their counterfactual value by domestic governments in assessing SMC)
Government health spending: 5,639 units (this is also GW’s assumed value generated by philanthropic actors’s spending on VAS and vitamin A capsule procurement in other countries)
Gavi spending (US allocations to Gavi and Maternal and Child Health): 6,952 units
Global Fund (calculations here): 15,433 units
HKI spending on vitamin A supplementation in Kenya, before final adjustments: ~19,000 units
GW’s 10x cash bar for reference: 33,545 units
AMF spending on ITN distribution in DRC, before final adjustments: ~36,800 units
Open Phil’s current 2,100x bar: very naively ~70,000 units (21x GiveDirectly to oversimplify, using OP’s 100x ≈ 1x cash to GD)
Suppose there’s an org (“EffectiveHealth”) that could create 50k units worth of benefits given $1M in funding. If it came from GW with their 33.5k units counterfactual value, then a grant to EffectiveHealth from GW would be creating 16.5k more units of benefits than otherwise, while if it came from domestic govt spending (7.5x lower value at 5k per $1M) it’d create 45k more units of benefits. A hypothetical for-profit investor whose funding generates 10x less good than domestic govt spending (500 units) would get you not 10 x 45k = 450k more units of benefits, but 49.5k units. And if EffectiveHealth got funding from OP it would actually be net-negative by −20k units.
(maybe you meant something completely different, in which case apologies!)
I think this kind of investigation would be valuable, but I’m not sure what concrete questions you’d imagine someone answering to figure this out.
Questions would just be finding out what the cost-effectiveness of money from current individual donors, philanthropies, Gates Foundation was, then calculating the difference if it went to the most cost-effective rganisations.
https://www.gap-map.org/?sort=rank&fields=biosecurity
I think this is cool! It shows gaps in capabilities so that people can see what needs to be worked on.
Reflections on “Status Handcuffs” over one’s career
(This was edited using Claude)
Having too much professional success early on can ironically restrict you later on. People typically are hesitant to go down in status when choosing their next job. This can easily mean that “staying in career limbo” can be higher-status than actually working. At least when you’re in career limbo, you have a potential excuse.
This makes it difficult to change careers. It’s very awkward to go from “manager of a small team” to “intern,” but that can be necessary if you want to learn a new domain, for instance.
The EA Community Context
In the EA community, some aspects of this are tricky. The funders very much want to attract new and exciting talent. But this means that the older talent is in an awkward position.
The most successful get to take advantage of the influx of talent, with more senior leadership positions. But there aren’t too many of these positions to go around. It can feel weird to work on the same level or under someone more junior than yourself.
Pragmatically, I think many of the old folks around EA are either doing very well, or are kind of lost/exploring other avenues. Other areas allow people to have more reputable positions, but these are typically not very EA/effective areas. Often E2G isn’t very high-status in these clusters, so I think a lot of these people just stop doing much effective work.
Similar Patterns in Other Fields
This reminds me of law firms, which are known to have “up or out” cultures. I imagine some of this acts as a formal way to prevent this status challenge—people who don’t highly succeed get fully kicked out, in part because they might get bitter if their career gets curtailed. An increasingly narrow set of lawyers continue on the Partner track.
I’m also used to hearing about power struggles for senior managers close to retirement at big companies, where there’s a similar struggle. There’s a large cluster of highly experienced people who have stopped being strong enough to stay at the highest levels of management. Typically these people stay too long, then completely leave. There can be few paths to gracefully go down a level or two while saving face and continuing to provide some amount of valuable work.
But around EA and a lot of tech, I think this pattern can happen much sooner—like when people are in the age range of 22 to 35. It’s more subtle, but it still happens.
Finding Solutions
I’m very curious if it’s feasible for some people to find solutions to this. One extreme would be, “Person X was incredibly successful 10 years ago. But that success has faded, and now the only useful thing they could do is office cleaning work. So now they do office cleaning work. And we’ve all found a way to make peace with this.”
Traditionally, in Western culture, such an outcome would be seen as highly shameful. But in theory, being able to find peace and satisfaction from something often seen as shameful for (what I think of as overall-unfortunate) reasons could be considered a highly respectable thing to do.
Perhaps there could be a world where [valuable but low-status] activities are identified, discussed, and later turned to be high-status.
The EA Ideal vs. Reality
Back to EA. In theory, EAs are people who try to maximize their expected impact. In practice, EA is a specific ideology that typically has a limited impact on people (at least compared to strong Religious groups, for instance). I think that the EA scene has demonstrated success at getting people to adjust careers (in circumstances where it’s fairly cheap and/or favorable to do so), and has created an ecosystem that rewards people for certain EA behaviors. But at the same time, people typically feature with a great deal of non-EA constraints that must be continually satisfied for them to be productive; money, family, stability, health, status, etc.
Personal Reflection
Personally, every few months I really wonder what might make sense for me. I’d love to be the kind of person who would be psychologically okay doing the lowest-status work for the youngest or lowest-status people. At the same time, knowing myself, I’m nervous that taking a very low-status position might cause some of my mind to feel resentment and burnout. I’ll continue to reflect on this.
I’ve just ran into this, so excuse a bit of grave digging. As someone who has entered the EA community with prior career experience I disagree with your premise
“It’s very awkward to go from “manager of a small team” to “intern,” but that can be necessary if you want to learn a new domain, for instance.”
To me this kind of situation just shouldn’t happen. It’s not a question of status, it’s a question of inefficiency. If I have managerial experience and the organization I’d be joining can only offer me the exact same job they’d be offering to a fresh grad, then they are simply wasting my potential. I’d be better off at a place which can appreciate what I bring and the organization would be better off with someone who has a fresher mind and less tempting alternatives.
IMO the problem is not with the fact that people are unwilling to take a step down. The problem is with EA orgs unwilling or unable to leverage the transferrable skills of experienced professionals, forcing them into entry-level positions instead.
I agree with you. I think in EA this is especially the case because much of the community-building work is focused on universities/students, and because of the titling issue someone else mentioned. I don’t think someone fresh out of uni should be head of anything, wah. But the EA movement is young and was started by young people, so it’ll take a while for career-long progression funnels to develop organically.
Thanks for writing this, this is also something I have been thinking about and you’ve expressed it more eloquently.
One thing I have thought might be useful is at times showing restraint with job titling. I’ve observed cases where people have had a title for example Director in a small org or growing org, and in a larger org this role might be a coordinator, lead, admin.
I’ve thought at times this doesn’t necessarily set people up for long term career success as the logical career step in terms of skills and growth, or a career shift, often is associated with a lower sounding title. Which I think decreases motivation to take on these roles.
At the same time I have seen people, including myself, take a decrease in salary and title, in order to shift careers and move forward.
A related issue I have actually encountered is something like “but you seem overqualified for this role we are hiring for”. Even if previously successful people wanted to take a “less prestigious” role, they might encounter real problems in doing so. I hope the EA eco system might have some immunity to this though—as hopefully the mission alignment will be strong enough evidence of why such a person might show interest in a “lower” role.
As a single data point: seconded. I’ve explicitly been asked by interviewers (in a job interview) why I left a “higher title job” for a “lower title job,” with the implication that it needed some special justification. I suspect there have also been multiple times in which someone looking at my resume saw that transition, made an assumption about it, and choose to reject me. (although this probably happens with non-EA jobs more often than EA jobs, as the “lower title role” was with a well-known EA organization)
Good point. And sorry you had to go through that, it sounds quite frustrating.
Pragmatically, I think many of the old folks around EA are either doing very well, or are kind of lost/exploring other avenues. Other areas allow people to have more reputable positions, but these are typically not very EA/effective areas. Often E2G isn’t very high-status in these clusters, so I think a lot of these people just stop doing much effective work.
I haven’t really noticed this happening very much empirically, but I do think the effect you are talking about is quite intuitive. Have you seen many cases of this that you’re confident are correct (e.g. they aren’t lost for other reasons like working on non-public projects or being burnt out)? No need to mention specific names.
In theory, EAs are people who try to maximize their expected impact. In practice, EA is a light ideology that typically has a limited impact on people. I think that the EA scene has demonstrated success at getting people to adjust careers (in circumstances where it’s fairly cheap and/or favorable to do so)
This seems incorrect to me, in absolute terms. By the standards of ~any social movement, EAs are very sacrificial and focused on increasing their impact. I suspect you somewhat underrate how rare it is outside of EA to be highly committed to ~any non-self-serving principles seriously enough to sacrifice significant income and change careers, particularly in new institutions/movements.
I’m sure that very few of these are explained by “non-public projects”.
I’m unsure about burnout. I’m not sure where the line is between “can’t identify high-status work to do” and burnout. I expect that the two are highly correlated. My guess is that they don’t literally think of it as “I’m low status now”, instead I’d expect them to feel emotions like resentment / anger / depression. But I’d also expect that if we could change the status lever, other negative feelings would go away. (I think that status is a big deal for people! Like, status means you have a good career, get to be around people you like, etc)
> I suspect you somewhat underrate how rare it is outside of EA to be highly committed to ~any non-self-serving principles seriously enough to sacrifice significant income and change careers.
I suspect we might have different ideologies in mind to compare to, and correspondingly, that we’re not disagreeing much.
I think that a lot of recently-popular movements like BLM or even MAGA didn’t change the average lifestyle of the median participant much at all, though much of this is because they are far larger.
But religious groups are far more intense, for example. Or maybe take dedicated professional specialties like ballet or elite music, which can require intense sacrifices.
Announcing PauseCon, the PauseAI conference.
Three days of workshops, panels, and discussions, culminating in our biggest protest to date.
Twitter: https://x.com/PauseAI/status/1915773746725474581
Apply now: https://pausecon.org
The recent rise of AI Factory/Neocloud companies like CoreWeave, Lambda and Crusoe strikes me as feverish and financially unsound. These companies are highly overleveraged, offering GPU access as a commodity to a monopsony. Spending vast amounts of capex on a product that must be highly substitutive to compete with hyperscalers on cost strikes me as an unsustainable business model in the long term. The association of these companies with the ‘AI Boom’ could cause collateral reputation damage to more reputable firms if these Neoclouds go belly up.
And, if you’re in a race-to-the-bottom price war on margins, fewer resources are devoted to risk management and considerate deployment. A recent SemiAnalysis article confirms the “long tail” of these companies that lack even basic ISO27001 or SOC 2 certs.
I should say that the Sovereign Neocloud model seems to alleviate both the financial and x-risk concerns; however, that’s mostly relevant outside the US and China.
I’d be excited to see 1-2 opportunistic EA-rationalist types looking into where marginal deregulation is a bottleneck to progress on x-risk/GHW, circulating 1-pagers among experts in these areas, and then pushing the ideas to DOGE/Mercatus/Executive Branch. I’m thinking things like clinical trials requirements for vaccines, UV light, anti-trust issues facing companies collaborating on safety and security, maybe housing (though I’m not sure which are bottlenecked by federal action). For most of these there’s downside risk if the message is low fidelity, the issue becomes polarized, or priorities are poorly set, hence collaborating with experts. I doubt there’s that much useful stuff to be done here, but marginal deregulation looks very easy right now and looks good to strike while the iron is hot.
There’s this ACX post (that I only skimmed and don’t have strong opinions about) which mostly seems to do this, minus the “pushing” part.
Thought these quotes from Holden’s old (2011) GW blog posts were thought-provoking, unsure to what extent I agree. In In defense of the streetlight effect he argued that
(I appreciate the bolded part, especially as something baked into GW’s approach and top recs by $ moved.)
That last link is to The most important problem may not be the best charitable cause. Quote that caught my eye:
LLMs seem more like low-level tools to me than direct human interfaces.
Current models suffer from hallucinations, sycophancy, and numerous errors, but can be extremely useful when integrated into systems with redundancy and verification.
We’re in a strange stage now where LLMs are powerful enough to be useful, but too expensive/slow to have rich scaffolding and redundancy. So we bring this error-prone low-level tool straight to the user, for the moment, while waiting for the technology to improve.
Using today’s LLM interfaces feels like writing SQL commands directly instead of using a polished web application. It’s functional if that’s all you have, but it’s probably temporary.
Imagine what might happen if/when LLMs are 1000x faster and cheaper.
Then, answering a question might involve:
Running ~100 parallel LLM calls with various models and prompts
Using aggregation layers to compare responses and resolve contradictions
Identifying subtasks and handling them with specialized LLM batches and other software
Big picture, I think researchers might focus less on making sure any one LLM call is great, and more that these broader setups can work effectively.
(I realize this has some similarities to Mixture of Experts)
I guess orgs need to be more careful about who they hire as forecasting/evals researchers in light of a recently announced startup.
Sometimes things happen, but three people at the same org...
This is also a massive burning of the commons. It is valuable for forecasting/evals orgs to be able to hire people with a diversity of viewpoints in order to counter bias. It is valuable for folks to be able to share information freely with folks at such forecasting orgs without having to worry about them going off and doing something like this.
However, this only works if those less worried about AI risks who join such a collaboration don’t use the knowledge they gain to cash in on the AI boom in an acceleratory way. Doing so undermines the very point of such a project, namely, to try to make AI go well. Doing so is incredibly damaging to trust within the community.
Now let’s suppose you’re an x-risk funder considering whether to fund their previous org. This org does really high-quality work, but the argument for them being net-positive is now significantly weaker. This is quite likely to make finding future funding harder for them.
This is less about attacking those three folks and more just noting that we need to strive to avoid situations where things like this happen in the first place. This requires us to be more careful in terms of who gets hired.
There’s been some discussions on the EA forum along the lines of “why do we care about value alignment shouldn’t we just hire who can best do the job”. My answer to that is that it’s myopic to only consider what happens whilst they’re working for you. Hiring someone or offering them an opportunity empowers them, you need to consider whether they’re someone who you want to empower[1].
Admittedly, this isn’t quite the same as value alignment. Suppose someone were diligent, honest, wise and responsible. You might want to empower them even if their views were extremely different from yours. Stronger: even if their views were the opposite in many ways. But in the absence of this, value alignment matters.
If you only hire people who you believe are intellectually committed to short AGI timelines (and who won’t change their minds given exposure to new evidence and analysis) to work in AGI forecasting, how can you do good AGI forecasting?
One of the co-founders of Mechanize, who formerly worked at Epoch AI, says he thinks AGI is 30 to 40 years away. That was in this video from a few weeks ago on Epoch AI’s YouTube channel.
He and one of his co-founders at Mechanize were recently on Dwarkesh Patel’s podcast (note: Dwarkesh Patel is an investor in Mechanize) and I didn’t watch all of it but it seemed like they were both arguing for longer AGI timelines than Dwarkesh believes in.
I also disagree with the shortest AGI timelines and found it refreshing that within the bubble of people who are fixated on near-term AGI, at least a few people expressed a different view.
I think if you restrict who you hire to do AGI forecasting based on strong agreement with a predetermined set of views, such as short AGI timelines and views on AGI alignment and safety, then you will just produce forecasts that re-state the views you already decided were the correct ones while you were hiring.
I wasn’t suggesting only hiring people who believe in short-timelines. I believe that my original post adequately lays out my position, but if any points are ambiguous, feel free to request clarification.
I don’t know how Epoch AI can both “hire people with a diversity of viewpoints in order to counter bias” and ensure that your former employees won’t try to “cash in on the AI boom in an acceleratory way”. These seem like incompatible goals.
I think Epoch has to either:
Accept that people have different views and will have different ideas about what actions are ethical, e.g., they may view creating an AI startup focused on automating labour as helpful to the world and benign
or
Only hire people who believe in short AGI timelines and high AGI risk and, as a result, bias its forecasts towards those conclusions
Is there a third option?
Presumably there are at least some people who have long timelines, but also believe in high risk and don’t want to speed things up. Or people who are unsure about timelines, but think risk is high whenever it happens. Or people (like me) who think X-risk is low* and timelines very unclear, but even a very low X-risk is very bad. (By very low, I mean like at least 1 in 1000, not 1 in 1x10^17 or something. I agree it is probably bad to use expected value reasoning with probabilities as low as that.)
I think you are pointing at a real tension though. But maybe try to see it a bit from the point of view of people who think X-risk is real enough and raised enough by acceleration that acceleration is bad. It’s hardly going to escape their notice that projects at least somewhat framed as reducing X-risk often end up pushing capabilities forward. They don’t have to be raging dogmatists to worry about this happening again, and it’s reasonable for them to balance this risk against risks of echo chambers when hiring people or funding projects.
*I’m less surely merely catastrophic biorisk from human misuse is low sadly.
Why don’t we ask ChatGPT? (In case you’re wondering, I’ve read every word of this answer and I fully endorse it, though I think there are better analogies that the journalism example ChatGPT used).
Hopefully, this clarifies a possible third option (one that my original answer was pointing at).
So, you want to try to lock in AI forecasters to onerous and probably illegal contracts that forbid them from founding an AI startup after leaving the forecasting organization? Who would sign such a contract? This is even worse than only hiring people who are intellectually pre-committed to certain AI forecasts. Because it goes beyond a verbal affirmation of their beliefs to actually attempting to legally force them to comply with the (putative) ethical implications of certain AI forecasts.
If the suggestion is simply promoting “social norms” against starting AI startups, well, that social norm already exists to some extent in this community, as evidenced by the response on the EA Forum. But if the norm is too weak, it won’t prevent the undesired outcome (the creation of an AI startup), and if the norm is too strong, I don’t see how it doesn’t end up selecting forecasters for intellectual conformity. Because non-conformists would not want to go along with such a norm (just like they wouldn’t want to sign a contract telling them what they can and can’t do after they leave the forecasting company).
I agree that we need to be careful about who we are empowering.
“Value alignment” is one of those terms which has different meanings to different people. For example, the top hit I got on Google for “effective altruism value alignment” was a ConcernedEAs post which may not reflect what you mean by the term. Without knowing exactly what you mean, I’d hazard a guess that some facets of value alignment are pretty relevant to mitigating this kind of risk, and other facets are not so important. Moreover, I think some of the key factors are less cognitive or philosophical than emotional or motivational (e.g., a strong attraction toward money will increase the risk of defecting, a lack of self-awareness increases the risk of motivated reasoning toward goals one has in a sense repressed).
So, I think it would be helpful for orgs to consider what elements of “value alignment” are of particular importance here, as well as what other risk or protective factors might exist outside of value alignment, and focus on those specific things.
Agreed. “Value alignment” is a simplified framing.
Why not attack them? They defected. They did a really bad thing.
Also, it is worrying if the optimists easily find financial opportunities that depend on them not changing their minds. Even if they are honest and have the best of intentions, the disparity in returns to optimism is epistemically toxic.
I’d like to suggest a little bit more clarity here. The phrases you use refer to some knowledge that isn’t explicitly stated here. “in light of a recently announced startup” and “three people at the same org” make sense to someone who already knows the context of what you are writing about, but it is confusing to a reader who doesn’t have the same background knowledge that you do.
Once upon a time, some people were arguing that AI might kill everyone, and EA resources should address that problem instead of fighting Malaria. So OpenPhil poured millions of dollars into orgs such as EpochAI (they got 9 million). Now 3 people from EpochAI created a startup to provide training data to help AI replace human workers. Some people are worried that this startup increases AI capabilities, and therefore increases the chance that AI will kill everyone.
100 percent agree. I dont understand the entire post because I don’t know the context. I don’t think alluding to something helps, better to say it explicitly.
I tend to agree; better to be explicit especially as the information is public knowledge anyway.
It refers to this: https://forum.effectivealtruism.org/posts/HqKnreqC3EFF9YcEs/
I’ve spent some time in the last few months outlining a few epistemics/AI/EA projects I think could be useful.
Link here.
I’m not sure how to best write about these on the EA Forum / LessWrong. They feel too technical and speculative to gain much visibility.
But I’m happy for people interested in the area to see them. Like with all things, I’m eager for feedback.
Here’s a brief summary of them, written by Claude.
---
1. AI-Assisted Auditing
A system where AI agents audit humans or AI systems, particularly for organizations involved in AI development. This could provide transparency about data usage, ensure legal compliance, flag dangerous procedures, and detect corruption while maintaining necessary privacy.
2. Consistency Evaluations for Estimation AI Agents
A testing framework that evaluates AI forecasting systems by measuring several types of consistency rather than just accuracy, enabling better comparison and improvement of prediction models. It’s suggested to start with simple test sets and progress to adversarial testing methods that can identify subtle inconsistencies across domains.
3. AI for Epistemic Impact Estimation
An AI tool that quantifies the value of information based on how it improves beliefs for specific AIs. It’s suggested to begin with narrow domains and metrics, then expand to comprehensive tools that can guide research prioritization, value information contributions, and optimize information-seeking strategies.
4. Multi-AI-Critic Document Comments & Analysis
A system similar to “Google Docs comments” but with specialized AI agents that analyze documents for logical errors, provide enrichment, and offer suggestions. This could feature a repository of different optional open-source agents for specific tasks like spot-checking arguments, flagging logical errors, and providing information enrichment.
5. Rapid Prediction Games for RL
Specialized environments where AI agents trade or compete on predictions through market mechanisms, distinguishing between Information Producers and Consumers. The system aims to both evaluate AI capabilities and provide a framework for training better forecasting agents through rapid feedback cycles.
6. Analytics on Private AI Data
A project where government or researcher AI agents get access to private logs/data from AI companies to analyze questions like: How often did LLMs lie or misrepresent information? Did LLMs show bias toward encouraging user trust? Did LLMs employ questionable tactics for user retention? This addresses the limitation that researchers currently lack access to actual use logs.
7. Prediction Market Key Analytics Database
A comprehensive analytics system for prediction markets that tracks question value, difficulty, correlation with other questions, and forecaster performance metrics. This would help identify which questions are most valuable to specific stakeholders and how questions relate to real-world variables.
8. LLM Resolver Agents
A system for resolving forecasting questions using AI agents with built-in desiderata including: triggering experiments at specific future points, deterministic randomness methods, specified LLM usage, verifiability, auditability, proper caching/storing, and sensitivity analysis.
9. AI-Organized Information Hubs
A platform optimized for AI readers and writers where systems, experts, and organizations can contribute information that is scored and filtered for usefulness. Features would include privacy levels, payment proportional to information value, and integration of multiple file types.
I’m not sure how to word this properly, and I’m uncertain about the best approach to this issue, but I feel it’s important to get this take out there.
Yesterday, Mechanize was announced, a startup focused on developing virtual work environments, benchmarks, and training data to fully automate the economy. The founders include Matthew Barnett, Tamay Besiroglu, and Ege Erdil, who are leaving (or have left) Epoch AI to start this company.
I’m very concerned we might be witnessing another situation like Anthropic, where people with EA connections start a company that ultimately increases AI capabilities rather than safeguarding humanity’s future. But this time, we have a real opportunity for impact before it’s too late. I believe this project could potentially accelerate capabilities, increasing the odds of an existential catastrophe.
I’ve already reached out to the founders on X, but perhaps there are people more qualified than me who could speak with them about these concerns. In my tweets to them, I expressed worry about how this project could speed up AI development timelines, asked for a detailed write-up explaining why they believe this approach is net positive and low risk, and suggested an open debate on the EA Forum. While their vision of abundance sounds appealing, rushing toward it might increase the chance we never reach it due to misaligned systems.
I personally don’t have a lot of energy or capacity to work on this right now, nor do I think I have the required expertise, so I hope that others will pick up the slack. It’s important we approach this constructively and avoid attacking the three founders personally. The goal should be productive dialogue, not confrontation.
Does anyone have thoughts on how to productively engage with the Mechanize team? Or am I overreacting to what might actually be a beneficial project?
Two of the Mechanize co-founders were on Dwarkesh Patel’s podcast recently to discuss AGI timelines, among other things: https://youtu.be/WLBsUarvWTw
(Note: Dwarkesh Patel is listed on Mechanize’s website as an investor. I don’t know if this is disclosed in the podcast.)
I’ve only watched the first 45 minutes, but it seems like these two co-founders think AGI is decades away (e.g. one of them says 30-40 years). Dwarkesh seems to believe AGI will come much sooner and argues with them about this.
The situation doesn’t seem very similar to Anthropic. Regardless of whether you think Anthropic is good or bad (I think Anthropic is very good, but I work at Anthropic, so take that as you will), Anthropic was founded with the explicitly altruistic intention of making AI go well. Mechanize, by contrast, seems to mostly not be making any claims about altruistic motivations at all.
You’re right that this is an important distinction to make.
What concerns are there that you think the mechanize founders haven’t considered? I haven’t engaged with their work that much, but it seems like they have been part of the AI safety debate for years now, with plenty of discussion on this Forum and elsewhere (e.g. I can’t think of many AIS people that have been as active on this Forum as @Matthew_Barnett has been for the last few years). I feel like they have communicated their models and disagreements a (more than) fair amount already, so I don’t know what you would expect to change in further discussions?
You make a fair point, but what other tool do we have than our voice? I’ve read Matthew’s last post and skimmed through others. I see some concerning views, but I can also understand how he arrives at them. But what puzzles me often with some AI folks is the level of confidence needed to take such high-stakes actions. Why not err on the side of caution when the stakes are potentially so high?
Perhaps instead of trying to change someone’s moral views, we could just encourage taking moral uncertainty seriously? I personally lean towards hedonic act utilitarianism, yet I often default to ‘common sense morality’ because I’m just not certain enough.
I don’t have strong feelings on know how to best tackle this. I won’t have good answers to any questions. I’m just voicing concern and hoping others with more expertise might consider engaging constructively.
“AIs doing Forecasting”[1] has become a major part of the EA/AI/Epistemics discussion recently.
I think a logical extension of this is to expand the focus from forecasting to evaluation.
Forecasting typically asks questions like, “What will the GDP of the US be in 2026?”
Evaluation tackles partially-speculative assessments, such as:
“How much economic benefit did project X create?”
“How useful is blog post X?”
I’d hope that “evaluation” could function as “forecasting with extra steps.” The forecasting discipline excels at finding the best epistemic procedures for uncovering truth[2]. We want to maintain these procedures while applying them to more speculative questions.
Evaluation brings several additional considerations:
We need to identify which evaluations to run from a vast space of useful and practical options.
Evaluations often disrupt the social order, requiring skillful management.
Determining the best ways to “resolve” evaluations presents greater challenges than resolving forecast questions.
I’ve been interested in this area for 5+ years but struggled to draw attention to it—partly because it seems abstract, and partly because much of the necessary technology wasn’t quite ready.
We’re now at an exciting point where creating LLM apps for both forecasting and evaluation is becoming incredibly affordable. This might be a good time to spotlight this area.
There’s a curious gap now where we can, in theory, envision a world with sophisticated AI evaluation infrastructure, yet discussion of this remains limited. Fortunately, researchers and enthusiasts can fill this gap, one sentence at a time.
[1] As opposed to [Forecasting About AI], which is also common here.
[2] Or at least, do as good a job as we can.
“Chief of Staff” models from a long-time Chief of Staff
I have served in Chief of Staff or CoS-like roles to three leaders of CEA (Zach, Ben and Max), and before joining CEA I was CoS to a member of the UK House of Lords. I wrote up some quick notes on how I think about such roles for some colleagues, and one of them suggested they might be useful to other Forum readers. So here you go:
Chief of Staff means many things to different people in different contexts, but the core of it in my mind is that many executive roles are too big to be done by one person (even allowing for a wider Executive or Leadership team, delegation to department leads, etc). Having (some parts of) the role split/shared between the principal and at least one other person increases the capacity and continuity of the exec function.
Broadly, I think of there being two ways to divide up these responsibilities (using CEO and CoS as stand-ins, but the same applies to other principal/deputy duos regardless of titles):
Split the CEO’s role into component parts and assign responsibility for each part to CEO or CoS
Example: CEO does fundraising; CoS does budgets
Advantages: focus, accountability
Share the CEO’s role with both CEO and CoS actively involved in each component part
Example: CEO speaks to funders based on materials prepared by CoS; CEO assigns team budget allocations which are implemented by CoS
Advantages: flex capacity, gatekeeping
Some things to note about these approaches:
In practice, it’s inevitably some combination of the two, but I think it’s really important to be intentional and explicit about what’s being split and what’s being shared
Failure to do this causes confusion, dropped balls, and duplication of effort
Sharing is especially valuable during the early phases of your collaboration because it facilitates context-swapping and model-building
I don’t think you’d ever want to get all the way or too far towards split, because then you functionally have one more department-lead-equivalent, and you lose a lot of the benefits in terms of flex capacity and especially continuity
Both approaches depend on trust, and maximising them depends on an unusually high degree of trust
CEO trusting CoS to act on their behalf
In turn, this depends on trusting their judgement, and in particular trusting their judgement of when it’s appropriate to act unilaterally and when it’s appropriate to get input/approval from CEO
Others trusting that CoS is empowered to and capable of acting on CEO’s behalf
Doesn’t work if CEO and CoS disagree or undermine each other’s decisions in view of others, or if others expect CoS decisions to be overturned by CEO
It being easier to burn credibility than to build it is something close to an iron law, which means CoS should tread carefully while establishing the bounds of their delegated authority
It’s not a seniority thing: an Executive Assistant having responsibility for scheduling is an example of splitting the role; a Managing Director doing copyedits for the CEO’s op-ed is an example of sharing the role
I don’t think the title “CoS” matters, but I do think maximising the benefits of both models requires the deputy to have a title that conveys that they both represent and can act unilaterally on behalf of the principal to some meaningful degree
Managing Director and Chief of Staff do this; Project Manager and Exec Assistant do not
In ~2014, one major topic among effective altruists was “how to live for cheap.”
There wasn’t much funding, so it was understood that a major task for doing good work was finding a way to live with little money.
Money gradually increased, peaking with FTX in 2022.
Now I think it might be time to bring back some of the discussions about living cheaply.
Related recent discussion here: https://forum.effectivealtruism.org/posts/eMWsKbLFMy7ABdCLw/alfredo-parra-s-quick-takes?commentId=vo3jDMAhFFd2XQgYa
The one thing that matters more for this than anything else is setting up an EA hub in a low cost of living area with decent visa options. The thing that matters second most is setting up group houses in high cost of living cities with good networking opportunities.
I used to feel so strongly about effective altruism. But my heart isn’t in it anymore.
I still care about the same old stuff I used to care about, like donating what I can to important charities and trying to pick the charities that are the most cost-effective. Or caring about animals and trying to figure out how to do right by them, even though I haven’t been able to sustain a vegan diet for more than a short time. And so on.
But there isn’t a community or a movement anymore where I want to talk about these sorts of things with people. That community and movement existed, at least in my local area and at least to a limited extent in some online spaces, from about 2015 to 2017 or 2018.
These are the reasons for my feelings about the effective altruist community/movement, especially over the last one or two years:
-The AGI thing has gotten completely out of hand. I wrote a brief post here about why I strongly disagree with near-term AGI predictions. I wrote a long comment here about how AGI’s takeover of effective altruism has left me disappointed, disturbed, and alienated. 80,000 Hours and Will MacAskill have both pivoted to focusing exclusively or almost exclusively on AGI. AGI talk has dominated the EA Forum for a while. It feels like AGI is what the movement is mostly about now, so now I just disagree with most of what effective altruism is about.
-The extent to which LessWrong culture has taken over or “colonized” effective altruism culture is such a bummer. I know there’s been at least a bit of overlap for a long time, but ten years ago it felt like effective altruism had its own, unique culture and nowadays it feels like the LessWrong culture has almost completely taken over. I have never felt good about LessWrong or “rationalism” and the more knowledge and experience of it I’ve gained, the more I’ve accumulated a sense of repugnance, horror, and anger toward that culture and ideology. I hate to see that become what effective altruism is like.
-The stories about sexual harassment are so disgusting. They’re really, really bad and crazy. And it’s so annoying how many comments you see on EA Forum posts about sexual harassment that make exhausting, unempathetic, arrogant, and frankly ridiculous statements, if not borderline incomprehensible in some cases. You see these stories of sexual harassment in the posts and you see evidence of the culture that enables sexual harassment in the comments. Very, very, very bad. Not my idea of a community I can wholeheartedly feel I belong to.
-Kind of a similar story with sexism, racism, and transphobia. The level of underreaction I’ve seen to instances of racism has been crazymaking. It’s similar to the comments under the posts about sexual harassment. You see people justifying or downplaying clearly immoral behaviour. It’s sickening.
-A lot of the response to the Nonlinear controversy was disheartening. It was disheartening to see how many people were eager to enable, justify, excuse, downplay, etc. bad behaviour. Sometimes aggressively, arrogantly, and rudely. It was also disillusioning to see how many people were so… easily fooled.
-Nobody talks normal in this community. At least not on this forum, in blogs, and on podcasts. I hate the LessWrong lingo. To the extent the EA Forum has its own distinct lingo, I probably hate that too. The lingo is great if you want to look smart. It’s not so great if you want other people to understand what the hell you are talking about. In a few cases, it seems like it might even be deliberate obscurantism. But mostly it’s just people making poor choices around communication and writing style and word choice, maybe for some good reasons, maybe for some bad reasons, but bad choices either way. I think it’s rare that writing with a more normal diction wouldn’t enhance people’s understanding of what you’re trying to say, even if you’re only trying to communicate with people who are steeped in the effective altruist niche. I don’t think the effective altruist sublanguage is serving good thinking or good communication.
-I see a lot of interesting conjecture elevated to the level of conventional wisdom. Someone in the EA or LessWrong or rationalist subculture writes a creative, original, evocative blog post or forum post and then it becomes a meme, and those memes end up taking on a lot of influence over the discourse. Some of these ideas are probably promising. Many of them probably contain at least a grain of truth or insight. But they become conventional wisdom without enough scrutiny. Just because an idea is “homegrown” it takes on the force of a scientific idea that’s been debated and tested in peer-reviewed journals for 20 years, or a widely held precept of academic philosophy. That seems just intellectually the wrong thing to do and also weirdly self-aggrandizing.
-An attitude I could call “EA exceptionalism”, where people assert that people involved in effective altruism are exceptionally smart, exceptionally wise, exceptionally good, exceptionally selfless, etc. Not just above the average or median (however you would measure that), but part of a rare elite and maybe even superior to everyone else in the world. I see no evidence this is true. (In these sorts of discussions, you also sometimes see the lame argument that effective altruism is definitionally the correct approach to life because effective altruism means doing the most good and if something isn’t doing the most good, then it isn’t EA. The obvious implication of this argument is that what’s called “EA” might not be true EA, and maybe true EA looks nothing like “EA”. So, this argument is not a defense of the self-identified “EA” movement or community or self-identified “EA” thought.)
-There is a dark undercurrent to some EA thought, along the lines of negative utilitarianism, anti-natalism, misanthropy, and pessimism. I think there is a risk of this promoting suicidal ideation because it basically is suicidal ideation.
-Too much of the discourse seems to revolve around how to control people’s behaviours or beliefs. It’s a bit too House of Cards. I recently read about the psychologist Kurt Lewin’s study on the most effective ways to convince women to use animal organs (e.g. kidneys, livers, hearts) in their cooking during meat shortages during World War II. He found that a less paternalistic approach that showed more respect for the women’s was more effective in getting them to incorporate animal organs into their cooking. The way I think about this is: you didn’t have to be manipulated to get to the point where you are in believing what you believe or caring this much about this issue. So, instead of thinking of how to best manipulate people, think about how you got to the point where you are and try to let people in on that in an honest, straightforward way. Not only is this probably more effective, it’s also more moral and shows more epistemic humility (you might be wrong about what you believe and that’s one reason not to try to manipulate people into believing it).
-A few more things but this list is already long enough.
Put all this together and the old stuff I cared about (charity effectiveness, giving what I can, expanding my moral circle) is lost in a mess of other stuff that is antithetical to what I value and what I believe. I’m not even sure the effective altruism movement should exist anymore. The world might be better off if it closed down shop. I don’t know. It could free up a lot of creativity and focus and time and resources to work on other things that might end up being better things to work on.
I still think there is value in the version of effective altruism I knew around 2015, when the primary focus was on global poverty and the secondary focus was on animal welfare, and AGI was on the margins. That version of effective altruism is so different from what exists today — which is mostly about AGI and has mostly been taken over by the rationalist subculture — that I have to consider those two different things. Maybe the old thing will find new life in some new form. I hope so.
On cause prioritization, is there a more recent breakdown of how more and less engaged EAs prioritize? Like an update of this? I looked for this from the 2024 survey but could not find it easily: https://forum.effectivealtruism.org/posts/sK5TDD8sCBsga5XYg/ea-survey-cause-prioritization
Thanks for sharing this, while I personally believe the shift in focus on AI is justified (I also believe working on animal welfare is more impactful than global poverty), I can definitely sympathize with many of the other concerns you shared and agree with many of them (especially LessWrong lingo taking over, the underreaction to sexism/racism, and the Nonlinear controversy not being taken seriously enough). While I would completely understand in your situation if you don’t want to interact with the community anymore, I just want to share that I believe your voice is really important and I hope you continue to engage with EA! I wouldn’t want the movement to discourage anyone who shares its principles (like “let’s use our time and resources to help others the most”), but disagrees with how it’s being put into practice, from actively participating.
My memory is a large number of people to the NL controversy seriously, and the original threads on it were long and full of hostile comments to NL, and only after someone posted a long piece in defence of NL did some sympathy shift back to them. But even then there are like 90-something to 30-something agree votes and 200 karma on Yarrow’s comment saying NL still seem bad: https://forum.effectivealtruism.org/posts/H4DYehKLxZ5NpQdBC/nonlinear-s-evidence-debunking-false-and-misleading-claims?commentId=7YxPKCW3nCwWn2swb
I don’t think people dropped the ball here really, people were struggling honestly to take accusations of bad behaviour seriously without getting into witch hunt dynamics.
Good point, I guess my lasting impression wasn’t entirely fair to how things played out. In any case, the most important part of my message is that I hope he doesn’t feels discouraged from actively participating in EA.
I’d distinguish here between the community and actual EA work. The community, and especially its leaders, have undoubtedly gotten more AI-focused (and/or publicly admittted to a degree of focus on AI they’ve always had) and rationalist-ish. But in terms of actual altruistic activity, I am very uncertain whether there is less money being spent by EAs on animal welfare or global health and development in 2025 than there was in 2015 or 2018. (I looked on Open Phil’s website and so far this year it seems well down from 2018 but also well up from 2015, but also 2 months isn’t much of a sample.) Not that that means your not allowed to feel sad about the loss of community, but I am not sure we are actually doing less good in these areas than we used to.
Yes, this seems similar to how I feel: I think the major donor(s) have re-prioritized, but am not so sure how many people have switched from other causes to AI. I think EA is more left to the grassroots now, and the forum has probably increased in importance. As long as the major donors don’t make the forum all about AI—then we have to create a new forum! But as donors change towards AI, the forum will inevitable see more AI content. Maybe some functions to “balance” the forum posts so one gets representative content across all cause areas? Much like they made it possible to separate out community posts?
The key objection I always have to starting new charities, as Charity Entrepreneurship used to focus on is that I feel is money usually not the bottleneck? I mean, we already have a ton of amazing ideas of how to use more funds, and if we found new ones, it may be very hard to reduce the uncertainty sufficiently to be able to make productive decisions. What do you think Ambitious Impact ?
A new organization can often compete for dollars that weren’t previously available to an EA org—such as government or non-EA foundation grants that are only open to certain subject areas.
That is actually a good point, thanks Jason.
Hot take, but political violence is bad and will continue to be bad in the foreseeable near-term future. That’s all I came here to say folks, have a great rest of your day.
Back in October 2024, I tried to test various LLM Chatbots with the question:
”Is there a way to convert a correlation to a probability while preserving the relationship 0 = 1/n?”
Years ago, I came up with an unpublished formula that does just that:
p(r) = (n^r * (r + 1)) / (2^r * n)
So I was curious if they could figure it out. Alas, back in October 2024, they all made up formulas that didn’t work.
Yesterday, I tried the same question on ChatGPT and, while it didn’t get it quite right, it came, very, very close. So, I modified the question to be more specific:
”Is there a way to convert a correlation to a probability while preserving the relationships 1 = 1, 0 = 1/n, and −1 = 0?”
This time, it came up with a formula that was different and simpler than my own, and… it actually works!
I tried this same prompt with a bunch of different LLM Chatbots and got the following:
Correct on the first prompt:
GPT4o, Claude 3.7
Correct after explaining that I wanted a non-linear, monotonic function:
Gemini 2.5 Pro, Grok 3
Failed:
DeepSeek-V3, Mistral Le Chat, QwenMax2.5, Llama 4
Took too long thinking and I stopped it:
DeepSeek-R1, QwQ
All the correct models got some variation of:
p(r) = ((r + 1) / 2)^log2(n)
This is notably simpler and arguably more elegant than my earlier formula. It also, unlike my old formula, has an easy to derive inverse function.
So yeah. AI is now better than me at coming up with original math.
Per Bloomberg, the Trump administration is considering restricting the equivalency determination for 501(c)3s as early as Tuesday. The equivalency determination allows for 501(c)3s to regrant money to foreign, non-tax-exempt organisations while maintaining tax-exempt status, so long as an attorney or tax practitioner claims the organisation is equivalent to a local tax-exempt one.
I’m not an expert on this, but it sounds really bad. I guess it remains to be seen if they go through with it.
Regardless, the administration is allegedly also preparing to directly strip environmental and political (i.e. groups he doesn’t like, not necessarily just any policy org) non-profits of their tax exempt status. In the past week, he’s also floated trying to rescind the tax exempt status of Harvard. From what I understand, such an Executive Order is illegal under U.S. law (to whatever extent that matters anymore), unless Trump instructs the State Department to designate them foreign terrorist organisations, at which point all their funds are frozen too.
These are dark times. Stay safe 🖤
Is anyone in the U.S. savvy with how to deduct from your taxes the value of stocks which have been donated to eligible charities? The stocks have been held for decades with a very high value and capital gains. Would love help as my tax guy hasn’t seen this before.
Update for anyone else who may find it useful:
You need to fill out Form 8283: https://www.irs.gov/pub/irs-pdf/f8283.pdf
You can calculate the “Fair Market Value” of the stock(s) you donated by averaging the highest and lowest price of that stock on the day you donated it. I used this page to find that, but you can replace “MSFT” in the URL with whatever stock it is you sold.
https://www.wsj.com/market-data/quotes/MSFT/historical-prices