I’m currently researching forecasting and epistemics as part of the Quantified Uncertainty Research Institute.
Ozzie Gooen
On EAs in policy, I’d flag that:
- There’s a good number of people currently working in AI governance, Bio governance, and animal law.
- Very arguably, said people have had a decent list of accomplishments and power positions, given that such work was fairly recent. See Biden’s executive orders on AI, or the UK AI Security Institute. https://www.aisi.gov.uk/
- People like Dustin Moskovitz and SBF were some highly prominent donors to the Democratic party.
I think the EA policy side might not get a huge amount of popularity here, but it seems decently reputable to me. Mistakes have been made, but I think a decent report on the wins and losses would include several wins.
I do agree that finding others doing well and helping them is one important way to help. I’d suspect that the most obvious EA work would look like prioritization for policy efforts. This has been done before, and there’s a great deal more that could be done here.
I disagree, but this has me curious.
My impression from other writing I’ve seen of yours is that you don’t think that EAs are good at too many things. What do you think EAs are best at, and/or should be doing? Perhaps, narrow GiveWell-style research on domains with lots of data?
I think in-comparison to the space, EA has a comparative advantage more in talent than in money. I think the Harris campaign got $2B or so of donations, but I get the impression that it could have used smarter + more empirically-minded people. That said, there is of course the challenge or actually getting those people to be listened to.
(I rewrote with Claude, I think it’s much more understandable now)
I’ve substantially revised my views on QURI’s research priorities over the past year, primarily driven by the rapid advancement in LLM capabilities.Previously, our strategy centered on developing highly-structured numeric models with stable APIs, enabling:
Formal forecasting scoring mechanisms
Effective collaboration between human forecasting teams
Reusable parameterized world-models for downstream estimates
However, the progress in LLM capabilities has updated my view. I now believe we should focus on developing and encouraging superior AI reasoning and forecasting systems that can:
Generate high-quality forecasts on-demand, rather than relying on pre-computed forecasts for scoring
Produce context-specific mathematical models as needed, reducing the importance of maintaining generic mathematical frameworks
Leverage repositories of key insights, though likely not in the form of formal probabilistic mathematical models
This represents a pivot from scaling up traditional forecasting systems to exploring how we can enhance AI reasoning capabilities for forecasting tasks. The emphasis is now on dynamic, adaptive systems rather than static, pre-structured models.
I kind of hate to say this, but in the last year I’ve become much less enamored by this broad idea. Due to advances in LLMs, my guess now is that:
1. People will ask LLMs for ideas/forecasts at the point that they need them, and the LLMs will do much of the work right then.
2. In terms of storing information and insights about the world, Scorable functions are probably not the best (it’s not clear what is)
3. Ideally, we could basically treat the LLMs as the “Scorable Function”. As in, we have a rating for how good a full LLM is. This becomes more important than any Scorable Function.
That said, Scorable Functions could be a decent form of LLM output here and there. It would be obvious to train LLMs to be great at outputting Scorable Functions.
More info here:
https://forum.effectivealtruism.org/posts/mopsmd3JELJRyTTty/ozzie-gooen-s-shortform?commentId=vxiAAoHhmQqe2Afc9
Quick list of some ideas I’m excited about, broadly around epistemics/strategy/AI.
1. I think AI auditors / overseers of critical organizations (AI efforts, policy groups, company management) are really great and perhaps crucial to get right, but would be difficult to do well.
2. AI strategists/tools telling/helping us broadly what to do about AI safety seems pretty safe.
3. In terms of commercial products, there’s been some neat/scary military companies in the last few years (Palantir, Anduril). I’d be really interested if there could be some companies to automate core parts of the non-military government. I imagine there are some parts of the government that are particularly tractable/influenceable/tractable. For example, just making great decisions on which contractors the government should work with. There’s a ton of work to do here, between the federal government / state government / local government.
4. Epistemic Evals of AI seem pretty great to me, I imagine work here can/should be pushed more soon. I’m not a huge fan of emphasizing “truthfulness” specifically, I think there’s a whole lot to get right here. I think my post here is relevant—it’s technically specific to evaluating math models, but I think it applies to broader work. https://forum.effectivealtruism.org/posts/fxDpddniDaJozcqvp/enhancing-mathematical-modeling-with-llms-goals-challenges
5. One bottleneck to some of the above is AI with strong guarantees+abilities of structured transparency. It’s possible that more good work here can wind up going a long way. That said, some of this is definitely already something companies are trying to do for commercial reasons. https://forum.effectivealtruism.org/posts/piAQ2qpiZEFwdKtmq/llm-secured-systems-a-general-purpose-tool-for-structured
6. I think there are a lot of interesting ways for us to experiment with [AI tools to help our research/epistemics]. I want to see a wide variety of highly creative experimentation here. I think people are really limiting themselves in this area to a few narrow conceptions of how AI can be used in very specific ways that humans are very comfortable with. For example, I’d like to see AI dashboards of “How valuable is everything in this space” or even experiments where AIs negotiate on behalf of people and they use the result of that. A lot of this will get criticized for being too weird/disruptive/speculative, but I think that’s where good creative works should begin.
7. Right now, I think the field of “AI forecasting” is actually quite small and constrained. There’s not much money here, and there aren’t many people with bold plans or research agendas. I suspect that some successes / strong advocates could change this.
8. I think that it’s likely that Anthropic (and perhaps Deepmind) would respond well to good AI+epistemics work. “Control” was quickly accepted at Anthropic, for example. I suspect that it’s possible that things like the idea of an “Internal AI+human auditor” or an internal “AI safety strategist” could be adopted if done well.
I think SBF was fairly unique in caring about this, and his empire collapsed before he did anything like this in that election. When I said “undervalue”, I wasn’t referring to SBF, given he hasn’t been active in this time. (Obvious flag that while I might have sympathized with some of SBF’s positions, I very much disagree with many of his illegal and fraudulent actions)
Looking back though, paying $5B for Trump to definitely not run/win seems like a great deal to me, though the act of paying him raises a lot of qualms I’d be uncomfortable with.
Looking back, it seems quite possible to me that EAs undervalued the importance of helping with the previous election. For one limited thing, having this administration be in power when we’re getting so close to TAI seems like a major failure.
Personally, I think I’ve become more convinced recently that generic US policy, especially focusing on long-term issues (like US governance, or US decisions on questions like Nuclear/bio/AI) might be a good use of EA funds.
I would have really guessed that a lot of this area wouldn’t at all be neglected, but in practice, it seems to be far more ignored than I think is reasonable.
The author was transparent about this, with the quote that you highlight. I feel like when the author is clear on the source, and has provided some amount of effort of oversight, then the information could be handled responsibly by readers.
Thanks for raising the concern!
I agree that testing it is difficult. I partially addressed this above in the section on “Strategy and Verifiability”.
I would flag that people should arguably be equally suspicious of most humans. As we come up with various tests and evals, I expect that mostly the best AIs will have mediocre results, and most prominent humans will just refuse to be tested (we can still do lighter evals on public intellectuals and such using their available works, but this will be more limited).Prediction markets seem like a pretty good test to me, though they are only one implementation.
I expect that with decent systems, we should have new “epistemic table stakes” of things like:Can do forecasting in a wide variety of fields, at least roughly as good as Metaculus forecasters with say 10hrs per question
In extensive simulations, has low amounts of logical inconsistencies
Flags all claims that users might not agree with
Very low rates of hallucinations
Biases have been extensively tested under different situations
Extensive red-teaming by other top AI systems
Predictions of how well this AI will hold up, in comparison to better AI intellectual systems in 10 to 40 years.
Full oversight/visibility of potential conflicts of interest.
(I’m not saying that these systems will be broadly-trusted, just that they will exist. I would expect the smarter people at least to trust them, in accordance to their evals.)
6 (Potential) Misconceptions about AI Intellectuals
Thanks for the feedback!
We’re just looking for a final Fermi model. You can use or not use AI to come up with this.
“Surprise” is important because that’s arguably what makes a model interesting. As in, if you have a big model about the expected impact of AI, and then it tells you the answer you started out expecting, then arguably it’s not an incredibly useful model.
The specific “Surprise” part of the rubric doesn’t require the model to be great, but the other parts of the rubric do weight that. So if you have a model that’s very surprising but otherwise poor, then it might do well on the “Surprise” measure, but won’t on the other measures, so on average will get a mediocre score.
Note that there were a few submissions on LessWrong so far, those might make things clearer:
https://www.lesswrong.com/posts/AA8GJ7Qc6ndBtJxv7/usd300-fermi-model-competition#comments
”On the one hand it says “Our goal is to discover creative ways to use AI for Fermi estimation” but on the other hand it says “AI tools to generate said estimates aren’t required, but we expect them to help.”-> We’re not forcing people to use AI, in part because it would be difficult to verify. But I expect that many people will do so, so I still expect this to be interesting.
Thanks for the reminder!
Important Note: The EA Infrastructure Fund is currently unable to make grants with an end date after August 31st 2025, and any applications to this fund must have a grant period which ends on or before this date.
I assume that:
1. This restriction is only for the EA Infrastructure Fund, not the LTFF.
2. Given that it takes 8 weeks to get a response, this means that projects can only take roughly 5.5 months or less.
Are those points correct?
I think it’s interesting and admiral that you’re dedicated on a position that’s so unusual in this space.
I assume I’m in the majority here that my intuitions are quite different from yours, however.
One quick point when we’re here:
> this view is likely rooted in a bias that automatically favors human beings over artificial entities—thereby sidelining the idea that future AIs might create equal or greater moral value than humans—and treating this alternative perspective with unwarranted skepticism.
I think that a common, but perhaps not well vocalized, utilitarian take is that humans don’t have much of a special significance in terms of creating well-being. The main option would be a much more abstract idea, some kind of generalization of hedonium or consequentialism-ium or similar. For now, let’s define hedonium as “the ideal way of converting matter and energy into well-being, after a great deal of deliberation.”
As such, it’s very tempting to try to separate concerns and have AI tools focus on being great tools, and separately optimize hedonium to be efficient at being well-being. While I’m not sure if AIs would have zero qualia, I’d feel a lot more confident that they will have dramatically less qualia per unit resources than a much more optimized substrate.
If one follows this general logic, then one might assume that it’s likely that the vast majority of well-being in the future would exist as hedonium, not within AIs created to ultimately make hedonium.
One less intense formulation would be to have both AIs and humans focus only on making sure we get to the point where we much better understand the situation with qualia and hedonium (a la the Long Reflection), and then re-evaluate. In my strategic thinking around AI I’m not particularly optimizing for the qualia of the humans involved in the AI labs or the relevant governments. Similarly, I’d expect not to optimize hard for the qualia in the early AIs, in the period when we’re unsure about qualia and ethics, even if I thought they might have experiences. I would be nervous if I thought this period could involve AIs having intense suffering or be treated in highly immoral ways.
Reminder that this ends soon! Get your submissions in.
Thanks for reaching out about this, it seems like a task that others likely have too.
I know a handful of people who could retire soon, but instead stay active in the space.
At a high level, I really don’t think that [being able to retire] should change your plans that much. The vast majority of recommendations from 80,000 Hours, and work done by Effective Altruists, wouldn’t be impacted by this. For instance, for most of the important positions, money to hire a specific candidate isn’t a major bottleneck—if you’re good enough to provide a lot of value, then a livable/basic salary really shouldn’t be a deal-breaker.
There are some situations where it can be very useful to basically do useful independent projects for a few years without needing to raise funding. But these are pretty niche, and require a lot of knowledge about what to do.
From what I’ve seen, most people who can retire and want to help out, typically don’t really want to do the work, or don’t want to accept positions that aren’t very high status (as is typically needed to at least get started in a new position). These people seem to have a habit of trying a little bit with something they would enjoy a lot or identify with, finding that that doesn’t work great, then completely giving up.
So while having the extra money can be useful, it can just as easily be long-term damaging for making an impact. I think it can be very tempting to just enjoy the retirement life.All this to say, if you think that might be a risk for you, it’s something I’d recommend you think long and hard about, consider how much you care about making an impact in the rest of your life, then come up with strategies to make sure you actually do that.
Personally, I think the easy thing to advise is something like, “keep as much money as you basically need to not worry too much about your future”, generally donate everything above that threshold, then think of yourself as a regular person attempting a career in charity/altruism. The good organizations will still pay you a salary, and you can donate (basically) everything you make.
There was discussion on LessWrong:
https://www.lesswrong.com/posts/YqrAoCzNytYWtnsAx/the-failed-strategy-of-artificial-intelligence-doomers
Obvious point, but I assume that [having a bunch of resources, mainly money] is a pretty safe bet for these worlds.
AI progress could/should bring much better ideas of what to do with said resources/money as it happens.
It looks like Concept2, a popular sports equipment company, just put ownership into a Purpose Trust.
As we look toward stepping back from day-to-day operations, we have researched options to preserve the company’s long-held values and mission into the future, including Purpose Trust ownership. With a Purpose Trust as owner, profits are reinvested in the business and also used to fulfill the designated purpose of the Trust. Company profits do not flow to individual shareholders or beneficiaries. And a Purpose Trust can endure in perpetuity.
We are excited to announce we have transferred 100% of Concept2’s ownership to the Concept2 Perpetual Purpose Trust as of January 1, 2025. The Concept2 Perpetual Purpose Trust will direct the management and operations of Concept2 in a manner that maintains continuity. The value we create through our business will be utilized for a greater purpose in serving the Concept2 community. Our vision and mission will carry on in the hands of our talented employee base, and Concept2 will remain the gold standard for providing best in-class products and unmatched customer service. We hope you share in our enthusiasm and will join us on this next phase of our journey as a company.′
I asked Perplexity for other Purchase Trusts, it mentioned that Patagonia is one, plus a few other companies I don’t know of.
My impression is that B-Corps have almost no legal guarantees of public good, and that 501c3s also really have minimal guarantees (if 501c3s fail to live up to their mission, the worst that happens is that they lose their charity and thus tax-deductability status. But this isn’t that bad otherwise).I imagine that Trusts could be far more restrictive (in a good way). I worked with a company that made Irrevocable Trusts before, I think these might be the structure that would provide the best assurances that we currently have.
Thinking about this a bit more—
My knee-jerk reaction is to feel attacked by this comment, on behalf of the EA community.
I assume that one thing that might be going on is a miscommunication. Perhaps you believe that I was assuming that EAs could quickly swoop in, spent a little time on things, and be far more correct than many experience political experts and analysts.
I’m not sure if this helps, but the above really doesn’t align with what I’m thinking. More something like, “We could provide more sustained help through a variety of methods. People can be useful for many things, like direct volunteering, working in think tanks, being candidates, helping prioritization, etc. I don’t expect miracle results—I instead expect roughly the results of adding some pretty smart and hardworking people.”