I’m currently researching forecasting and epistemics as part of the Quantified Uncertainty Research Institute.
Ozzie Gooen
Karma: 9,901
If the confusion is that you expected us to have more runway, I’m not very sure what to say. I think this sector can be pretty difficult. We’re in talks for funding from one donor, which would help cover this gap, but I’d like to not depend that much on them.
We also do have a few months of reserves that we could spend in 2025 if really needed.
We so far raised $62,000 for 2025, from the Survival and Flourishing Fund.
Slava and myself are both senior software engineers, I’m in Berkeley (to be close to the EA scene here). Total is roughly $200k for the two of us (including taxes and health care).
In addition, we have server and software payments, plus other misc payments.
We then have a 14% overhead from our sponsorship with Rethink Priorities.
I said around $200k, so this assumes basically a $262k budget. This is on the low end for what I’d really prefer, but given the current EA funding situation, is what I’ll aim for now.
If we had more money we could bring in contractors for things like research and support.
Answering on behalf the Quantified Uncertainty Research Institute!
We’re looking to raise another ~$200k for 2025, to cover our current two-person team plus expenses. We’d also be enthusiastic about expanding our efforts if there is donor interest.
We at QURI have been busy on software infrastructure and epistemics investigations this last year. We currently have two full-time employees—myself and Slava Matyuhin. Slava focuses on engineering, I do a mix of engineering, writing, and admin.Our main work this year has been improving Squiggle and Squiggle Hub.
In the last few months we’ve built Squiggle AI as well, which we’ve started getting feedback on and will write more about here shortly. Basically, we believe that BOTECs and cost-benefit models are good fits for automation. So far, with some tooling, we think that we’ve created a system that produces decent first passes on many simple models. This would ideally be something EAs benefit from directly, and something that could help inspire other epistemic AI improvements.
On the side of software development, we’ve posted a series of articles about forecasting, epistemics, and effective altruism. Recently these have focused on the combination of AI and epistemics.
For 2025, we’re looking to expand more throughout the EA and AI safety ecosystems. We have a backlog of Squiggle updates to inform people, and have a long list of new things we expect people to like. We’ve so far focused on product experimentation and development, and would like to spend more time on education and outreach. In addition, we’ll probably continue focusing a lot on AI—both on improving AI systems to write and audit cost-effectiveness models and similar, and also on helping build cost-effectiveness models to guide AI safety.
If you support this sort of work and are interested in chatting or donating, please reach out! You can reach me at ozzie@quantifieduncertainty.org. We’re very focused on helping the EA ecosystem, and would really like to diversify our base of close contacts and donors.
Donate
QURI is fiscally sponsored by Rethink Priorities. We have a simple donation page here.
I still think that EA Reform is pretty important. I believe that there’s been very little work so far on any of the initiatives we discussed here.
My impression is that the vast majority of money that CEA gets is from OP. I think that in practice, this means that they represent OP’s interests significantly more than I feel comfortable with. While I generally like OP a lot, I think OP’s focuses are fairly distinct from those of the regular EA community.
Some things I’d be eager to see funded:
- Work with CEA to find specific pockets of work that the EA community might prioritize, but OP wouldn’t. Help fund these things.
- Fund other parties to help represent / engage / oversee the EA community.
- Audit/oversee key EA funders (OP, SFF, etc); as these often aren’t reviewed by third parties.
- Make sure that the management in key EA orgs are strong, including the boards.
- Make sure that many key EA employees and small donors are properly taken care of and are provided with support. (I think that OP has reason to neglect this area, as it can be difficult to square with naive cost-effectiveness calculations)
- Identify voices that want to tackle some of these issues head-on, and give them a space to do so. This could mean bloggers / key journalists / potential community leaders in the future.
- Help encourage or set up new EA organizations to sit apart from CEA, but help oversee/manage the movement.
- Help out the Community Health team at CEA. This seems like a very tough job that could arguably use more support, some of which might be best done outside of CEA.
Generally, I feel like there’s a very significant vacuum of leadership and managerial visibility in the EA community. I think that this is a difficult area to make progress on, but also consider it much more important than other EA donation targets.
Thanks for bringing this up. I was unsure what terminology would be best here.
I mainly have in mind fermi models and more complex but similar-in-theory estimations. But I believe this could extend gracefully for more complex models. I don’t know of many great “ontologies of types of mathematical models,” so am not sure how to best draw the line.
Here’s a larger list that I think could work.Fermi estimates
Cost-benefit models
Simple agent-based models
Bayesian models
Physical or social simulations
Risk assessment models
Portfolio optimization models
I think this framework is probably more relevant for models estimating an existing or future parameter, than models optimizing some process, if that helps at all.
Enhancing Mathematical Modeling with LLMs: Goals, Challenges, and Evaluations
Ah, I didn’t quite notice that at the time—that wasn’t obvious from the UI (you need to hover over the date to see the time of it being posted).
Anyway, happy this was resolved! Also, separately, kudos for writing this up, I’m looking forward to seeing where Metaculus goes this next year +.
(The opening line was removed)
I feel like the bulk of this is interesting, but the title and opening come off as more grandiose than necessary.
This is neat to see!
Obviously, some of these items are much more likely than others to kill 100M+ lives.
WW3 seems like a big wild card to me. I’d be curious if there are any/many existing attempts to try to estimate would it would look like and how bad it would be.
I think animals are generally more efficient/effective as a way of converting money into short-term (the next 50 years) well-being.
My impression is that the mean global health intervention does not significantly improve the long-term future. However, I could definitely be convinced otherwise, and that would get me to change my answer.
All that said, if one is focused on improving the long-term future, it seems suspicious to focus on global health, as opposed to other interventions that are clearly more focused on that.
Around discussions of AI & Forecasting, there seems to be some assumption like:
1. Right now, humans are better than AIs at judgemental forecasting.
2. When humans are better than AIs at forecasting, AIs are useless.
3. At some point, AIs will be better than humans at forecasting.
4. At that point, when it comes to forecasting, humans will be useless.
This comes from a lot of discussion and some research comparing “humans” to “AIs” in forecasting tournaments.
As you might expect, I think this model is incredibly naive. To me, it’s asking questions like,
”Are AIs better than humans at writing code?”
“Are AIs better than humans at trading stocks?”
”Are AIs better than humans at doing operations work?”
I think it should be very clear that there’s a huge period, in each cluster, where it makes sense for humans and AIs to overlap. “Forecasting” is not one homogeneous and singular activity, and neither is programming, stock trading, or doing ops. There’s no clear line for automating “forecasting”—there are instead a very long list of different skills one could automate, with a long tail of tasks that would get increasingly expensive to automate.Autonomous driving is another similar example. There’s a very long road between “helping drivers with driver-assist features” and “complete level-5 automation, to the extent that almost no human are no longer driving for work purposes.”
A much better model is a more nuanced one. Break things down into smaller chunks, and figure out where and how AIs could best augment or replace humans at each of those. Or just spend a lot of time working with human forecasting teams to augment parts of their workflows.
Some ideas:
1. What are the main big mistakes that EAs are making? Maybe have a few people give 30-minute talks or something.
2. A summary of the funding ecosystem and key strategic considerations around EA. Who are the most powerful actors, how competent are they, what are our main bottlenecks at the moment?
3. I’d like frank discussions about how to grow funding in the EA ecosystem, outside of the current donors. I think this is pretty key.
4. It would be neat to have a debate or similar on AI policy legislation. We’re facing a lot of resistance here, and some of it is uncertain.
5. Is there any decent 5-10 year plan of what EA itself should be? Right now most of the funding ultimately comes from OP, and there’s very little non-OP community funding or power. Are there ideas/plans to change this?
I generally think that EA Globals have had far too little disagreeable content. It feels like they’ve been very focused on making things seem positive for new people, instead of focusing more on candid and more raw disagreements and improvement ideas.
I really would like to see more communication with the Global Catastrophic Risks Capacity Building Team at Open Philanthropy, given that they’re the ones in charge of funding much of the EA space. Ideally there would be a lot of capacity for Q&A here.
Quick point—I think the relationship between CEA and Leverage was pretty complicated during a lot of this period.
There was typically a large segment of EAs who were suspicious of Leverage, ever since their founding. But Leverage did collaborate with EAs on some specific things early on (like the first EA Summit). It felt like an uncomfortable alliance type situation. If you go back on the forum / Lesswrong, you can read artifacts.
I think the period of 2018 or so was unusual. This was a period where a few powerful people at CEA (Kerry, Larissa) were unusually pro-Leverage and got to power fairly quickly (Tara left, somewhat suddenly). I think there was a lot of tension around this decision, and when they left (I think this period lasted around 1 year), I think CEA became much less collaborative with Leverage.
One way to square this a bit is that CEA was just not very powerful for a long time (arguably, its periods of “having real ability/agency to do new things” have been very limited). There were periods where Leverage had more employees (I’m pretty sure). The fact that CEA went through so many different leaders, each with different stances and strategies, makes it more confusing to look back on.
I would really love for a decent journalist to do a long story on this history, I think it’s pretty interesting.
I think Garry Tan is more left-wing, but I’m not sure. A lot of the e/acc community fights with EA, and my impression is that many of them are leftists.
I think that the right-wing techies are often the loudest, but there are also lefties in this camp too.
(Honestly though, the right-wing techies and left-wing techies often share many of the same policy ideas. But they seem to disagree on Trump and a few other narrow things. Many of the recent Trump-aligned techies used to be more left-coded.)
Random Tweet from today: https://x.com/garrytan/status/1820997176136495167
Garry Tan is the head of YCombinator, which is basically the most important/influential tech incubator out there. Around 8 years back, relations were much better, and 80k and CEA actually went through YCombinator.I’d flag that Garry specifically is kind of wacky on Twitter, compared to previous heads of YC. So I definitely am not saying it’s “EA’s fault”—I’m just flagging that there is a stigma here.
I personally would be much more hesitant to apply to YC knowing this, and I’d expect YC would be less inclined to bring in AI safety folk and likely EAs.
My personal take is that there are a bunch of better trade-offs between the two that we could be making. I think that the narrow subset of risks is where most of the value is, so from that standpoint, that could be a good trade-off.
Also, I suspect that the current EA AI policy arm could find ways to be more diplomatic and cooperative
My impression is that the current EA AI policy arm isn’t having much active dialogue with the VC community and the like. I see Twitter spats that look pretty ugly, I suspect that this relationship could be improved on with more work.
At a higher level, I suspect that there could be a fair bit of policy work that both EAs and many of these VCs and others would be more okay with than what is currently being pushed. My impression is that we should be focused on narrow subsets of risks that matter a lot to EAs, but don’t matter much to others, so we can essentially trade and come out better than we are now.
Thanks for the clarification!
Yea, I’m not very sure what messaging to use. It’s definitely true that there’s a risk we won’t be able to maintain our current team for another year. At the same time, if we could get more than our baseline of funding, I think we could make good use of it (up to another 1-2 FTE, for 2025).
I’m definitely still hoping that we could eventually (next 1-5 years) either significantly grow (this could mean up to 5-7 FTE) or scale in other ways. Our situation now seems pretty minimal to me, but I still strongly prefer it to not having it.
I’d flag that the funding ecosystem feels fairly limited for our sort of work. The main options are really the SFF and the new Open Philanthropy forecasting team. I’ve heard that some related groups have also been having challenges with funding.