I’m currently researching forecasting and epistemics as part of the Quantified Uncertainty Research Institute.
Ozzie Gooen
Karma: 11,899
I feel like EAs might be sleeping a bit on digital meetups/conferences.
My impression is that many people prefer in-person events to online ones. But at the same time, a lot of people hate needing to be in the Bay Area / London or having to travel to events.
There was one EAG online during the pandemic (I believe the others were EAGxs), and I had a pretty good experience there. Some downsides, but some strong upsides. It seemed very promising to me.
I’m particularly excited about VR. I have a Quest3, and have been impressed by the experience of chatting to people in VRChat. The main downside is that there aren’t any professional events in VR that would interest me. Quest 3s are expensive ($500), but far cheaper than housing and office space in Berkeley or London.
I’d also flag:
1. I think that video calls can be dramatically improved with better microphone and camera setups. These can cost $200 to $2k or so, but make a major difference.
2. I’ve been doing some digging into platforms similar to GatherTown. I found GatherTown fairly ugly, off-putting, and limited. SpatialChat seems promising, though it’s more expensive. Zoom seems to be experimenting in the space with products like Zoom Huddles (for coworking in small groups), but these are new.
3. I like Focusmate, but think we could have better spaces for EAs/community members.
4. I think that people above the age of 25 or so find VR weird for what I’d describe as mostly status quo bias. Younger people seem to be far more willing and excited to hangout in VR.
5. I obviously think this is a larger business question. It seems like there was a wave of enthusiasm for remote work at COVID, and this has mostly dried up. However, there are still a ton of remote workers. My guess is that businesses are making a major mistake by not investing enough in better remote software and setups.
6. Organizing community is hard, even if its online. I’d like to see more attempts to pay people to organize online coworking spaces and meetups more.
7. I think that online events/conferences have become associated with the most junior talent. This seems like a pity to me.
8. I expect that different online events should come with different communities and different restrictions. A lot of existing online events/conferences are open to everyone, but then this means that they will be optimized for the most junior people. I think that we want a mix here.
9. Personally, I abhor the idea that I need to couple the place where I physically live with the friends and colleagues I have. I’d very much prefer optimizing for these two separately.
10. I think our community would generally be better off if remote work were easier to do. I’d expect this would help on multiple fronts—better talent, happier talent, lower expenses, more resilience from national politics, etc. This is extra relevant giving the current US political climate—this makes it tougher to recommend that others immigrate to the US or even visit (and the situation might get worse).
11. I’d definitely admit that remote work has a lot of downsides right now, especially with the current tech. So I’m not recommending that all orgs go remote. Just that we work on improving our remote/online infrastructure.
I want to agree, but “best people who ever lived” is a ridiculously high bar! I’d imagine that both of them would be hesitant to claim anything quite that high.
Happy to see this, but of course worried about the growth of insect farming. I didn’t realize it was so likely to grow.
One small point: I like that this essay goes into detail on a probabilistic estimate. I’d find it really useful if there were some other “sanity checks” from other parties to go along with that. For example, questions on Manifold or Metaculus, or even using AI forecasters/estimators to give their takes.
This strikes me as a pretty good forecasting question (very precise, not too far away). I imagine it would be easy to throw it on Manifold and spend $20 or so subsidizing it.
Yea, I broadly agree with Mjreard here.
The BlueDot example seems different to what I was pointing at.
I would flag that lack of EA funding power sometimes makes xrisk less of an issue.
Like, some groups might not trust that OP/SFF will continue to support them, and then do whatever they think they need to in order to attract other money—and this often is at odds with xrisk prioritization.
(I clearly see this as a issue with the broader world, not with OP/SFF)
Quickly:
1. I agree that this is tricky! I think it can be quite tough to be critical, but as you point out, it can also be quite tough to be positive.
2. One challenge with being positive to those in power is that people can have a hard time believing it. Like, you might just be wanting to be liked. Of course, I assume most people would still recommend you being honest, its just can be hard for others to know how to trust it. Also, the situation obviously changes when you’re complementing people without power. (i.e. emerging/local leaders)
I see it a bit differently.
> For example, it doesn’t seem like your project is at serious risk of defunding if you’re 20-30% more explicit about the risks you care about or what personally motivates you to do this work.
I suspect that most nonprofit leaders feel a great deal of funding insecurity. There’s always neat new initiatives that a group would love to expand to, and also, managers hate the risk of potentially needing to fire employees. They’re often thinking about funding on the margins—either they are nervous about firing a few employees, or they are hoping to expand to new areas.
> There are probably only about 200 people on Earth with the context x competence for OP to enthusiastically fund for leading on this work
I think there’s more competition. OP covers a lot of ground. I could easily see them just allocating a bit more money to human welfare later on, for example.
> My wish here is that specific people running orgs and projects were made of tougher stuff re following funding incentives.
I think that the issue of incentives runs deeper than this. It’s not just a matter of leaders straightforwardly understanding the incentives and taking according actions. It’s also that people will start believing things that are convenient to said incentives, that leaders will be chosen who seem to be good fits for the funding situation, and so on. The people who really believe in other goals often get frustrated and leave.
I’d guess that the leaders of these orgs feel more aligned with the OP agenda then they do the agenda you outline, for instance.
Happy to see development and funding in this field.
I would flag the obvious issue that a very small proportion of wild animals live in cities, given that cities take up a small proportion of the world. But I do know that there have been investigation into rats, which do exist in great numbers in cities.
The website for this project shows a fox—but I presume that this was chosen because it’s a sympathetic animal—not because foxes in cities represent a great deal of suffering.
I understand that tradeoffs need to be made to work with different funding sources and circumstances. But I’m of course curious what the broader story is here.
I definitely sympathize, though I’d phrase things differently.
As I’ve noted before, I think much of the cause is just that the community incentives very much come from the funding. And right now, we only have a few funders, and those funders are much more focused on AI Safety specifics then they are things like rationality/epistemics/morality. I think these people are generally convinced on specific AI Safety topics and unconvinced by a lot of more exploratory / foundational work.
For example, this is fairly clear at OP. Their team focused on “EA” is formally called “GCR Capacity Building.” The obvious goal is to “get people into GCR jobs.”
You mention a frustration about 80k. But 80k is getting a huge amount of their funding from OP, so it makes sense to me that they’re doing the sorts of things that OP would like.
Personally, I’d like to see more donations come from community members, to be aimed at community things. I feel that the EA scene has really failed here, but I’m hopeful there could be changes.
I don’t mean to bash OP / SFF / others. I think they’re doing reasonable things given their worldviews, and overall I think they’re both very positive. I’m just pointing out that they represent about all the main funding we have, and that they just aren’t focused on the EA things some community members care about.
Right now, I think that EA is in a very weak position. There just aren’t that many people willing to put in time or money to push forward the key EA programs and mission, other than using it as a way to get somewhat narrow GCR goals.
Or, in your terms, I think that almost no one is actually funding the “Soul” of EA, including the proverbial EA community.
I’d be interested to hear you disagreements with the Marxist-leaning influences. Could you give a few examples?
This is a very long + complex topic.
To me, much of it is a deeper issue. I see EA as coming from academic movements such as the Enlightenment, Empiricism, Analytic Philosophy, Humanism. While I see Marxist-leaning clusters as having influences more like Romanticism, Continental Philosophy, Postmodernism, etc. These are two clusters that have had a few-hundred-year argument/disagreement with each other. I’m sure you can find more with more searches and LLM prompts.
This is the first post in a series on uniting these two movements. We are stronger together, and I hope to demonstrate that each movement contains immense power to help the other. I see myself as a radical feminist and an Effective Altruist and I view those identities as symbiotic rather than contradictory.
Quickly—I think that all smart and truth-seeking people have a lot to learn from each other. My quick impression is that the radical feminist academic community has a mix of good and bad work, as is true with similar movements. I personally admire some of the combination of radicalism and scholarship, but I have disagreements with a lot of the Marxist-leaning influences that seem prevalent.
At the same time, it’s not clear to me what it would even mean to “unite” exactly. I imagine both communities would feel anxious about some of this.
I was reading a blog about EA by the Guerilla Foundation, which contained the quote:
[EA] provides wealth owners with a saviour narrative and a ‘veil of impartiality’ that might hinder deeper scrutiny into the origins of philanthropic money, and stifle personal transformation and solidarity.
And how do EAs respond to this?
I can’t respond for “EAs in total” but I can respond for myself.
For this specific point, I find it a very vague and early hypothesis. A much more concrete and precise claim might be,
”Donors that give to EA causes do so at the expense of greater altruism. We should generally expect that in empirical settings, donors that think they have some sort of ‘veil of impartiality’ fail to do much investigation, and thus wind up donating to worse causes.”
This sounds interesting to me, but it seems like an empirical question, and I’d really want some data or something before making big decisions with it. I could easily see the opposite being true, like,
”Donors who give to causes they think are highly effective will think of themselves as people who care about effectiveness, and then would be more likely to do research and prioritization in the future.”
Basically, this seems to me a lot like a just-so story at this stage.
“starting new public ambitious projects is much less fun if there are a bunch of people on a forum who are out to get you”
To be clear, I assume that the phrase “are out to get you” is just you referring to people giving regular EA Forum critique?
The phrase sounds to me like this is an intentional, long-term effort from some actors to take one down, and they just so happen to use critique as a way of doing that.
As I’m sure many would imagine, I think I disagree.
There are almost no examples of criticism clearly mattering (e.g. getting someone to significantly improve their project)
There’s a lot here I take issue with:
1. I’m not sure where the line is between “criticism” and “critique” or “feedback.” Would any judgements about a project that aren’t positive be considered “criticism”? We don’t have specific examples, so I don’t know what you refer to.
2. This jumps from “criticism matters” to “criticism clearly matters” (which is more easily defensible, but less important), to “criticism clearly mattering (e.g. getting someone to significantly improve their project)”, which is one of several ways that criticism could matter, clearly or otherwise. The latter seems like an incredibly specific claim that misses much of the discussion/benefits of criticism/critique/feedback.I’d rate this post decently high on the “provocative to clarity” measure, as in it’s fairly provocative while also being short. This isn’t something I take issue with, but I just wouldn’t spend too much attention/effort on it, given this. But I would be a bit curious what a much longer and detailed version of this post would be like.
AGI by 2028 is more likely than not
Sorry—my post is coming with the worldview/expectations that at some point, AI+software will be a major thing. I was flagging that in that view, software should become much better.
The question of “will AI+software” be important soon is a background assumption, but a distinct topic. If you are very skeptical, then my post wouldn’t be relevant to you.Some quick points on that topic, however:
1. I think there’s a decent coalition of researchers and programmers who do believe that AI+software will be a major deal very soon (if not already). Companies are investing substantially into it (i.e. Anthropic, OpenAI, Microsoft, etc).
2. I’ve found AI programming tools to be a major help, and so have many other programmers I’ve spoken to.
3. I see the current tools as very experimental and new, still. Very much as a proof of concept. I expect it to take a while to ramp up their abilities / scale. So the fact that the economic impact so far is limited doesn’t surprise me.
4. I’m not very set on extremely short timelines. But I think that 10-30 years would still be fairly soon, and it’s much more likely that big changes will happen on this time frame.
There’s a famous quote, “It’s easier to imagine the end of the world than the end of capitalism,” attributed to both Fredric Jameson and Slavoj Žižek.
I continue to be impressed by how little the public is able to imagine the creation of great software.
LLMs seem to be bringing down the costs of software. The immediate conclusion that some people jump to is “software engineers will be fired.”
I think the impacts on the labor market are very uncertain. But I expect that software getting overall better should be certain.
This means, “Imagine everything useful about software/web applications—then multiply that by 100x+.”
The economics of software companies today are heavily connected to the price of software. Primarily, software engineering is just incredibly expensive right now. Even the simplest of web applications with over 100k users could easily cost $1M-$10M/yr in development. And much of the market cap of companies like Meta and Microsoft is made up of their moat of expensive software.
There’s a long history of enthusiastic and optimistic programmers in Silicon Valley. I think that the last 5 years or so have seemed unusually cynical and hopeless for true believers in software (outside of AI).
But if software genuinely became 100x cheaper (and we didn’t quickly get to a TAI), I’d expect a Renaissance. A time for incredible change and experimentation. A wave of new VC funding and entrepreneurial enthusiasm.
The result would probably feature some pretty bad things (as is always true with software and capitalism), but I’d expect some great things as well.
LLMs seem more like low-level tools to me than direct human interfaces.
Current models suffer from hallucinations, sycophancy, and numerous errors, but can be extremely useful when integrated into systems with redundancy and verification.
We’re in a strange stage now where LLMs are powerful enough to be useful, but too expensive/slow to have rich scaffolding and redundancy. So we bring this error-prone low-level tool straight to the user, for the moment, while waiting for the technology to improve.
Using today’s LLM interfaces feels like writing SQL commands directly instead of using a polished web application. It’s functional if that’s all you have, but it’s probably temporary.
Imagine what might happen if/when LLMs are 1000x faster and cheaper.
Then, answering a question might involve:
Running ~100 parallel LLM calls with various models and prompts
Using aggregation layers to compare responses and resolve contradictions
Identifying subtasks and handling them with specialized LLM batches and other software
Big picture, I think researchers might focus less on making sure any one LLM call is great, and more that these broader setups can work effectively.
(I realize this has some similarities to Mixture of Experts)
I’ve spent some time in the last few months outlining a few epistemics/AI/EA projects I think could be useful.
Link here.I’m not sure how to best write about these on the EA Forum / LessWrong. They feel too technical and speculative to gain much visibility.
But I’m happy for people interested in the area to see them. Like with all things, I’m eager for feedback.
Here’s a brief summary of them, written by Claude.
---1. AI-Assisted Auditing
A system where AI agents audit humans or AI systems, particularly for organizations involved in AI development. This could provide transparency about data usage, ensure legal compliance, flag dangerous procedures, and detect corruption while maintaining necessary privacy.
2. Consistency Evaluations for Estimation AI Agents
A testing framework that evaluates AI forecasting systems by measuring several types of consistency rather than just accuracy, enabling better comparison and improvement of prediction models. It’s suggested to start with simple test sets and progress to adversarial testing methods that can identify subtle inconsistencies across domains.
3. AI for Epistemic Impact Estimation
An AI tool that quantifies the value of information based on how it improves beliefs for specific AIs. It’s suggested to begin with narrow domains and metrics, then expand to comprehensive tools that can guide research prioritization, value information contributions, and optimize information-seeking strategies.
4. Multi-AI-Critic Document Comments & Analysis
A system similar to “Google Docs comments” but with specialized AI agents that analyze documents for logical errors, provide enrichment, and offer suggestions. This could feature a repository of different optional open-source agents for specific tasks like spot-checking arguments, flagging logical errors, and providing information enrichment.
5. Rapid Prediction Games for RL
Specialized environments where AI agents trade or compete on predictions through market mechanisms, distinguishing between Information Producers and Consumers. The system aims to both evaluate AI capabilities and provide a framework for training better forecasting agents through rapid feedback cycles.
6. Analytics on Private AI Data
A project where government or researcher AI agents get access to private logs/data from AI companies to analyze questions like: How often did LLMs lie or misrepresent information? Did LLMs show bias toward encouraging user trust? Did LLMs employ questionable tactics for user retention? This addresses the limitation that researchers currently lack access to actual use logs.
7. Prediction Market Key Analytics Database
A comprehensive analytics system for prediction markets that tracks question value, difficulty, correlation with other questions, and forecaster performance metrics. This would help identify which questions are most valuable to specific stakeholders and how questions relate to real-world variables.
8. LLM Resolver Agents
A system for resolving forecasting questions using AI agents with built-in desiderata including: triggering experiments at specific future points, deterministic randomness methods, specified LLM usage, verifiability, auditability, proper caching/storing, and sensitivity analysis.
9. AI-Organized Information Hubs
A platform optimized for AI readers and writers where systems, experts, and organizations can contribute information that is scored and filtered for usefulness. Features would include privacy levels, payment proportional to information value, and integration of multiple file types.
“AIs doing Forecasting”[1] has become a major part of the EA/AI/Epistemics discussion recently.
I think a logical extension of this is to expand the focus from forecasting to evaluation.
Forecasting typically asks questions like, “What will the GDP of the US be in 2026?”
Evaluation tackles partially-speculative assessments, such as:
“How much economic benefit did project X create?”
“How useful is blog post X?”
I’d hope that “evaluation” could function as “forecasting with extra steps.” The forecasting discipline excels at finding the best epistemic procedures for uncovering truth[2]. We want to maintain these procedures while applying them to more speculative questions.
Evaluation brings several additional considerations:
We need to identify which evaluations to run from a vast space of useful and practical options.
Evaluations often disrupt the social order, requiring skillful management.
Determining the best ways to “resolve” evaluations presents greater challenges than resolving forecast questions.
I’ve been interested in this area for 5+ years but struggled to draw attention to it—partly because it seems abstract, and partly because much of the necessary technology wasn’t quite ready.
We’re now at an exciting point where creating LLM apps for both forecasting and evaluation is becoming incredibly affordable. This might be a good time to spotlight this area.
There’s a curious gap now where we can, in theory, envision a world with sophisticated AI evaluation infrastructure, yet discussion of this remains limited. Fortunately, researchers and enthusiasts can fill this gap, one sentence at a time.
[1] As opposed to [Forecasting About AI], which is also common here.
[2] Or at least, do as good a job as we can.
Thanks for clarifying your take!
I’m sorry to hear about those experiences.
Most of the problems you mention seem to be about the specific current EA community, as opposed to the main values of “doing a lot of good” and “being smart about doing so.”
Personally, I’m excited for certain altruistic and smart people to leave the EA community, as it suits them, and do good work elsewhere. I’m sure that being part of the community is limiting to certain people, especially if they can find other great communities.
That said, I of course hope you can find ways for the key values of “doing good in the world” and similar to work for you.