I currently lead EA funds.
Before that, I worked on improving epistemics in the EA community at CEA (as a contractor), as a research assistant at the Global Priorities Institute, on community building, and Global Health Policy.
Unless explicitly stated otherwise, opinions are my own, not my employer’s.
You can give me positive and negative feedback here.
Following up via dm for more details.
Maybe it’s obvious, but there are lots of situations where believing an incorrect fact is less harmful than learning the incorrect fact too slowly. “Real” bayesians who are maximising EV won’t have this issue, as they’ll be happy to make decisions on the basis of things they think are unlikely to be true.
A more realistic choice is between being the kind of person who doesn’t respond quickly to changes in the world, or being someone who might respond quickly enough but is sometimes (maybe even often) wrong.
Also what are the main or best open source projects in the space? Or if someone wanted to actually use LMs for forecasting, what is better than just asking o3 to produce a forecast?
I haven’t been following this area closely, but why aren’t they making a lot of money on polymarket?
I was going for the second, adding some quotes to make it clearer.
Any of those options sounds reasonable to me. If it were me and I no longer endorsed the post, then I would delete it (though I wouldn’t do this for posts giving a substantive take or more object-level content).
Nw, I think
It dawned on me why my proposals were ignored: Some no-name guy with a moonshot project, which feasibility is hard to assess without a proper expertise. It’s easier to ignore than apologize for having no capacity to evaluate.
Implies that you applied and were rejected because of a lack of expertise from EA Funds—might be worth quickly clarifying that you haven’t applied yet (if that’s the case).
I’m not sure I quite understand the take from this post, but in case it’s clarifying many fund managers do talk to external people to help them evaluate a proposal, in cases where they don’t it’s generally because they feel they can evaluate the project by looking at sections of the application like their goals and track record.
> If this is the case, maybe it makes sense for EA Funds to consider having a separate fund for expert evaluation of promising projects?
This is supposed to be what each fund does right now.
I’m not sure why your application was rejected, as when I did a quick search, I couldn’t see any applications from your forum username.
“Plausibly best people who have ever lived” is a much lower bar than “best people who have ever lived”.
I’m a huge fan of having high standards. Posts that are like “we reproduced this published output and think they made these concrete errors” are often great. But I notice much more “these people did a bad job or spent too much money” takes often from people who afaict haven’t done a bunch of stuff themselves so aren’t very calibrated, and don’t seem very scope sensitive. If people saw their projects being critiqued and were then motivated to go and do more things more quickly I’d think that was great (or were encouraged to do more things more quickly from “fear” of critiques) I think we’d be in a better equilibrium.
For example people often point out that LW and the forum are somewhat expensive per user as evidence they are being mismanaged and imo this is a bad take which is rarely made by people who have built or maintained popular software projects/forums or used the internet enough to know that discussion of the kind in these venues is really quite rare and special.
To be clear, I think the “but have they actually done stuff” critique should also be levelled at grantmakers. I’m sympathetic to grantmakers who are like “the world is burning and I just need to do a leveraged thing right now” but my guess is that if more grantmakers had run projects in the reference class of things they want to fund (or founded any complicated or unusual and ambitious projects) we’d be in a better position. I think this general take is very common in YC/VC spaces, which perform a similar function to grantmaking for their ecosystem.
Many examples of criticism in replies, are high quality posts that I think improve standards. I may spend an hour going through the criticism tag and sorting them into posts I think are useful/anti-useful to check.
Disproportionately deterring bad projects is a crux. I think if people are running a “minimise criticism policy” they aren’t going to end up doing very useful things (or they’ll do everything in secret or far from EA). I currently don’t think nearly enough people are trying to start projects and many project explorations seem good to me on net, so the discrimination power needs to be pretty strong for the benefits to pencil.
I think there are positives about criticism which I didn’t focus on, but yeah if I were to write a more comprehensive post I think the points you raised are good to include.
I’m not convinced that criticism is very counterfactually educational for 3rd parties. Particularly when imo lots of criticism is bad. Feels like it could go either way. If more criticism came from people who had run substantial projects or operated in the same field or whatever I think I’d trust their takes more. Many of the examples raised in this thread are good imo and have this property.
I agree that this is one of the upsides of criticism on the forum. I don’t think it outweighs the costs in many cases.
Have you run a public EA project before or spent time talking to founders of similar projects? This seems extremely off to me.
Damnit!
Rohin and Ben provided some examples that updated me upwards a little on critique posts being useful.
I think most of my points are fairly robust to the different definitions you gave so the line isn’t super important to me. This feels a bit nitpicky.
I don’t think that “criticism clearly mattering (e.g. getting someone to significantly improve their project)” is a very specific claim. I think that one of the main responses people would like to see to criticism of a specific project is for that project to change in line with the criticism. Unlike many of the other proposed benefits of criticism, it is a very empirical claim.
It suspect you think that this post should have been closer to “here are some points for and against criticism” on the EA Forum, but I don’t think posts need to be balanced or well-rounded like that, especially because, from my perspective, the forum is too pro-criticism but yeah, seems fine for you not to engage with this kind of content—I definitely don’t think you’re obliged to.
Those are great examples.
on
Almost all EA projects have low downside risk in absolute terms
I agree with some of the points on point 1, though other than FTX, I don’t think the downside risk of any of those examples is very large. I’d walk back my claim to the downside risk to most EA projects seems low (but there are ofc exceptions).
on
There are almost no examples of criticism clearly mattering
Agree that criticisms of AI companies can be good, I don’t really consider them EA projects but it wasn’t clear that was what I was referring to in my post—my bad. Responding quickly to some of the other ones.
Concerns with Intentional Insights
This seems good, though it was a long time ago.
It’s hard to tell, but I’d guess Critiques of Prominent AI Safety Labs changed who applied to the critiqued organizations
Idk if these are “EA” projects. I think I’m much more pessimistic than you are that these posts made better things happen in the world. I’d guess that people overupdated on these somewhat. That said, I quite like these posts and the discussion in the commentts.
Gossip-based criticism of Leverage clearly mattered and imo it would have been better if it was more public
This also seems good, though it was a long time ago and I wasn’t around when leverage was a thing.
Sharing Information About Nonlinear clearly mattered in the sense of having some impact, though the sign is unclear
Sign seems pretty negative to me.
Same deal for Why did CEA buy Wytham Abbey?
Sign seems pretty negative to me. Like even the title is misleading and this generated a lot of drama.
Back in the era when EA discussions happened mainly on Facebook there were all sorts of critiques and flame wars between protest-tactics and incremental-change-tactics for animal advocacy, I don’t think this particularly changed what any given organization tried to do, but it surely changed views of individual people
Not familiar but maybe this is useful? Idk.
Open Phil and RP both had pieces that were pretty critical of clean meat work iirc that were large updates for me. I don’t think they were org-level critiques, but I could imagine a version of them being critiques of GFI.
So overall, I think I stand by the claim that there aren’t many criticisms that clearly mattered, but this was a positive update for me. Maybe I should have said that a very small fraction of critical EA forum posts have clear positive effects or give people useful information.
This was a great comment—thanks for writing it.
a. An intelligence explosion like you’re describing doesn’t seem very likely to me. It seems to imply a discontinuous jump (as opposed to regular acceleration), and also implies that this resulting intelligence would have profound market value, such that the investments would have some steeply increased ROI at this point.
I’m not exactly sure what you mean by discontinuous jump. I expect the usefulness of AI systems to be pretty “continuous” inside AI companies and “discontinuous” outside AI companies. If you think that:
1. model release cadence will stay similar
2. but, capabilities will accelerate
3. then you should also expect external AI progress to be more “discontinuous” than it currently is.
I gave some reasons why I don’t think AI companies will want to externally deploy their best models (like less benefit from user growth), so maybe you disagree with that, or do you disagree with 1,2, or 3?
b. This model also implies that it might be feasible for multiple actors, particularly isolated ones, to make an “intelligent explosion.” I’d naively expect there to be a ton of competition in this area, and I’d expect that competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do). I’d naively expect that if there are any discontinuous gains to be made, they’ll be made by the largest actors.
I do think that more than one actor (e.g. 3 actors) may be trying to IE at the same time, but I’m not sure why this is implied by my post. I think my model isn’t especially sensitive to single vs multiple competing IEs, but possible you’re seeing something I’m not. I don’t really follow
competition would greatly decrease the value of the marginal intelligence gain (i.e. cheap LLMs can do much of the work that expensive LLMs do)
Do you expect competition to increase dramatically from where we are at rn? If not then I think current level of competition empirically do lead to people investing a lot in AI development so I’m not sure I quite follow your line of reasoning.