I currently lead EA funds.
Before that, I worked on improving epistemics in the EA community at CEA (as a contractor), as a research assistant at the Global Priorities Institute, on community building, and Global Health Policy.
Unless explicitly stated otherwise, opinions are my own, not my employer’s.
You can give me positive and negative feedback here.
(edit: I was mostly thinking of public criticism of EA projects—particularly projects with <10 FTE. This isn’t clear from the post. )
I think it’s wild how pro-criticism of projects the EA forum is when:
Most people agree that there are a lack of good projects, and public critique clearly creates barriers to starting new projects
Almost all EA projects have low downside risk in absolute terms
There are almost no examples of criticism clearly mattering (e.g. getting someone to significantly improve their project)
Criticism obviously driving away valuable people from the forum—like (at least in part) the largest ever EA donor
(Less important, but on priors, I’m not sure you should expect high-quality criticism of EA projects because they are often neglected, and most useful criticism comes from people who have operated in a similar area before. )
Edit: I’m curious about counterexamples or points against any of the bullets. There are lots of disagree reacts, and presumably some of those people have seen critques that were actually useful.
Oh right, I’m a bit confused about why you asked this question. It seems unreasonable to say that there are no hypothetical EA orgs that should be democratically elected, but also that result isn’t very decision relevant.
Like I can imagine a fictional “EAs for democratically elected EA orgs” org which should imo be democratically elected, does that mean I should have said yes to this poll? Or should we mostly consider real and highly plausible new orgs?
I like this post a lot. I often use “regret minimisation” as a frame when making important decisions, which feels quite similar to “doing things you would endorse on your deathbed”.
Some Positions In EA Leadership Should Be Elected
I think the way the question is framed is a bit leading. The optimal amount of democratically elected EA leadership is probably non-zero (idk lots of things are non-zero), but I struggled to come up with any examples of currently existing EA leadership positions that I felt should be elected.
I agree with this take, but I think that this is primarily an agency and “permission to just do things” issue, rather than people not being good at prioritisation. It requires some amount of being willing to be wrong publicly and fail in embarrassing ways to actual go and explore under-explored things, and in general, I think that current EA institutions (including people on the EA forum) don’t celebrate people actually getting a bunch of stuff done and testing a bunch of hypothesis (otoh, shutting down your project is celebrated which is good imo—though ~only if you write a forum post about it).
I guess overall, I don’t think that people are as bottlenecked on how much prioritisation they are doing, but are pretty bottlenecked on “just doing things” and not staring into the void enough to realise that they should move on from their current project or pivot (even when people on the forum will get annoyed with them). In part, these considerations converge because it turns out that many projects are fairly bottlenecked by the quality of execution rather than by the ideas themselves, and actually trying out a project and iterating on it seems to reliably improve projects.
In general, my modal advice for EAs has shifted from “try to think hard about what is impactful” towards “consider just doing stuff now” or “suppose you have 10x more agency, what would you do? Maybe do that thing instead?”.
There’s an adjacent take I agree which is more like:
1. AI will likely create many high-stakes decisions and a confusing environment
2. The situation would be better if we could use AI to stay in-step with AI progress on our ability to figure stuff out
3. rather than waiting until the world is very confusing, maybe we should use AIs right now to do some kinds of intellectual writing, in ways we expect to improve as AIs improve (even if AI development isn’t optimising for intellectual writing).
I think this could look a bit like company with mostly AI workers that produces writing on a bunch of topics, or as a first step, heavily LM written (but still high-quality) substack.
I’d bet that current models with less than $ 100,000 of post-training enhancements achieve median human performance on this task.
Seems plausible the metaculus judges would agree, especially given that that comment is quite old.
I think the mainline plan looks more like use the best agents/model internally and release significantly less capable general agents/models, very capable but narrow agents/models, or AI generated products.
Yeah that’s fair. I’m a lot more bullish on getting AI systems that satisfy the linked question’s definition than my own one.
Did you look at the metaculus resolution criteria? They seem extremely weak to me, would be intersted to know which critiera you think o3 (or whatever the best OAI model is) is furthest away from.
AGI by 2028 is more likely than not
Most of my uncertainty is from potentially not understanding the criteria. They seem extremely weak to me:
* Able to reliably pass a Turing test of the type that would win the Loebner Silver Prize.
Able to score 90% or more on a robust version of the Winograd Schema Challenge, e.g. the “Winogrande” challenge or comparable data set for which human performance is at 90+%
Be able to score 75th percentile (as compared to the corresponding year’s human students; this was a score of 600 in 2016) on all the full mathematics section of a circa-2015-2020 standard SAT exam, using just images of the exam pages.
Be able to learn the classic Atari game “Montezuma’s revenge” (based on just visual inputs and standard controls) and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play (see closely-related question.)
I wouldn’t be surprised if we’ve already passed this.
Bioweapons are an existential risk
Note that imo almost all the x-risk from bio routes through AI, and is better thought of as an AI-risk threat model.
You might believe future GPU hours are currently underpriced (e.g. maybe we’ll soon develop AI systems that can automate valuable scientific research). In such a scenario, GPU hours would become much more valuable, while standard compute credits (which iiuc are essentially just money designated for computing resources) would not increase in value. Buying the underlying asset directly might be a straightforward way to invest in GPU hours now before their value increases dramatically.
Maybe there are cleverer ways to bet on price of GPU hours dramatically increasing that are conceptually simpler than nvidia share prices increasing, idk.
I think that “moods” should be a property of the whole discourse, as opposed to specific posts. I find it a bit annoying when commenters say a specific post has a missing mood—most posts don’t aim to represent the whole discourse.
I think this kind of investigation would be valuable, but I’m not sure what concrete questions you’d imagine someone answering to figure this out.
Yeah, I think starting research nonprofits is an especially different skillset. Good point.
Fwiw, I am a fan of open-source public goods. I think more nonprofits should try to build products rather than papers (or at least be in that mindset). I am a little worried that doing research is just more accepted than producing products that people might use and that creates unhelpful barriers. Relatedly, I quite like the move towards essay websites more care goes towards thinking about the UX and trying to actually interact with users.
I mostly agree with this. I wasn’t really aiming to give a balanced take here on whether people should start for-profit instead of nonprofit—I just meant to list a few (imo) underrated by EAs features of for-profits.
I’m more confused about the poor personal fit point. I suspect that many EAs are also a bad fit (at least initially) for starting nonprofits, but the EA ecosystem makes it somewhat easier for nonprofits to survive (which imo is probably a good thing overall).
The version of your claim I most agree with is:
* An EAs comparative advantage is identifying the most overlooked and big deal moral considerations, and access to minimal viable nonprofit capital
* This selects for highly neglected things where you might be able to have an impact at a small scale, with not particularly high standards
* So even if EAs aren’t sufficiently competent to build high-growth companies, they can still have an outsized impact via founding nonprofits
I didn’t realise this, but apparently “regret minimisation” was popularised by Jeff Bezos, and this is literally how Jeff describes it. I guess it’s a very natural concept and this name is pretty descriptive.
https://www.fastcompany.com/90662406/jeff-bezos-uses-a-simple-framework-for-making-big-decisions-heres-how-it-works