Co-founder of Arb, an AI / forecasting / etc consultancy. Doing a technical AI PhD.
Conflicts of interest: ESPR, EPSRC, Emergent Ventures, OpenPhil, Infrastructure Fund, Alvea.
Co-founder of Arb, an AI / forecasting / etc consultancy. Doing a technical AI PhD.
Conflicts of interest: ESPR, EPSRC, Emergent Ventures, OpenPhil, Infrastructure Fund, Alvea.
The main problem with Terminator is not that it is silly and made-up (though actually that has been a serious obstacle to getting the proudly pragmatic majority in academia and policy on board).
It’s that it embeds false assumptions about AI risk: “no risk without AGI malevolence, no risk without conscious AGI, no risk without greedy corporations, AGI danger is concentrated in androids”, etc. These have caused a lot of havoc.
If I could choose between a world where no one outside the field has ever heard of AI risk and a world where everyone has but as a degraded thought-terminating meme, I think I’d choose the first one.
On fancy credentials: most EAs didn’t go to fancy universities*. And I guess that 4% of EAs dropped out entirely. Just the publicly known subset includes some of the most accomplished: Yudkowsky, Muehlhauser, Shlegeris?, Kelsey Piper, Nuno Sempere. (I know 5 others I admire greatly.)
On intelligence: You might be over-indexing to research, and to highly technical research. Inside research / writing the peak difficulty is indeed really high, but the average forum post seems manageable. You don’t need to understand stuff like Löb’s theorem to do great work. I presume most great EAs don’t understand formal results of this sort. I often feel dumb when following alignment research, but I can sure do ordinary science and data analysis and people management, and this counts for a lot.
On the optics of the above two things: seems like we could do more to make people feel welcome, and to appreciate the encouraging demographics and the world’s huge need for sympathetic people who know their comparative advantage. (I wanted to solve the education misconception by interviewing great dropouts in EA. But it probably would have landed better with named high-status interviewees.)
* Link is only suggestive evidence cos I don’t have the row-level data.
Quick update since April:
We got seed funding.
We formed a board, including some really impressive people in bio risk and AI.
We’re pretty far through hiring a director and other key crew, after 30 interviews and trials.
We have 50 candidate reservists, as well as some horizon-scanners with great track records. (If you’re interested in joining in, sign up here.)
Bluedot and ALLFED have kindly offered to share their monitoring infrastructure too.
See the comments in the job thread for more details about our current structure.
Major thanks to Isaak Freeman, whose Future Forum event netted us half of our key introductions and let us reach outside EA.
The above says nothing about the UN’s cost, or the expected cost of fixing it (including, most of all, the careers it would consume). As such the post doesn’t bear on our decisions at all.
In 2020, the UN spent $62,581,351,665.73.[1] This is actually only ~one and a halforders of magnitude above EA spending. It seems pretty clear to me they don’t produce 40x the impact.
Envelope time: We should expect the UNDP and the WFP to be the highest-impact programmes. (COVAX did good things, but it wasn’t “a UN project”, it was a CEPI and GAVI project with institutional cover from the WHO and distribution help from UNICEF.[2]) Roughly how cost-effective are they?
In 2018, two years before it won a Peace Prize,[3] the World Food Programme was ranked worst of 40 largest aid agencies on the QuODA scale (decent proxies for aid quality). A 2008 study found that UN agencies were by far the least efficient agencies, with the WFP disbursing just $30,000 per employee, where the average was $1,000,000.
“Aha!” you reply. “But that inefficiency, combined with their massive spend, just makes it more important for us to fix!”
To put it lightly, this seems prohibitively hard. Anecdata: Two idealistic friends of friends joined the UN straight out of college, joining different agencies. Both quit a couple years later, shocked at the sheer size of the hospitality budgets and the rigidity of the bureaucracy. You can see this as removing their naivete, but I prefer to say that the UN burns an extremely scarce resource: idealistic agency.
There is no steering such a huge and multipolar organisation. Even having a thousand bright people climb the ranks might have little effect, because the UN is the way it is because of powerful national incentives and organisational cruft, and because there are already 44,000 former idealists in it. The UN is so large, it spends hundreds of millions of dollars a year on coordinating between different parts of itself (the 2011 number was $237m). Smaller organisations outmanoeuvre them all the time.
I take the point about the Gates Foundation’s UN programme and wish them luck. But GF has a different option set from EA (: they ignore lots of the most high-impact things), and so their best option is unlikely to be the same as ours even if they are best allocating their resources.
The above is only a sketch of a real cost-benefit analysis, but it’s enough for me.
It’s always good to look at precedents, and as we grow and exhaust better opportunities, there may come a day when it makes sense to reform the ancient giants. But please notice the skulls.
I got this by naively summing this data, and am probably doublecounting something.
For an INGO project, COVAX moved fast and was cheap. And it cost $16bn, all on logistics and admin (~all of the vaccines were donated from national governments). They distributed half of the doses they were donated.
$16 per dose is a good deal (though if I was doing a proper impact analysis I would have to count the ~$40 dose unit cost somewhere, or else share COVAX’s impact with the donor countries).
This tells you something about the Peace Prize.
A big part of my getting into EA was this debate between Oxford lefties and the baby 80k staff. The socialist/deontological case was weaker. But the points that Mills makes about systemic change and the streetlight fallacy describe the two biggest ways EA practice has changed in the last decade. We moved in his direction, despite him.
Valid—but for what it’s worth, I get a lot of real connection out of them. And I’ve never raced toward anyone just because they’re successful, and I accepted every meeting request.
Serendipity is good, but it would be very strange if trying to aim for conversations you like didn’t improve the resulting conversations.
Don’t underestimate the loss of being stuck in conversations for longer than you wanted. There are a lot of personalities who benefit from a time limit.
Agree with the spirit—there is too much herding, and I would love for Schubert’s distinctions to be core concepts. However, I think the problem you describe appears in the gap between the core orgs and the community, and might be pretty hard to fix as a result.
What material implies that EA is only about ~4 things?
the Funds
semi-official intro talks and Fellowship syllabi
the landing page has 3 main causes and mentions 6 more
the revealed preferences of what people say they’re working on, the distribution of object-level post tags
What emphasises cause divergence and personal fit?
80k have their top 7 of course, but the full list of recommended ones has 23
Personal fit is the second thing they raise, after importance
New causes, independent thinking, outreach, cause X, and ‘question > ideology’ is a major theme at every EAG and (by eye) in about a fifth of the top-voted Forum posts.
So maybe limited room for improvements to communication? Since it’s already pretty clear.
Intro material has to mention some examples, and only a couple in any depth. How should we pick examples? Impact has to come first. Could be better to not always use the same 4 examples, but instead pick the top 3 by your own lights and then draw randomly from the top 20.
Also, I’ve always thought of cause neutrality as conditional—“if you’re able to pivot, and if you want to do the most good, what should you do?” and this is emphasised in plenty of places. (i.e. Personal fit and meeting people where they are by default.) But if people are taking it as an unconditional imperative then that needs attention.
[Was this title written by an inner optimiser?]
More seriously, this is a very powerful set of ideas and attitudes and I wish I had known them about 15 years earlier. (For contrast, during my school work experience I painted lines on country roads.)
You know my views about high schoolers being systematically underestimated and fully capable of greatness, so well done for bucking the trend. That said, there is such a thing as too much agency (e.g. starting a company without checking the competition or without knowing what the market fit is; e.g. starting a big impact-oriented project without looking to see if it’s been done).
It seems likely that summers spent reading whatever you feel like, and even years spent just becoming yourself, yields certain virtues and groundedness which full blown first-order life optimisation doesn’t. The annoying thing is that I can’t say which of the two any given person needs more of on the margin.
(See also Owen on overoptimisation or Elizabeth on being a potted plant.)
Peter Hartree made a shockingly useful plugin for Google docs; lets you search comments, loads > 10x faster than Google’s native comments.
My thoughts and picks from judging the contest.
Many of my picks narrowly missed prizes and weren’t upvoted much at the time, so check it out.
The criticism contest has an anonymous submission form too.
Thanks for this honest account; I think it’s extremely helpful to see where we’re failing to communicate. It also took me a long time (like 3 years) to really understand the argument and to act on it.
At the risk of being another frustrating person sending you links: I wrote a post which attempts to communicate the risk using empirical examples, rather than grand claims about the nature of intelligence and optimisation. (But obviously the post needs to extrapolate from its examples, and this extrapolation might fall foul of the same things that make you sceptical / confused already.) Several people have found it more intuitive than the philosophical argument.
Happy to call to discuss!
Well done on public correction! That’s always hard.
It’s key to separate out “social agency” from the rest of the concept, and coining that term makes this post worthwhile on its own. Your learned helplessness is interesting, because to me the core of agency is indeed nonsocial: fixing the thing yourself, thinking for yourself, writing a blog for yourself, taking responsibility for your own growth (including emotional growth, wisdom, patience, and yes chores).
has inside views
I think you mean “has strong inside views which overrule the outside view”. Inside views are innocuous if you simultaneously maintain an “all things considered” view.
Because of a quirk of the instructors and students that landed in our sample, ESPR 2021 went a little too hard on agency. We try to promote agency and wisdom in equal measure, which usually ends up sounding a lot like this post. Got there in the end!
Shouldn’t the title be “Proportional Representation seems overrated”?
PR is often what people mean by voting reform, in the UK, but there are options without these problems, e.g. approval voting.
Inspired by Jaime’s charming rundown of his quarterly(!) output, I’ll put something up:
In 2021, I
led a study of mask-wearing for COVID at the most zoomed-out level (unpicking heavily confounded national epidemiology stats). This was one of the hardest things I’ve ever done for a few reasons: my first big Bayesian model, my first big journal paper, the worst peer review I’ve ever seen, the incredibly poor data, taking on a field I’ve never taken a class in, a mob of hooting trolls on Twitter.
This led to me advising the British government on winter covid policy wtf.
recovered from 5 months of that by coming out of pandemic mode. I travelled to Estonia, Czechia, Stonehenge, Iceland, and did my first ever trip to the east coast of America. Saw my family for the first time in 2 years.
won an Emergent Ventures grant despite my application being fairly deranged
got into a conference, my first AI safety paper (a negative result)
won a cybercrime hackathon run with the Dutch Serious Crime Unit
taught at two amazing maths camps for teenagers. This was probably the single best thing all year.
a blogpost from last year blew up and earned me three job offers (Roam, Neuro, CEA?) and an invite to write for Nature. Some people actually in the field adopted and expanded it.
started an EA consultancy, Arb, with a friend. We got three big contracts, and have finished 4 subprojects so far, watch this space.
got rejected for an Amazon Research Internship within 4 hours
got rejected for the Vitalik AI Safety Fellowship, no reason given.
got rejected for the GovAI Summer Fellowship. No reason given, but it might be because my proposal was a little edgy: “Mediocre AI Safety As Existential Risk”.
couldn’t find a venue for our seasonality paper somehow
got my first EA grant, to help with executive dysfunction in EA students
made a bunch of friends and was adopted as an Irish neoliberal(?)
quit caffeine and booze entirely (from low levels)
did a bunch of reviews for the AI Safety Camp. The standard is pretty intense now
tried vyvanse and wellbutrin
turned off all morning alarms and wake whenever
finally got some crypto and ended up 10x in 5 months
got a laptop for ~free because Lenovo’s website was broken
Currently doing 3 months at the FTX Bahamas thing and have suspended my PhD. It is pretty amazing.
I noticed this a while ago. I don’t see large numbers of low-quality low-karma posts as a big problem though (except that it has some reputation cost for people finding the Forum for the first time). What really worries me is the fraction of high-karma posts that neither original, rigorous, or useful. I suggested some server-side fixes for this.
PS: #3 has always been true, unless you’re claiming that more of their output is private these days.
Maybe the lesson is: “even if you don’t win, you might shape the movement”
It would be extremely surprising if all of them were being given the correct amount of attention. (For a start, #10 is vanilla and highly plausible, and while I’ve heard it before, I’ve never given it proper attention. #5 should worry us a lot.) Even highly liquid markets don’t manage to price everything right all the time, when it comes to weird things.
What would the source of EA’s perfect efficiency be? The grantmakers (who openly say that they have a sorta tenuous grasp on impact even in concrete domains)? The perfectly independent reasoning of each EA, including very new EAs? The philosophers, who sometimes throw up their hands and say “ah hold up we don’t understand enough yet, let’s wait and think instead”?
For about 4 years I’ve spent most of my time on EA, and 7 of these ideas are new to me. Even if they weren’t, lack of novelty is no objection. Repetition is only waste if you assume that our epistemics are so good that we’re processing everything right the first (or fourth) time we see it.
What do you think EA’s biases and blindspots are?
The closing remarks about CH seem off to me.
Justice is incredibly hard; doing justice while also being part of a community, while trying to filter false accusations and thereby not let the community turn on itself, is one of the hardest tasks I can think of.
So I don’t expect disbanding CH to improve justice, particularly since you yourself have shown the job to be exhausting and ambiguous at best.
You have, though, rightly received gratitude and praise—which they don’t often, maybe just because we don’t often praise people for doing their jobs. I hope the net effect of your work is to inspire people to speak up.
The data on their performance is profoundly censored. You simply will not hear about all the times CH satisfied a complainant, judged risk correctly, detected a confabulator, or pre-empted a scandal through warnings or bans. What denominator are you using? What standard should we hold them to? You seem to have chosen “being above suspicion” and “catching all bullies”.
It makes sense for people who have been hurt to be distrustful of nearby authorities, and obviously a CH team which isn’t trusted can’t do its job. But just to generate some further common knowledge and meliorate a distrust cascade: I trust CH quite a lot. Every time I’ve reported something to them they’ve surprised me with the amount of skill they put in, hours per case. (EDIT: Clarified that I’ve seen them work actual cases.)