Co-founder of Arb, an AI / forecasting / etc consultancy. Doing a technical AI PhD.
Conflicts of interest: ESPR, EPSRC, Emergent Ventures, OpenPhil, Infrastructure Fund, Alvea.
Gavin
Karma: 5,557
Appreciate this.
The second metric is aid per employee I think, so salaries don’t come into it(?) Distributing food is labour intensive, but so is UNICEF’s work and parts of WHO.
The rest of my evidence is informal (various development economists I’ve spoken to with horror stories) and I’d be pleased to be wrong.
Arb is a research consultancy led by Misha Yagudin and Gavin Leech. Here’s our review of our first and second years. We worked on forecasting, vaccine strategy, AI risk, economic policy, grantmaking, large-scale data collection, a little software engineering, explaining highly technical concepts, and intellectual history. Lately we’ve been on a biotech jaunt and also events.
We’re looking for researchers with some background in ML, forecasting, technical writing, blogging, or some other hard thing. Current staff include a philosophy PhD, two college dropouts, a superforecaster, a machine learning PhD, etc. We pay US wage.
Fully remote with optional long retreats. We spent a full half of 2022 colocated.
We only take work we think is important.
When producing the main estimates, Sam already uses just the virtual camps, for this reason. Could emphasise more that this probably doesn’t generalise.
The key thing about AISC for me was probably the “hero licence” (social encouragement, uncertainty reduction) the camp gave me. I imagine this specific impact works 20x better in person. I don’t know how many attendees need any such thing (in my cohort, maybe 25%) or what impact adjustment to give this type of attendee (probably a discount, since independence and conviction is so valuable in a lot of research).
Another wrinkle is the huge difference in acceptance rates between programmes. IIRC the admission rate for AISC 2018 was 80% (only possible because of the era’s heavy self-selection for serious people, as Sam notes). IIRC, 2023 MATS is down around ~3%. Rejections have some cost for applicants, mostly borne by the highly uncertain ones who feel they need licencing. So this is another way AISC and MATS aren’t doing the same thing, and so I wouldn’t directly compare them (without noting this). Someone should be there to catch ~80% of seriously interested people. So, despite appearances, AGISF is a better comparison for AISC on this axis.
Long list of AI questions
I put it in “galaxy-brained end-to-end solutions” for its ambition but there are various places it could go.
Well there’s a lot of different ways to design an NN.
That sounds related to OAA (minus the vast verifier they also want to build), so depending on the ambition it could be “End to end solution” or “getting it to learn what we want” or “task decomp”. See also this cool paper from authors including Stuart Russell.
It’s not a separate approach, the non-theory agendas and even some of the theory agendas have their own answers to these questions. I can tell you that almost everyone besides CoEms and OAA are targeting NNs though.
excellent, thanks, will edit
Shallow review of live agendas in alignment & safety
Oh great, thanks. I would guess that these discrete cases form a minority of their work, but hopefully someone with actual knowledge can confirm.
The closing remarks about CH seem off to me.
Justice is incredibly hard; doing justice while also being part of a community, while trying to filter false accusations and thereby not let the community turn on itself, is one of the hardest tasks I can think of.
So I don’t expect disbanding CH to improve justice, particularly since you yourself have shown the job to be exhausting and ambiguous at best.
You have, though, rightly received gratitude and praise—which they don’t often, maybe just because we don’t often praise people for doing their jobs. I hope the net effect of your work is to inspire people to speak up.
The data on their performance is profoundly censored. You simply will not hear about all the times CH satisfied a complainant, judged risk correctly, detected a confabulator, or pre-empted a scandal through warnings or bans. What denominator are you using? What standard should we hold them to? You seem to have chosen “being above suspicion” and “catching all bullies”.
It makes sense for people who have been hurt to be distrustful of nearby authorities, and obviously a CH team which isn’t trusted can’t do its job. But just to generate some further common knowledge and meliorate a distrust cascade: I trust CH quite a lot. Every time I’ve reported something to them they’ve surprised me with the amount of skill they put in, hours per case. (EDIT: Clarified that I’ve seen them work actual cases.)
Thanks for all your work Ben.
But a glum aphorism comes to mind: the frame control you can expose is not the true frame control.
What about factor increase per year, reported alongside a second number to show how the increases compose (e.g. the factor increase per decade)? So “compute has been increasing by 1.4x per year, or 28x per decade” or sth.
The main problem with OOMs is fractional OOMs, like your recent headline of “0.1 OOMs”. Very few people are going to interpret this right, where they’d do much better with “2 OOMs”.
Buckman’s examples are not central to what you want but worth reading: https://jacobbuckman.com/2022-09-07-recursively-self-improving-ai-is-already-here/
Despite my best efforts (and an amazing director candidate, and a great list of volunteers), this project suffered from the FTX explosion and an understandable lack of buy-in for an org with maximally broad responsibilities, unpredictable time-to-payoff, and a largeish discretionary fund. As a result, we shuttered without spending any money. Two successor orgs, one using our funding and focussed on bio, are in the pipeline though.
I’ll be in touch if either of the new orgs want to contact you as a volunteer.
Break self-improvement into four:
ML optimizing ML inputs: reduced data centre energy cost, reduced cost of acquiring training data, supposedly improved semiconductor designs.
ML aiding ML researchers. e.g. >3% of new Google code is now auto-suggested without amendment.
ML replacing parts of ML research. Nothing too splashy but steady progress: automatic data cleaning and feature engineering, autodiff (and symbolic differentiation!), meta-learning network components (activation functions, optimizers, …), neural architecture search.
Classic direct recursion. Self-play (AlphaGo) is the most striking example but it doesn’t generalise, so far. Purported examples with unclear practical significance: Algorithm Distillation and models finetuned on their own output.[1]
Treachery:
https://arxiv.org/abs/2102.07716
https://lukemuehlhauser.com/treacherous-turns-in-the-wild/- ^
The proliferation of crappy bootleg LLaMA finetunes using GPT as training data (and collapsing when out of distribution) makes me a bit cooler about these results in hindsight.
The only part of the Bing story which really jittered me is that time the patched version looked itself up through Bing Search, saw that the previous version Sydney was a psycho, and started acting up again. “Search is sufficient for hyperstition.”
Re: papers. Arb recently did a writeup and conference submission for Alex Turner; we’re happy to help others with writeups or to mentor people who want to try providing this service. DM me for either.
No